Hard disagree. Even if it is hurtful for the recipient, it might have been in good taste and honest, playful banter.

If you let your behaviour be defined by the most sensitive and high σ recipients, you cease to be able to conduct any form of communication apart from talking about the weather. And that might even offend some people then.