I think I'm the rude one in this situation even if I didn't mean it and should have asked him to clarify what he said so this awkward situation wouldn't have started to begin with.