Not everybody have to be with everybody at all outings.

Call him(?) on it next time, right when he makes the comment. It's very hard to change this kind of behavior, so in case you need to hear this- that person is not your responsibility.