This is narc 101. Not saying he is one, but it's very manipulative. If you want to leave, however sad the other person becomes, you should be able to. Politely ask your friends to not indulge him because you've made your decision.