I don't think there is any salvaging of this relationship. Unless you are willing to strain your familial relationship. Implies that you painted a bad picture of him to your family. And with that bias they may not have liked him when they met him. But with how he reacted to you regarding their perception of him I don't it will be good for your mental health to stay in this relationship.

I did have a brother-in-law who my sister divorced after complaining all the time to our dad's side of the family about the negatives. But mom's side of the family loved him because they were always told about the positives in their relationship. She lived closer to dad so they influenced her to divorce him. It's sad how this worked out.

So point being if they already have a bad perception of him that is unlikely to change but it sounds like he reinforced that bad preception with the way he responded. So my verdict is to break up.