That all sounds really awful. Here's what you do:

1. break it off. The worst your family can do is go "aw that's a shame :/"
2. enjoy being free from someone who antagonizes you.
3. spend the next three years working out and getting absolutely shredded.
4. find this dude again and then rip him in half like a phonebook.

I think you'll be much happier without all this mess.