You need to tell him that if he’s not fine with you going and he should let you cancel, otherwise he’s it’s like he just wants to keep punishing you without accepting a solution. Isn’t there a way he could come with you? If not, you should remind him that you put up with his family for 3 years- he can take a bit of inconvenience for a few days.