I usually provide a much more thoughtful answer than this, but simple is sometimes better.

Hell nah, dude. This man ain't worth being with.

That's a hard one to approach.

I dated my ex for about four years, and this was never a concern of hers, outside of her asking me to text her brother now and again. That said, I wasn't close with her older sister or younger brother. Sometimes you just don't have a lot in common with people - that's okay. If you have a family that gets together often (uncles, aunts, cousins, etc.), you'll see a lot of people that don't really have a lot in common but are cordial. You don't have to be close to everyone. Does her brother even care?

Flee to Cuba.

But honestly, man, I think things are just fine. Play it off as a joke, if you must. A bad one. But a joke. You two will be okay.