I know it’s just a fantasy but the way he talks about it, makes me thinks he really wants it.

Then it isn't a fantasy

I’ve done a LOT to be in this relationship with him

This statement concerns me. In every relationship there is a level of compromise that each partner makes but it is generally innocuous (what restaurant to eat at, what color curtains), it shouldn't be a compromise of your morals, self-respect, or safety and I hope what you've done to be in this relationship is more like the former rather than the latter.

It sounds like you want a traditional monogamous relationship that does not include risque or non-traditional partnerships. The 2 of you are fundamentally incompatible. It sounds like you have been incompatible since you decided that threesomes were no longer of interest to you. You should have left then but there's always today.