There always is going to be someone who has been in a similar situation and made it work unless it's something REALLY weird. With that being said there also are people who did not make it work.

So you have to decide what you want.

If you want monogamy then you need to break up and be with someone who wants what you want including the sexual aspect of a relationship. You aren't wrong for wanting your partner to want you sexually.

But I personally say try out his offer first if you are going to break up with him.