Why are you a doormat? Why do you avoid conversations and tell him "no, I will never do that"? Why do you not believe what he says and prefer to live in fantasy land?

This started 3 years ago. You ignored it, even had a baby, and then kept ignoring it? You had sex with him while he was saying all of that, putting you off, and you still kept having sex with him? Why? Why didn't you tell him, no, I'm not into that, then stop having sex and leave the room?

Your husband doesn't respect your boundaries. Your husband doesn't respect you. I can tell you with 90% certainty that if he sees you with another man, then he'll become insecure and blame you for his own insecurity. The number of times on reddit and elsewhere in which men are mad at their wive's for having a good time because they had sex with someone else is huge. Or then, he'll tell you that since you fucked someone, now he can go fuck another woman to be even.

This is fucking mess and the worst part is that you have a one year old baby that you are putting in the middle of this time bomb.

Get a grip! Tell him he is out of his fucking mind.

You are also putting yourself in a very dangerous situation. What if they spring on you and want you to have sex with this man and you freeze? It's a very common reaction, so you freeze, he has sex with you even if you don't want to but you are unable to say no, they don't ask for your consent. Then what?