You and your husband have a serious communication problem.

As it seems from your post, you never had a real conversation about both your needs? You were "unenthusiastic" when he mentioned his fantasies years ago and now that he mentioned it again you were quiet to see how far he will take it? You never sat down and said, "that's absolutely not my thing, I don't want to do anything like it"? Your husbands fantasy isn't a bad thing. Some couples might enjoy it. Some not.

That he talked about it outside your marriage without your clear consent is not okay, but neither is you being just passive about it.

You are desperately in need of couples therapy to learn how to communicate with each other.