>He brought it up now though, saying that “If we are going to continue dating it’s something I have to get used to.”

Not sure how much more clear he can be.

If you're going to be with him, you'll have to deal with visiting his hoarder mommy.

And that this issue is enough to bring you to "not talking" ... why are you bothering trying to continue the relationship? If it were a great partnership, you'd be talking through compromise or how to work with Mom to straighten out her life, but instead, he's hunkering down and you're just not talking to him.