He may not have mentioned this to you, but along with lots of recreational drugs goes a lot of recreational sex. He's presumably done with that phase, and he probably doesn't like the idea of you doing it!

Maybe you two are not on the same page with regard to expectations. It may just be incompatibility.