Yeah, you messed up here. Best you can do is apologize, and remind him that size isn't all of that important. Sometimes we say things without thinking them through. What matters most is that you like each other. It sucks, but hopefully you can move forward from this