If you really don't want to wear the shirt, being honest with him is the way to go. Thank him for the gift and let him know how much you appreciate his thoughtfulness. Then tell him the shirt isn't your style and you'd like to exchange it for something you would be able to wear more often. He can choose to be hurt or he can be a grown-up about it. It's just a shirt.