Do you feel this way because of the comment he made.

Or do you feel this way because when you thought about your relationship as a whole and this is the conclusion you came to.

If it's the former, you can drop this, if it's the latter, then it's a problem.