I hope nobody gives you a solution because this is too cute to put an end to.

Anyway, his target was lower down, you moved his hand up, he figured you were humoring him with a little bit of touching but were not in the mood for full-on sex.

You both comprehended each other's cues incorrectly, so any communication built on top of that false understanding was sure to go nowhere.