I do the same thing, but my reason is that if you're going to hit me I want you to live with the knowledge that you were looking me in the eye when you did it.

(your reason is better, but I'm kind of face-blind)