I do exactly what you do. Take it out of my case, put it in the palm of my hand and put some soap on the phone. Wash phone thoroughly and rinse with water.

The shits water proof, and it touches my hands all day. Why would I not want to clean it every so often?