That happened to me with jogging. I think sometimes it's just not your thing. For me, I realized my body type is just not right for jogging. Reading is a physical activity in that the eye movement is rather unnatural; not eye movement you do in every day activities. Also, you have to give yourself over to the writer---not everyone is okay with that. That communion with the writer doesn't come naturally to extroverted personality types because they are more action oriented. So maybe you should try table games or even chess to whet your intellect.