Processing in a second language is not at all the same as listening to your first language. I'm a massive book addict, both text and audio. But when I try to read text in my second language (Spanish) it's really hard work. I can't keep it up for long at a stretch.

I haven't even attempted an audio book in Spanish. I love them in English, but 125% is the very most I can speed things up and enjoy them. I'd say, give yourself a break on this one. Accept that for now, English is work for your brain and don't expect 100% fluency instantly.