This happens to me with fiction audibooks as well; I can't focus on what's going on in the story and focus on something else at the same time for some reason. I can't stand them; the Graphic Audio ones are even worse, with the sound effects and music and stuff it all just gets scrambled in my head the same way a loud crowd in a touristy area would.

I can do non-fiction audiobooks depending on the narrator or subject occasionally, and I don't mind podcasts either.

For most books, I almost exclusively read either physical print or ebooks.