It takes more concentration to follow audio in a second language even if you can understand it all. I don't know why it works that way but I'm a native English speaker and went through the same thing with French podcasts. Start with audiobooks of books you've read before so if you miss a chunk you won't be lost, it'll get easier to follow as you go.