I'd say religion is the most significant factor, but not because we dislike people who follow other religions than us. It's more that the news in our media about the followers of these religions is generally negative (and has been for some time now).

Even Polish atheists would probably dislike Muslims/Jews more on average than atheists in other European countries, so it's not a case of hating all other religions, more that hardly anyone has anything positive to say about these religions (both in the media and in daily life).

And as you can see from the chart, there aren't many Muslims/Jews in Poland, so not enough people to disprove unfair stereotypes / fight or defend against racism.