You are forgetting the scale we are talking about. Invasive species often don't do well in environments. When they do the area is usually very similar to their home. Meanwhile that ice has been frozen for millions of years. There were very few mammals when that was warm. While I won't say it's impossible for them to effect modern creatures, it makes sense that it won't be as easy as an animal moving to another part of the world better suited for them.