You are incorrect in thinking that the hot air outside immediately flooding back in when you turn it off.

When you turn on a single hose unit, it is CONSTANTLY bringing in hot air when it is in use. The air has to come from somewhere and if you are dumping the air from inside to the outside, it must be replaced with outside air to equalize the pressure. This comes from every crack in your house, every tiny gap, etc.

so sure you are blowing cold air around in the house but to do that, you’ve turned your whole house into a vacuum pulling in hot air from the outside. It’s pretty inefficient.