It's if the price setting is a result of stifling of competition.

The way collusion distorts free markets is if the colluding companies have enough clout to gang up and stop those that want to undercut them.

If you decide to charge $10k for your apartment, and I see that you are able to rent it and I then raise my rent to $10k, that's not an efficiency problem.

If you decide to charge $10k for your apartment and I decide that you are an idiot and I will take your customers by undercutting your prices, but then you are able to start hassling my tenants or making frivolous charges about alleged violations then it is an issue.

That's the problem. Not that companies cooperate in some fashion against consumers, but that they cooperate in some fashion against potential competitors... which then obviously creates higher prices and hurts consumers.