Because of the frequency? Obviously, if the range of even the mid-band 5G is a fraction of 4G, you need to build a lot more cell towers. And you know where that 4G, 5G and future 6G data travels the majority of the time? In the same fiber-optic cables as everything else. Operators need to build the cables all the way up to the 5G towers. And as far as I'm aware, the 5G towers don't generally communicate with each other.

Also, the core network needs to be fast enough to accommodate increased traffic. More undersea cables between continents, more underground cables within countries.

So no, the difference in infrastructure is obviously not minimal.

There are probably tons more nuances. Like I said, I'm not an expert.