I think your problem here is that you're assuming that space is a perfect vacuum. It isn't. There is material floating around out there, and the atoms floating around in space are not at absolute zero.

So I think you're making an assumption on how much of a vacuum space is compared to how much it really is.

And yes, there isn't much stuff in space. In interplanetary space there are about 5 particles per cubic centimeter, and outside our solar system it is significantly less density than that.