There is a short answer to your title question: "nobody knows and currently have no way of knowing". Any contradictions come from comparing ideas that aren't meant to go together because they're parts of different models.

We don't know if the universe is infinite. We don't know what was happening at the earliest moments of the universe. Was it a single point? Maybe, although that's a very unpopular idea among physicists, just like the singularity in black holes. It contradicts quantum mechanics. There's currently not a better model, but they're pretty certain infinite density should be false.

We simply do not have the maths or technology to answer those questions. It would require the "Theory of Everything", with quantum gravity and all that. And the tech to test it.

To the followup questions.

It's absolutely possible for infinite space to have infinite matter and still be relatively "empty". There's just more space than matter. Just repeat what you can see in the sky infinitely in all directions. There's no contradiction.

Expansion can be weird to think about, but that's why the baloon/rubber band imagery is helpful. If every point in space stretches the same amount, something 1 "unit" away will stretch to 2, but 2 will stretch to 4. So distance matters. "Our local area" is the few neighbouring galaxies that are gravitationally bound, so any expansion of space (at observed rates) that close wouldn't be significant.

Also, expansion isn't actually uniform, gravity does counteract it. There is no expansion within a galaxy, or even within clusters (as far as I understand). The concentration of mass counteracts expansion.

BTW, our immediate vicinity being fairly empty actually IS weird. Our galaxy is actually in a void, imaginatively called the Local Void. The galactic neighbourhoods are usually more busy.

As for CMB and flatness, I only have a rough idea based on what I've read, but essentially is goes like this: "flatness" of soacetime is all about parallel lines being parallel and angles in a triangle adding up to 180 degrees. So geometry you'd do on a sheet of paper. For example, on the surface of the earth which is NOT flat, you can draw three lines (along the great circles) which will meet at 90 degrees and yet form a triangle. That's non-euclidian geometry.

Where the CMB comes in is the fact that it represents the very early universe. But it directly corresponds to today's universe, but everything has grown since then. So depending on the curvature of the universe, the patterns in the CMB should have specific shapes, corresponding to the massive structures in the visible universe. So astronomers made the observations, made the modelling, and the shape of the CMBs patters corresponds to the model where spacetime is flat.