It isn't unrealistic at all if the odds of abiogenesis are prohibitively small. We can imagine all sorts of events with vanishingly small possibilities. We might not be able to wrap our human minds around the numbers involved, but that's kind of the problem: we look at the size of the known universe and say, well surely, x must have happened at least once. But without a sense of the probabilities involved, we simply do not have any reason to say whether x has happened or not.

There are plenty of conceivable events that happen precisely zero times (things that would violate the laws of physics), and we can imagine possible events that never happen at all -- simply because they are so unlikely that not even trillions of years of interactions between gazillions of particles will bring them about. We might posit that somewhere a teacup from the 1972 Sears-Roebuck catalog is orbiting a planet made of leather. This is certainly possible -- in the sense of not contradicting physical laws -- but it is so unlikely that, no matter how vast the universe is, we cannot be certain that such an item exists. Abiogenesis might simply be one of these mathematically highly unlikely events.

I'm actually not as skeptical about extraterrestrial life as I sound. I do think that, given the tendency that compounds have of quite naturally bunching together into slightly more complex compounds, it does seem reasonable to think that life is fairly abundantly distributed around the universe. But we simply don't know enough about life or about the universe yet. For aught we know, life could exist in the cores of neutron stars and on every god-forsaken rock in the universe -- or just here on this little blue rock for the past few billion years or so. Nobody knows.

Actually, for intelligent life to occur (and not be destroyed) there may be even more conditions to be fulfilled than the amount of galaxies.

Besides, you are looking only at one dimension here - size of the universe. There is one, even more hard to comprehend - time. And I find it entirely possible that there may have been civilizations before but what can be almost impossible - is to have the right matching for time and place.

Anyways, not taking aliens as a matter of fact is the right scientific approach for now. And probability may go to hell with such a sample size, as someone pointed above.