Most of the time people who die from being inverted is not due to asphyxiation or blood pooling in their brain. Instead it is heart failure.

Blood is mostly water, and water is fairly heavy. Most people have around 1.5 gallons of blood in their body which would weigh about 12.5 pounds. This is constantly being circulated around the body by the heart, day in and day out, but for much of the time people's bodies are oriented vertically. Blood must move down into the lower extremities (legs mostly) and then be pushed back up by the pressure of more blood being pushed down. Blood coming directly from the heart is in arteries which are at relatively high pressure (the ones that do the pulsing spurts when punctured), while the return to the heart is through veins which are relatively low pressure.

So there is a significant weight of fluid that needs to be pushed up from the legs to the level of the heart, but from a heart that only pushes some of the time and using pressure that has filtered through leg tissue reducing it significantly. It needs some help! To aid in this the body has specially adapted one-way valves in the veins, basically flaps that prevent blood from flowing backwards down into the legs. Blood can sort of ratchet its way back up to the heart, making the flow much more efficient. Another factor is that the legs have large muscles which when they contract will squeeze on the veins and actually push blood out of the tissue, increasing pressure aiding the return of blood back into the torso. This is why you are told not to lock your legs when standing up such as in military formation, marching band, choir, etc. Locking your legs will relieve pressure from your leg muscles but this means they aren't pushing blood which causes blood pressure to drop and potentially cause someone to faint.

Now if you flip someone upside-down they don't have any of those valves to prevent back flow of blood. The body just isn't adapted to having the head on the bottom and legs on top, Also things like your head simply don't have muscles capable of helping out with blood circulation either. Instead to keep blood circulating properly your heart needs to just pump harder. Which it will, until it cannot anymore. At that point the heart failure occurs and the person dies.

For babies, there just isn't much difference between their top and bottom. The distance doesn't change the necessary pressure that much.