It makes more sense that the downtown was built. And then a less dense area to support that. And instead of replacing all that low density (and the best neighborhoods in hindsight) they just expanded above that area. Explains the age too; lower Manhattan is 400 years old while midtown started building up in the late 1800s.