Cities like Athens, Rome or Constantinople had huge populations in ancient times, but they dwindle with time. According to some estimates, Constantinople had up to 500,000 inhabitants during the 8th century, and shrank to 45,000 at the time of the Fall. Rome had 800,000 people in 400 AD and 30,0000 in the middle of the 6th century. Athens had up to 600,000 in the 5th century BC, and by 1833 it was a town of only 4,000.

A big city need people to take care of it. When their populations shrank, the people try to stay close together, so entire neigborhoods in the outskirts had barely any people. And with barely any people, maintaining a building, or knocking it down to build something else, is not feasible in the majority of cases, because there's no need to use the place as a building.