So there are a few possible reasons. Some buildings fell out of use because they built newer more modern buildings.

Some buildings were for religious purposes. So as people moved away from the ancient religions these buildings didn't serve much of a purpose.

Buildings were also subject to environmental destruction such as earthquakes, erosion, etc. A major earthquake is the reason why part of the one side is missing in the Colosseum.

But probably the biggest reason is that they were made out of useful or valuable resources like marble and iron. It was common for builders to take parts of buildings and in corporate them into new buildings to cut down on cost for construction of the new building. Take the Colosseum in Rome for instance. The Colosseum was partly made out of marble and most of its marble was taken and used in the construction of St Peter's Basilica and possibly used for private residences of the wealthy. Also with the Colosseum theives would drill holes into the building to steal the iron.

However if a building was repurposed for another use then the building would be frequently maintained. Take for instance the Pantheon in Rome. It's one of the best preserved ancient Roman building. In the early 600s AD the former Roman temple was converted to a Catholic church. As a result it was maintained and perserved.