Yeah, no. For starters, both human and neanderthal population dropped significantly at around 70 000ish years ago, probably due to some kind of cataclysmic event (a supervulcano eruption was long suspected but it might not have been the cause). Human population recovered, neanderthal population didn't.

But even if it did, would you be able to tell neanderthal apart from modern humans?

EDIT: also the last known neanderthal population seem to have died out during time of periodic climate change that would have fucked with their food supply.