You may be confusing "memory" with "disk storage".

For memory, turning off power basically deletes all memory as it needs power to "maintain state".

For disk storage, deleting a file just means removing any reference to it, but leaving the contents on the disk as it's more work to "erase" the contents.

Imagine you have a large book like the Bible, and each story is a marked with a bookmark. "deleting" the file means just throwing the bookmark into the garbage. The text is still there, but there is nothing to point to it. If you write a new file, you put a new bookmark where there is space, maybe it's on top of the old one, then you just write over top of the old text. Overwriting the data changes it, it doesn't accumulate or anything.