Ttttrrrroooowwww t1_iyctkhw wrote
Normally your dataloader gets single samples from your dataset. Such as reading an image one by one. In that case RAM is never a problem.
If that is not an option for you (why I would not know), then numpy memmaps might be for you. Basically an array thats read from disk, not from RAM. I use it to handle arrays that are Billions of values.
somebodyenjoy OP t1_iycub7d wrote
I haven’t heard of mem mapping, seems like something I should look into, thanks!
Viewing a single comment thread. View all comments