Viewing a single comment thread. View all comments

Ttttrrrroooowwww t1_iyctkhw wrote

Normally your dataloader gets single samples from your dataset. Such as reading an image one by one. In that case RAM is never a problem.

If that is not an option for you (why I would not know), then numpy memmaps might be for you. Basically an array thats read from disk, not from RAM. I use it to handle arrays that are Billions of values.

2

somebodyenjoy OP t1_iycub7d wrote

I haven’t heard of mem mapping, seems like something I should look into, thanks!

1