Viewing a single comment thread. View all comments

HiPattern t1_iyd91t0 wrote

Reply to comment by somebodyenjoy in If the dataset is too big to fit into your RAM, but you still wish to train, how do you do it? by somebodyenjoy

hdf5 files are quite nice for that. You can write your X / y datasets in chunks into the file. When you access a batch, then it will only read the part of the hdf5 file where the batch is.

​

You can also use multiple numpy files, e.g. one for each batch, and then handle the file management in the sequence generator.

3

somebodyenjoy OP t1_iydqmu0 wrote

This is perfect, I won’t have to invest in additional RAM. Thanks for the tip!

3