Viewing a single comment thread. View all comments

trajo123 t1_jdhi7u8 wrote

Reply to comment by Rishh3112 in Cuda out of memory error by Rishh3112

The problem is likely in your training loop. Perhaps your computation graphs keeps going because you keep track of the average loss as an autograd variable rather than a plain numerical one. Make sure that for any metrics/logging you use loss.item().

5

humpeldumpel t1_jdhpl0w wrote

And also make use of the training and validation mode of the model

2

Rishh3112 OP t1_jdhib79 wrote

sure ill will give it a try thanks a lot.

1

Rishh3112 OP t1_jdhiguj wrote

i just checked in my training loop I'm using loss.item()

1

_vb__ t1_jdiwjqk wrote

Are you calling the zero_grad method on your optimizer in every step of your training loop?

3