Submitted by AutoModerator t3_11pgj86 in MachineLearning
rikiiyer t1_jddanig wrote
Reply to comment by djmaxm in [D] Simple Questions Thread by AutoModerator
The 30B params of the model are going onto your GPUs VRAM (which should be 24GB), which is causing the issue. You can try loading the model in 8bit which could reduce size
Viewing a single comment thread. View all comments