Viewing a single comment thread. View all comments

3deal t1_j2d5iuj wrote

System requierment : 4x RTX 4090


ThatInternetGuy t1_j2d5nkm wrote

170GB VRAM minimum.

So that's 8x RTX 4090.


3deal t1_j2d8bj5 wrote

I mean, for a startup it is not very expensive for all the benefit it gives.


Disastrous_Elk_6375 t1_j2de4o2 wrote

Can the 4090 pool their VRAM? I always thought that LLMs need GPUs from the A/V series so that they can pool memory. Am I wrong in thinking that?


zaptrem t1_j2e2lvb wrote

You can do pipeline parallelism via FairScale and HF Accelerate on any identical (and sometimes non identical) GPUs.