for a sense of scale, GPT-NeoX, a 20 billion parameter model, requires ~45GB of vram to run. gpt-3 davinci is 175 billion parameters.

unless these models can be pared down somehow (unlikely, the whole point of training these huge models is because their performance scales with size), we will have to wait a decade or two for consumer electronics to catch up

Interesting. So you’re saying rather than adjusting the models to work on current devices the better option is actually designing the devices to work with these models ?

running the full gpt-3 on a laptop would be like running crysis 3 on a commodore 64. you cant pare it down enough to run without ruining it

> we will have to wait a decade or two

The best I can is 4 years. Take it or leave it.

my current gpu is 4 years old 😖

state of the art has gotten a lot better since then but not that much better

There are advances such as quantization that have enabled edge devices to run some pretty spicy models so i wouldn’t be surprised if we got it down to within gaming computers reach pretty soon. Also Google research revealed that GPT-3 was not trained efficiently and has too many parameters. So a newly designed model with way fewer parameters trained on the same data should perform as well as GPT-3.

(I am only a machine learning enthusiast and not an expert so take everything I say with a grain of salt)