You must log in or register to comment.

currentscurrents t1_jdn0opn wrote

The Nvidia H100 marketing material does advertise a configuration for linking 256 of them to train trillion-parameter language models:

>With NVIDIA NVLink® Switch System, up to 256 H100 GPUs can be connected to accelerate exascale workloads. The GPU also includes a dedicated Transformer Engine to solve trillion-parameter language models.

Doesn't necessarily mean GPT-4 is that big, but it's possible. Microsoft and Nvidia were working closely to build the new Azure GPU cloud.


lanky_cowriter t1_jdoeyi4 wrote

Sam talked about this on the Lex Friedman podcast, it's not true


pornthrowaway42069l t1_jdn6noe wrote

Not going to deny that GPT-4 looks impressive, but, they could set up 10 bajillion-quadrillion parameters, question is, do they have the data to effectively utilize all of these? Maybe its time to start looking into decreasing number of parameters, and making more efficient use of the data.


currentscurrents t1_jdn7spo wrote

Bigger models are more sample efficient for a given amount of data.

Scale is a triangle of three factors; model size, data size, and compute size. If you want to make more efficient use of data, you need to increase the other two.

In practice LLMs are not data limited right now, they're limited by compute and model size. Which is why you see models like LLaMa that throw huge amounts of data at smaller models.


pornthrowaway42069l t1_jdnmf0j wrote

I'm confused, how is that different from what I said? Maybe I worded my response poorly, but I meant that we should focus on smaller models, rather than those gigantic ones.


frequenttimetraveler t1_jdo9gw5 wrote

Altman did not say anything about that in Lex Fridman show. He said the 100T rumor was just a meme

How would run time scale with parameter size? Can we infer if 1T is true from the latency of the responses?