RandomCandor t1_j47bx4j wrote on January 13, 2023 at 5:49 PM

Reply to comment by currentscurrents in [D] Bitter lesson 2.0? by Tea_Pearce

To me, all that means is that the lay people will always be a generation behind from what the rich can afford to run

currentscurrents t1_j48csbo wrote on January 13, 2023 at 9:37 PM

If it is true that performance scales infinitely with compute power - and I kinda hope it is, since that would make superhuman AI achievable - datacenters will always be smarter than PCs.

That said, I'm not sure that it does scale infinitely. You need not just more compute but also more data, and there's only so much data out there. GPT-4 reportedly won't be any bigger than GPT-3 because even terabytes of scraped internet data isn't enough to train a larger model.

BarockMoebelSecond t1_j48mepq wrote on January 13, 2023 at 10:39 PM

Which is and has been the Status Quo for the entire history of computing, I don't see how that's a new development?

currentscurrents t1_j490rvn wrote on January 14, 2023 at 12:18 AM

It's meaningful right now because there's a threshold where LLMs become awesome, but getting there requires expensive specialized GPUs.

I'm hoping in a few years consumer GPUs will have 80GB of VRAM or whatever and we'll be able to run them locally. While datacenters will still have more compute, it won't matter as much since there's a limit where larger models would require more training data than exists.