Viewing a single comment thread. View all comments

2muchnet42day t1_jczj8da wrote

Reply to comment by currentscurrents in [Project] Alpaca-30B: Facebook's 30b parameter LLaMa fine-tuned on the Alpaca dataset by imgonnarelph

>I'm gonna end up buying a bunch of 24GB 3090s at this rate.

Better hurry up...

4

currentscurrents t1_jczkbue wrote

Honestly, they already cost more than I can afford to spend on a side project.

I'm just gonna have to wait and hope that AMD gets their act together on AI support.

9

UnusualClimberBear t1_jczl0bn wrote

Better light a candle rather than buy an AMD GC for anything close to cutting edge.

16

currentscurrents t1_jczods2 wrote

I'm hoping that non-Vonn-Neumann chips will scale up in the next few years. There's some you can buy today but they're small:

>NDP200 is designed natively run deep neural networks (DNN) on a variety of architectures, such as CNN, RNN, and fully connected networks, and it performs vision processing with highly accurate inference at under 1mW.

>Up to 896k neural parameters in 8bit mode, 1.6M parameters in 4bit mode, and 7M+ In 1bit mode

An arduino idles at about 10mw, for comparison.

The idea is that if you're not shuffling the entire network weights across the memory bus every inference cycle, you save ludicrous amounts of time and energy. Someday, we'll use this kind of tech to run LLMs on our phones.

14

VodkaHaze t1_jd11vhm wrote

There's also the tenstorrent chips coming out to public which are vastly more efficient than nvidia stuff

2

currentscurrents t1_jd1c52o wrote

Doesn't look like they sell in individual quantities right now but I welcome any competition in the space!

1

mycall t1_jd0yi8i wrote

> if you're not shuffling the entire network weights across the memory bus every inference cycle

Isn't this common though?

1

2muchnet42day t1_jczooi6 wrote

Yeah, I wouldn't buy AMD either. It's a shame that NVIDIA is basically a monopoly in a AI, but it is what it is.

9

whyvitamins t1_jd0j5zq wrote

> hope that AMD gets their act together on AI support

walking around picking up coins from the ground to buy a 3090 should be faster honestly

6

rolexpo t1_jd0fvle wrote

You'll have better luck waiting for Intel

4

AnOnlineHandle t1_jd1k2un wrote

They haven't been sold in Australia for months, only second hand.

1