bryceschroeder

bryceschroeder t1_jcygn0x wrote

Reply to comment by Akimbo333 in Those who know... by Destiny_Knight

>strongest

I am running LLaMA 30B at home at full fp16. Takes 87 GB of VRAM on six AMD Insight MI25s and speed is reasonable but not fast (It can spit out a sentence in 10-30 seconds or so in a dialog / chatbot context depending on the length of the response.) While the hardware is not "consumer hardware" per se, it's old datacenter hardware, the cost was in line with the kind of money you would spend on a middling gaming setup. The computer cost about $1500 to build up and the GPUs to put in it set me back about $500.

1