Ex home server owner here. The problem of used servers is that they might require a lot of maintenance, you hardly ever have warranty left and you could easily end up with some components reaching the end of their lifespan (even tho one of the best points of server is their redundancy and failproof-ness, but still expensive)

Good point.

Any idea what a NEW 1TB PC with say 50TB hard disk would cost?

Totally clueless honestly.

I tried randomly checking with this site, adding only the most basic compatible CPU and getting to the 1TB ram + 50TB 10k HDD it goes up to ~$52k.

But again, I never worked with new components nor servers with this sort of high-end specs

Could it be done with a cluster?

Good find.

At least we now know that for $50k max a working AI platform can be built.

That could be a worthwhile investment for someone starting their career .. not much different from a typical student loan .. and probably much more useful!

make no mistake. those ram would not be beneficial for you. almost all ai algorithms running on gpu. otherwise just too slow.

for example rtx 3060 is 22 times faster than my core i7 10700 f

How Good is RTX 3060 for ML AI Deep Learning Tasks and Comparison With GTX 1050 Ti and i7 10700F CPU

I think that it would be a very bad idea in most cases.

​

But I'm confused on why you are talking about RAM as something very important for AI.

Vram is what you really need, I would say. (I mean, you need some ram, of course, but probably 16 GB is enought... 32 at the most)

And you need to have that VRAM in GPUs powerfull enought, of course, it's not just having enought vram, is also computing power.

Standard ram would be mostly useless as ML uses GPU memory. Older teslas will consume a ton of power and won't be able to compete with cloud services in cost efficiency.

Maybe op should look at multiple 4080s running together, something like used when mining crypto? I know you can get special motherboards that are designed to allow you to have multiple cards connected, just need a honking great power supply.

10k isnt 7k usd?

No.
To run GPT here's what it actually takes.

GPT-3 is 175 billion parameters. Each parameter is a 32-bit floating point number.

So you need 700 gigabytes of memory.

For it not to run unusably slow, you need thousands of teraflops - many times what an old server CPU is capable of.

One Nvidia A-100 comes in 80 gigabyte of GPU memory models, and they are $25,000 each. You cannot use consumer GPUs because there is an interconnect you have to have connecting multiple GPUs together.

Thus you need at least 9 of them, or $218,750 just for the GPUs.

The server that hosts them, cooling, racks, etc adds extra cost. Probably at least $300,000.

The power consumption is 400 watts per A100, so 3.2 kilowatts.

Interesting enough, that's not unaffordable for a great many businesses.

Nobody will give you the weights so you can run locally a SOTA model. These academic/test models, sure. But the advanced ones that are built for profit/high end will not be given out that way.

You'll have to pay for usage. I mean if $1 gets you what would take an hour of work for someone with a college degree, it's easily worth paying it.

Not sure what the pay rates will turn out to be but using the current chatGPT it can slam out what would have taken me several hours in 30 seconds.

You could download any of the larger GPT-J models. Or even GPT-NEO. Part of the point of Huggingface is to give their models away freely. There are other entities out there that do so too. Kind of like the old Stone Soup concept.

But the advanced ones that are built for profit/high end will not be given out that way.

"Each Parameter is a 32-bit floating point number." I have been wondering that that ment. So you have a video or documentation explaining that that means?

yep this would be much better

Well, today you can probably get it down to around 350 GB (fp16) so around 150.000.

​

And probably soon it might work well with around 175 GB with fp8 so.... around 75.000.

But yeah, for now, it's too expensive. IF fp8 works well with this it might be possible to think about building a machine for personal with second hand products in 3-5 years...

​

Anyway this year we'll probably get open source models with better performance than GPT 3 and far less parameters. Probably still too much for consumer GPUs anyway :(

It''s time to double vram on consumer GPUs.

Twice.

Pretty please.

It doesn't work that way. You can't reduce precision like that without tradeoffs. Reduced model accuracy for one thing.

You can in some cases add more weights and retrain for fp16.

Int8 may be out of the question.

Also like chatGPT is like the Wright Brothers. Nobody is going to settle for an AI that can't even see or control a robot. So it's only going to get heavier in weights and more computationally expensive.

https://en.wikipedia.org/wiki/Single-precision_floating-point_format

Sure, there is a tradeoff but I think that for fp16 it isn't that terrible.

For fp8 I just don't know. There is people working with int8 to fit 20B parameters in 3090/4090, but I have no idea of at what price... Just wanted to say that the posibility does exist.

I remember reading about fitting big models in low precision but it was focused in performance/memory usage, but it showed that it was a very useful technique...

​

Anyway I can't find it now, but I found this while looking for it, haha:

https://twitter.com/thukeg/status/1579449491316674560

They claim almost no degratation with int4 & 130B parameters.

​

No idea how this could apply to bigger ones, or even about the validity of the claim, but it does sound well. We would be fitting 40B parameters in a 3090 / 4090...

​

Anyway I think that fp8 might not be out of question at all, but we will see :P

​

I know that you say "chatGPT is like the Wright Brothers. Nobody is going to settle for an AI that can't even see or control a robot. So it's only going to get heavier in weights and more computationally expensive"

And...Sure, no one is going to settle for less. But consumer hardware is very far behind and people is going to try and work with what they have, for now.

And there is some interest for it. You have NovelAI, DungeonAI and KoboldAI, and people plays with them, when frankly, they work quite poorly.

I hope that with the release of good open sourced LLM with RHLF (I'm looking at you, CarperAI and StabilityAI) & this kind of techniques we start to see this tech becoming more comonplace, maybe even used in some indie games, to start pushing for more VRAM on consumer hardware. (Because if there is a need there is a way. Vram is not that expensive anyway given the prices of GPUs nowadays...)

>And...Sure, no one is going to settle for less. But consumer hardware is very far behind and people is going to try and work with what they have, for now.

No they won't. They are just going to rent access to the proper hardware. It's not that expensive.

Good idea.

I have checked the pricing of 12gb & 24GB GPUs ... not too bad currently.

I suspect that say 5 x 24GB GPUs would come to under $10k.

Add in the special motherboard and maybe we are looking at around $15k for a complete hardware system.

With careful purchasing that could possibly come down to under $10k total.

And, yes, GPT can be run across multiple GPUS, both when training and for running.