“Should I just buy a 4090”

Ok Jeff Bezos

4090 clock speed is going to be faster than similar instances that use T4s, plus wayyyyyyy more CUDA cores. Training will be significantly faster, if you can fit the model on the 4090. If you can “business expense” a 4090 for your own machine, good lord do that.

or even try to get an used 3090. if OP can afford 4090, just go with that.

It really depends on how often you think you'll train with the mode

If it's something that you'll do daily for at least 3 months, then I'd argue you can justify the 4090.

Otherwise, if this is a single model you want to play around with, then use an appropriate ec2 instance with gpus (remember: start with a small instance and then upgrade the instance as you need more compute, and remember to turn off your instance when you're not using it)

I don't really know what type of data you're playing around with (if it's image, text, or audio data for example), but you should be able to get pretty far without using a by doing small scale experiments and debugging, and then finally using a gpu for the final training

You can also use tensorflow datasets that have the ability to stream data from disk during training time, meaning that you won't need to store all of your files in memory during training, and be able to get away with a fairly decent computer.

Good luck!

Even bigger gains with fp8 support in CUDA 12.

We use Google cloud buckets + tensorflow - it works well since you can always point a VM to a cloud bucket (e.g. tfrecords) and it just has access to the data. I know you can do something similar in Jax, haven't tried pytorch. It's the same in a Colab notebook. Not sure if you can point it to a cloud location from local machine though but as others are saying the 4090 might not be the best use case (e.g. you can use a TPU in a Colab notebook to get similar performance)

Sorry, I wrote it in a hurry and now I realize it came out wrong.

What I meant is that in my experience dealing with: moving data between buckets/vms, uploading data, logging into a terminal via ssh or using notebooks that crash from time to time (Sagemaker is a bit buggy), or just training cloud models has some annoyances that are hard to avoid and make the whole experience horrible. So, maybe I should "just buy a good GPU" (4090 is a "good" deal where I live) and stop trying stuff around in the cloud.

My first idea was a 3090, but I'm not based in the US, and getting a used GPU here is risky, it's easy to be scammed. A 4080 is around 2000$ here, 3090 new is 1800$, and a 4900 is 2500$. So I thought that if I decide to get a desktop, I should "just" go for the 4090 cause is 500-700$ more but I'd get double the speed than a 3090 and 8+ vram.

Thanks for your comment! I have tried using ec2 and keeping data in EBS but not sure if it's the best solution, what is your workflow there?

I'm playing around mostly with NLP and image models. Right now I'm trying to process videos, like 200GB for a retrieval problem, what I do is: get frames, get feature vectors from pre trained resnet, and resnext (this takes a lot of time). And then I train a siamese network on all of those vectors. As I said I have tried with s3 and sagemaker, but I have to move data into sagemaker notebooks and I waste a lot of time there. Also tried to process stuff in ec2 but setting the whole thing took me a while (downloading data, installing libraries, creating scripts in the shell to process videos, etc).

Ohh, I didn't know that about GCP, so you can point a VM to a bucket and it just "reads" the data? you don't have to "upload" data into the VM?

As I said in a previous comment, my problem with AWS (S3 and Sagemaker), is that the data is in a different network, and even though is still an AWS network, you have to move data around and that takes a while (when it's 200 GB of data).

Yeah as long as your VM is in the same region as the bucket it should be fine. Even if you have 200GB it doesn't take that long to move between regions either

Thanks a lot, gonna try it!

Well, there are ec2 instances that are already setup. How often do you do this sort of thing? It might be justified to build your own home setup, but as someone who does that themselves, I can tell you it's kinda tedious and you end up being your own IT guy

Ah! I am also not from USA. I got my used 3090 for ~US$900. Could be cheaper now. 3090 & 4090 has same VRAM (24 GB)

I was checking Marketplace, couldn't find any used below 1500$. Also, I just discovered that 3090 is 2.2k$ here now lol (that would be the cheapest option)... meanwhile in BestBuy it costs 1k$, was just thinking about traveling to the US with the other k lol