[D] Neat project that would "fit" onto a 4090?

Submitted by lifesthateasy t3_11ktxjl in MachineLearning

I've finally pulled the plug on a 4090 that'll arrive by the end of this week after ages with a 1050, and besides throwing everything ray traced at it, I also want to use it to train some deep learning models.

I do know the talk of the town, LLMs, are waaay too big to be done on such a card (iirc ChatGPT was train on 1024 industrial cards), but I was wondering if there's some neat DIY projects I could set up and train in a human amount of time (something that's not neural style transfer, that already ran on the 1050 too).

FYI I'm not specifically looking for language modeling, Chat was just an example about a model that'd def be too big.

10

Comments

You must log in or register to comment.

Disastrous_Elk_6375 t1_jb8y5r2 wrote

GptNeoX should fit with 8bit and low prompt sizes. GptJ-7B should fit as well with 16bit inference. On smaller models you might even be able to do some finetuning for fun.

There's a couple of coding models from salesforce that you could fit comfortably. Check out FauxPilot for a copilot "clone".

8

Dankmemexplorer t1_jb9xjl9 wrote

-stable diffusion would be fun to play with

-you can try simple computer vision tasks / finetune a model to detect your cat or something

4

gopher9 t1_jbb9f2l wrote

RWKV works rather well on 4090.

4

thekevsh0w t1_jcnkybt wrote

>RWKV

How bout a 3080 TI? im guessing the 12 gigs vs 24 gigs of VRAM is gonna be rather lacking :(

1

gmork_13 t1_jbbj49n wrote

With fp16/int8 you can probably stick a couple of LLMs of smaller size onto that card.
Have a look around, with fp32 it's about 1B params per 4GB of VRAM. Halve it for fp16 and again for int8 (very roughly).

2

abstract000 t1_jbfcydi wrote

You will not be able to train huge models like ViT or BERT from scratch, but for fine tuning a 4090 is waaaaaaay enough. Even with my 2080Ti I achieve really fine results with a decent amount of time.

Your best friend is Huggingface, because transformer are life with a nice GPU

With your new config here would be my first try:

  • ViT
  • BERT
  • Donut because it's hell of a thing
  • Everything you can find on huggingface and makes you curious
2