karyo
karyo t1_jb2qo4e wrote
Reply to comment by ChristmasInOct in LLaMA model parallelization and server configuration by ChristmasInOct
For inference?yes. Look at eleutherai transformer math page. Also others are trying out llama rn so check them out
karyo t1_jb2qhcd wrote
karyo t1_jb03jq0 wrote
The first question is kinda difficult. Deep speed, zero, Megatron all play into it. There's a reason somebody recently said that there are only 200 people on the world atm that can pull it off.
For the second question ,
4090s just won't cut it. Nvidia fused off P2P this generation so unless you have an embarrassingly parallel pipeline ( which current llms aren't) they are not useful. Problem is ada a6000 was restricted severely P2P wise.
If you're doing llms at billion scale you gotta get v,a,h100s
karyo t1_jb2qwti wrote
Reply to Should I choose Colab or RTX3070 for deep learning? by Cyp9715
Unless everything you want to try out fits in 3070 memory(8gb), if recommend colab.