Viewing a single comment thread. View all comments

juliensalinas OP t1_jcktk8o wrote

Reply to comment by pitrucha in [D] An Instruct Version Of GPT-J Using Stanford Alpaca's Dataset by juliensalinas

This is maybe something I'll focus on in the future. But for the moment I find this fp16 version well suited for small budgets as it runs on a 16GB GPU while the native fp32 version of GPT-J requires at least 24GB of VRAM.

Also, with the bitsandbytes integration in HF Transformers you can use the model in 8 bits: https://huggingface.co/blog/hf-bitsandbytes-integration

12