I've recently discovered models such as ChatLLaMA that allows you to create a "ChatGPT" but you need Meta's LLaMA weights (yes, you can find them in torrents but that's not the point of the question). Similar limitations found in other cases.

Therefore I wanted to try to find an open source: dataset (in addition to hugging face), "base model", "chat model" AND that it is feasible to train with a commercial computer with a very good GPU (NVIDIA, etc.). With this get at least decent results.

Also would be interesting to distinguish between solutions with commercial limitations and those who don't.

Thanks!

• EDIT • A first solution I already found is this: https://github.com/databrickslabs/dolly based on this https://huggingface.co/EleutherAI/gpt-j-6B, but looking for some discussion and perhaps other/better solutions.

Comments

big_ol_tender t1_jdsvods wrote on March 26, 2023 at 10:47 PM

#2,379,087

Lol

Hands0L0 t1_jdsvv9h wrote on March 26, 2023 at 10:49 PM

#2,379,114

You and everyone else here

W_O_H t1_jdswox3 wrote on March 26, 2023 at 10:55 PM

#2,379,219

Ye it will be easy if you have a budget of 200+ mil.

LastVariation t1_jdsx10i wrote on March 26, 2023 at 10:58 PM

#2,379,281

Start by training a gpt-2, then add 2, then reinforce it to not be crazy by feeding chatGPT responses

BeautifulLazy5257 t1_jdsyfbc wrote on March 26, 2023 at 11:08 PM

#2,379,476

I'd start by going through the Hugging Facd courseware.

You'll learn in the first chapter of their courses that it is just better for people to fine-tune pre-trained models. That's what they are there for.

It costs a lot and produces a lot of energy and heat waste to train a LLM from scratch.

m0ushinderu t1_jdtbee7 wrote on March 27, 2023 at 12:50 AM

#2,381,294

GPT's model architecture has not exactly been shrouded in mystery anyways. It is all about the training data and training methodology. Thats why projects that crowd sources training data such as OpenAssistant is so important rn. You can check them out!

he_who_floats_amogus t1_jdtwp8t wrote on March 27, 2023 at 3:54 AM

#2,384,379

>open source dataset ... feasible to train with a commercial computer ... decent results

Choose two. Therefore, you can approach this one of three ways:

Use closed source data (eg. where your starting point is a pre-trained model and you're doing additional fine-tuning training)
Use millions of dollars of compute resource (a "very good GPU - nvidia etc" does not meet this standard)
Accept poor results

Deep-Station-1746 t1_jduhbth wrote on March 27, 2023 at 8:13 AM

#2,386,806

OP is peaking on Dunning-Kruger curve right now.

Deep-Station-1746 t1_jduxvm4 wrote on March 27, 2023 at 11:51 AM

#2,388,845

Replying to [deleted] (#2,387,841)

You know what? provide me with a 2-3 sample "good" responses to the above post, and explain why they make for a better response than what I wrote, and I'll actually use them from now on to respond to low-effort posts from this sub.