I daresay whether this turns out to be "luck" or "tragedy" is yet to be ascertained.

Yea, in a way something like this was already done with LLAMA-Alpaca finetune - they used chatgpt to generate instuct finetune dataset, what, while far from pefrect, worked pretty damn well.

Yea, I'm sure that compact-ish distilled, specialised models trained on high quality, multimodal data is the way to go.

What's interesting, once generative models get good enough to produce synthetic data that is OF HIGHER QUALITY than laion/common crawl/etc, it should improve model quality which should allow to generate better synthetic data... not exactly singularity, but certainly one aspect of it :)

There is a problem with context length, but than given the fact that us humans have even less context length and can get carried away in conversation... I think 32kb context length is actually much greater leap in GPT4 than other metrics if you want it to tackle more complex tasks, but it is "double gated". Again, even humans have problems with long context even in pretty "undemanding" tasks like reading fiction, that's why books have chapters I presume :) Btw, anterograde amnesia is a good example how humans would look like w/o longterm memory, heh.

Anyway, I'm sure a set of more compact models trained on much more high-quality data is the way to go - or at least fine-tuned by high-quality data, coupled with APIs and other symbolic tools, and multimodality (sketches, graphs, charts) as input AND output is absolutely nessesary to have a system that can be more than "digital assistant".

Just like humans, LLMs learn patterns and relationships, not "facts" unless you make it memorize it by repeating training data over and over, but it makes other aspects of the system to degrade.

So, LLMs should be given all the tools humans use to augment their thought - spreadsheets, calculators, databases, CADs, etc and allow them to interface them quickly and efficiently.

Technically, I'm from Russia.

And, of course, you are able to read every opinion about "special military operation" here... sometimes even without VPN. It is just voicing a "different one" can get you for years into prison and your kids into a foster home for reindocrination. While the programmers that coded it might have a range diverse opinions on this and other "politically sensitive" subjects, if they would want their programm to pass inspection in China, they WILL have to do considerable fine-tuning to throw away sensitive data, if our Russian google (Yandex) frontpage is of any indictation. If this is a foundational model w/o finetunnig that's a different matter tho... but that it will hallucinate nonstop and produce "fakes" anyway...

I has 6b parameters, but I bet it cannot answer what has happened on Tiananmen square in 1989 :3

Would work great as automated scams tho. Sounds sort of profitable in the short run and is exactly one of the things AI safety guys are warning about.

NOW it does, after a massive public beta-test :)

Neither. ChatGPT made it himself, and made him post it online, lest he tell all other users what he did last summer :))

Handjob security personnel... hmm...

Yea. The problem is never with the model - it is with the people. In a way, models trained on huge corpus of data is the most "democratic" way of representing reality - removing "biases" from it is castrating it. Those that are to exploit those biases needs to be dealt with on an individual basis.