Submitted by hducug t3_yanf0j in singularity
It’s been 2 years now since the release of gpt-3. Gpt-4 could be released any time now. I remember someone at open ai saying that gpt-4 would likely be released this year in July/august. 2 months later and there is no sign of gpt-4.
Microsoft owns GitHub as well, Satya is pretty big on open source recently
U-PaLM and Flan PaLM come to mind.
The first one shows we were noising the data incorrectly, different kind of noising has major benefits. Second shows that training on thousands of tasks boosts the model capability to follow instructions and also get better score. So maybe OpenAI had to change their plans midway.
It's also possible that they don't want to scale up even further because it's very impractical. Too expensive to use, not just to train. And recent models like Flan get GPT-3 scores on many (not all) tasks with just 3B parameters.
There's also a question about training data - where can they get 10x or 100x more? I bet they transcribe videos, probably all videos that can be accessed. Another approach is to use raw audio instead of text, works well. I bet they have a large team just for dataset building. BLOOM was managed by an organisation with 1000 people and a lot of their effort was into the dataset sourcing, trying to reduce biases.
was that about supporting open source or buying a large dataset with the excuse of supporting open source.
As far as GPT-4 being released in July/August, it's not possible to exactly predict how long it will take to have software ready because writing software requires solving problems for which you won't know the complexity of the solutions until after you've solved them.
But here's some info on Open Ai's progress and direction, for those who are interested.
Open Ai past timeline:
GPT-1 - 2018
GPT-2 - 2019
GPT-3 - 2020
Open Ai Codex - 2021 (GPT-3 but fine tuned to produce code)
Dalle-E - 2021 (Images from text)
Dalle-E 2 - 2022
Open Ai wants to be the first to build general artificial intelligence. Dalle-E helped them understand vision systems, and gpt helped them understand language models.
They are in a race to produce AGI, against companies such as deep mind.
>recent breakthroughs might've meant that they've restarted from scratch once or twice
That is almost certainly not happening given the cost of training. Probably more a case of "Measure thrice, check twice, cut once"
Supporting open source without being honest about their reasons is still supporting open source
If they are truly in for the long game of supporting open source that's good. The alternative is 'so long and thanks for all the fish'
Open source is open, there is nothing to buy.
look two comments up where the context of this is Microsoft now owning Github.
They did get into Linux a while ago, I believe they're playing the old 'if you can't beat them join them' game
I'm wondering, whether Microsofts CTO Kevin Scott is talking about GPT-4 here. GPT-4 could be released later this year.
Within a year IMHO.
A friendly race since they share so much of their findings periodically.
Can you guys imagine if you somehow directly linked a gpt-3 or 4 with a stable diffusion ML type platform. That would create kinda it's very own metaverse.
These are exciting times, really can’t wait to see the progress this enables over the next decade in medicine, science, gaming, VR, art, etc.
Just imagine what 2023 will bring us in AI advancements. Feels a lot like the semiconductor shrinking progress but further on the exponential curve. I wouldn’t be surprised if gpt-4 is delayed so they can incorporate all the new ways to train. I think the next step will be big+efficient and see if we can crack into those remaining cognition tests that Ai still falls short on
Friendly, just like Mohammad Ali's friendly public demonstrations of how to be the best boxer ever, followed by whooping you.
I wonder about the Graphcores "good computer" set for 2024 .
We are already living in GPT-4.
Interesting, how is sure that multi-modal models will be released later this year if he's not the one releasing them? Multi-modal means texts, pictures, touch and stuff like that combined into one?
Yeah I also heard that it would be released in December of 2022 but I doubt it! They'll probably release DallE 3
Graphcore had some business setbacks recently.
zero_for_effort t1_itbx41r wrote
I'm certainly in no position to provide any insight as a complete outsider to tech and AI, but I do wonder if all the recent breakthroughs might've meant that they've restarted from scratch once or twice. What do you do if you realise your model will be obsolete before it's even fully trained?