[N] 7 Predictions From The State of AI Report For 2023 ⭕

Submitted by LesleyFair t3_105n89m in MachineLearning

1) A SOTA Language Model is Trained on 10x More Data Than Chinchilla
-> Language models like Lambda and GPT3 are significantly undertrained. DeepMind proposed Chinchilla, a model which has similar performance to GPT3 with less than half the size (70B vs. 175B). Hence in 2023, significant performance gains will likely come from cleaner/larger datasets.

2) Generative Audio Tools Emerge and Will Attract 100K Developers
-> Audio generation has approached human levels. If enough data of your voice is available, the generated speech can even sound amazingly authentic (this is also true for Drake lyrics). Leaving the uncanny valley of awkward robot voices will make adoption surge.

3) NVIDIA Announces Strategic Partnership With AGI-Focused Company Organisation
-> Usage statistics in AI research show that NVIDIA's adoption is 20x-100x larger than that of competitors. If NVIDIA could pick or even help create a winning organization, this would cement their position.

4) Investment of >100M Into a Dedicated AI Alignment Organisation
-> Artists were not happy as model-generated artwork won an art competition in Colorado. Advances such as this will cause sentiments about AI safety to aggravate.

5) Proposal to Regulate AGI Labs Like Biosafety Labs Gets Backing By EU, GB, or US politicians
-> OpenAI scrambles to prevent ChatGPT from showing people how to build bombs. Responding to an outcry from artists, Stability AI has announced they will allow artists to opt-out such that their work is not used as training data. As negative impacts accumulate, regulation gains momentum.

6) GAFAM invests >$1B Into an AGI or Open-Source AI company Like OpenAI
-> The increase in the cost of model training has led to more and more innovation happening in industry. Regardless of the economy's choppy waters, big tech knows that staying ahead of the curve on machine learning will guarantee smooth sailing.

7) DeepMind Will Train 10B Parameter RL Model an Order of Magnitude Larger Than GATO
-> Currently, most machine learning models are very specialized. They can do one thing and one thing only. In 2022 DeepMind released GATO. This multi-modal model can, among other things, generate text, control a robot arm, and play video games. However, this line of research does not simply make models more versatile. The possibility of using sequence data for training increases the diversity and availability of training data.

If you made it all the way here, thank you! It was a privilege to make this for you.
At TheDecoding ⭕, I send out a thoughtful newsletter about ML research and the data economy once a week.
No Spam. No Nonsense. Click here to sign up!

3

Comments

You must log in or register to comment.

LesleyFair OP t1_j3c63gw wrote

Replying to manOnPavementWaving (#1,301,955)

Haha I see your point. According to VentureBeat, DeepMind has been making 60M in Profit this year from 1,06B in revenue. Google bought it for 500M in 2014. Though you are right in saying that GAFAM already pouts significant funding into ML, 1B would probably mean a significant step up.

1

keepthepace t1_j3e6gi8 wrote

Agreed on 1 and 2.

Not sure about 3: NVidia is dominant (maybe to the point of risking a monopoly litigation?) by providing to everyone. Making an "in" and "out" group carries little benefits and would push the out-group towards competition

4: I fail to see an "alignment organisation" that would provide 100M of value, either in tech or in reputation. It may emerge this year but I doubt there is one yet. Most valuable insights come from established AI shops

5: I doubt it. Artists are disregarded by politicians since forever. Copyright lobbyists have more power and they already outlawed generated images copyright

6: OpenAI is not an open source company. And this has already happened. Microsoft poured 1 billion into OpenAI

7: gosh I hope! Here is my own bold prediction: we will discover that multitask models require far less parameters for similar performance than language models and GATO successors will outperform similarly sized LLMs while simultaneously doing more tasks.

5

LesleyFair OP t1_j3hd415 wrote

Replying to keepthepace (#1,305,032)

>Copyright

Thank you for taking the time to write this up! I am super glad, if my writing spurs a discussion.

On 3) I think that their brand partnership would not necessarily need to an in- and out-group. If we take the example of Nike investing heavily into breaking the record of running a 2h marathon. They were not really creating two groups either. They were injecting themselves into the narrative of a community aspiring the do something great. This gave them loads of brand lift. If NVIDIA would have been part of the Alpha Fold project, it could have been the same. I think they would be happy if they could replace "Deepmind pushed the envelope of protein folding problem" with "DeepMind and NVIDIA...". Obviously, replace protein folding with any upcoming breakthrough.

On 4) I fully agree. I do also not see any alignment organization adding this much value. Not to throw any shade, but if we compared this to Web3 investment in 2022, this prediction from the report does not seem too far off. In the Web3 case heaps of money were poured onto crypto founders that did not know the difference between a mutual and an exchange-traded fund, but were certain that their immutable database would "obviously" disrupt legacy banking by the next week.

On 5) I can't argue with the low interest in artists' well-being in the broader political dialogue. I would think that a lot of the low-tier creative work will change significantly in the next years. Imho, this does not just include fine arts, but also things like copywriting, product photography, and design.

  1. Agreed. OpenAI is not open source. Thank you for pointing out the case of Microsoft. Seems like their prediction already came true. :)

  2. Hell yeah! That would be so cool!

​

Thanks again for taking the time and contributing your insights!

0

science-raven t1_j3segd3 wrote

5 Is tricky because its desireable to add custom training sets to giant models, so folk will add anything that was banned. And open source copies of ultra powerful tools will only be a few years away.

1