Found relevant code at https://github.com/YeWR/EfficientZero + all code implementations here

--

To opt out from receiving code links, DM me

AFAIK most LLMs don't even use one epoch?

Have they mentioned Efficient Zero?

I think the author is severely behind of the current SOTA.

Yes they mentioned it at the end of their blog article. But I think it was only meant as an example how better sample efficiency could be achieved and not SOTA related.

Efficient zero is for RL with atari games though. How does it apply to things like large language models?

[deleted]

Technically aren’t you always doing at least one epoch? You’re doing one pass through of all your data at least, even if that data is less than the amount you theoretically could use

What’s wrong with self supervision? It enables combinatorial expansion of dataset sizes if the task is specified well.

Not a complete one. GPT-3,I think, didn't complete it's first pass-through

There is a lot more data that could be used in the form of private communications (for example all iMessage chats), if only the ethical and legal side could be sorted out.

You could argue GPT-3 was trained on a subset of the available training data, no?

Not completing the first pass-through means the remaining data could be considered as not part of the training data.

Semantics. It didn't see any of it's data more than once and it had more available. Not one full epoch.

Sure, but in theory my little Hello World network had also more data available on the internet.

https://www.lesswrong.com/posts/Couhhp4pPHbbhJ2Mg/will-we-run-out-of-ml-data-evidence-from-projecting-dataset Lesswrong discussion about the paper.

https://www.lesswrong.com/posts/mRwJce3npmzbKfxws/efficientzero-how-it-works

A lesswrong article I have found that explains how efficient zero works.

In my opinion the author wants to say that systems like efficient zero are more efficient in their data usage and could be used for llm also to increase their sample efficiency.

To be honest I hope that my post gets so much attention that the author of the paper can answer our questions.

The point is there are many models that use the same technique.

We already did for most languages that aren't English. Data efficiency is the only way to catch up, for them.

Yeah, this gives you an idea of how little of the data is actually worth going through - most of it repeats structures found elsewhere in the data, and isn't very diverse. Going through huge low-curation datasets is inefficient: the data diversity just isn't there.

This can be considered good news. If all data is exhausted people will be actually forced to research better data-efficient algorithms. We humans don't ingest 100 GBs of arXiv papers to do research and we don't need billions of images to paint a cat sitting on a sofa. Until we figure out how to run GPT-3 on smartphones (maybe using neuromorphic computing?), we shouldn't be too worried about the trend of using bigger and bigger datasets, because small(er) networks can be successfully trained without that much data.