I'm not saying that it's using a terminal behind the scenes, I'm saying the data used to train this was likely generated by using an execution environment. There are serious NLP researchers believing this as well: https://twitter.com/yoavgo/status/1599886211656491008

Calm down?

It's not as if I'm the only one claiming that. https://twitter.com/yoavgo/status/1599886211656491008

Yea or you know you could just make up input, let it execute the code and get the output to create your training data...

There's still not enough data. I believe it must have had access to some environment in which it could have executed commands. Compare how well ChatGPT performs on computing stuff and how badly it performs on other topics. E.g. is there significantly more data available on the web on just some specific kind of shell command (note that it generates the correct shell output for any kind of input) compared to say blog posts on real analysis? If you try to query chatgpt for its understanding of real analysis definitions it will abysmally fail, but there should be way more text available on that topic than some random shell command and definitely not enough data for any kind of input. I really don't believe that current generation language models are capable of learning the semantics of terminal commands.

Because it's making almost no errors on basically any kind of shell input, there just isn't enough data on the web to allow current language models to generate such accurate output imo.

Does anybody have thoughts on how they might possibly have created data for this? I'm completely stumped about the knowledge it has of this kind of tech knowledge and don't think there's even remotely sufficient data on the web that would allow it to generate this kind of content. Did they use some self-learning environment in a terminal?!