In many ways it's been the same since 2010. We could talk, take photos, load web pages, use maps, set alarms and play games back then too, we even had Uber and AirBnb. Now the screens are a bit larger and the experience more polished.

I was expecting something more revolutionary - the phone is a pack of sensors, it has sight, hearing, touch, orientation, radio and many other sensors in the same package. But the amazing new applications didn't appear, except Pokemon Go?

Not even people working in the field have a good idea about 3 years ahead. Ten or twenty years ahead is just sci-fi.

Probably having to verify AI takes 50% of the time do do it manually, so the relative advantage is smaller.

But another advantage of teaming human+AI is that AI can be calibrated and ensure a baseline of quality. Humans might have higher variance, have a bad day, be tired, inattentive. So it is useful to increase consistency, not just volume.

> Humans are surprisingly adaptable, things that would have blown my mind even 5 years ago I take for granted now.

No way automation can keep up, we'll take everything for granted and still have to work to bring it to the next level.

I think raw materials might not be needed anymore if we can recycle everything. At some point we will have to treat industrial materials like biology, they will have an ecosystem of their own. Of course if we want to expand we need new materials and space.

We can have a model trained on a large video dataset, and then fine-tuned for various tasks like GPT3.

Using YouTube as training data we'd get video + audio which decompose in image, movement, body pose, intonation, text transcript, metadata all in parallel. This dataset could dwarf the text datasets we have now, and it will have lots of information that doesn't get captured in text, such as physical movements for achieving a specific task.

I think the OP was almost right. The multi-modal AI will be a good base for the next step, but it needs instruction tuning and RLHF. Just pre-training is not enough.

One immediate application I see - automating desktop activities. After watching many hours of screen casting from YT, the model will learn how to use apps and solve tasks at first sight like GPT-3.5, but not limited to just text.

A recent approach saves past experience data and loads it back for in-context-learning. The model itself can be task generic. So it learns by collecting new data.

But it is still preferable to train on synthetic images than on the original works, don't you agree?

When the artist refuses to allow their images be used for training AI models, or it is impossible to get permission for other reasons such as not knowing the correct contact information, if the AI uses variations it won't learn to imitate the originals closely. Variations should be OK because they have no copyright, as the courts decided. Seems like a better compromise than either indiscriminate training or making AI impossible to train.

> Many AI researchers believe that solving the language translation problem is the closest thing to producing Artificial General Intelligence (AGI).

I call bullshit on this. Show me one researcher or paper claiming this. MT is not the closest to AGI, we have been doing ok in MT even before GPT-3. The most advanced AI we have now can solve problems and handles general chat. MT is a much simpler, basic task.

You don't need to do all that. Train a model on your data without destroying your body, just what can be logged from outside. It will be enough. chatGPT can enter a persona even with just a handful of hints. I think the AI of the future will be able to replicate any personality without fine-tuning.

You still need materials, it doesn't just create out of thin air.

Apparently 13B models feel comparable with chatGPT on a 3090 card with 24gb vram (source). So it would be fast!

But they documented how to make it by sharing paper, code, dataset and hyper-parameters. So when Stability wants to replicate, it will be 10x cheaper. And they showed a small model can be surprisingly good, that means it is tempting for many to replicate it.

The cost of running inference on GPT-3 was a huge moat that is going away. I expect this year we will be able to run a chatGPT level model on a single GPU, so we get cheap to run, private, open and commercial AI soon. We can use it for ourselves, we can make projects with it.

> Could they get it to run on a phone one day? It would be awesome.

It would be Google's worst nightmare. Such a model could sit between the user and their ad-infested pages, extracting just the useful bits of information and ignoring the ads.

Using the internet without your local AI bot would be like walking outside without a mask during COVID waves. It's not just the ads and spam, but also the AIs used by various companies that don't have your best interest at heart. I expect all web browsers to have a LLM inside. Or maybe the operating systems.

It will be like "my lawyer will be talking to your lawyer" - but with AIs. You can't expose raw humans to external AI assault, humans need protection-AI just like we need an immune system to protect from viruses.

Wouldn't it be better if people could donate their interactions with chatGPT, BingChat and other models? Make a scraping extension, it should collect chat logs and anonymise them. Then you got a diverse distribution of real life tasks.

I suspect this is the reason OpenAI and Bing offered their models for free to the public - to find the real distribution of tasks people want to solve with AI bots.

Words in language are both observations and actions. So language modelling is also a kind of supervised policy learning?

So... Self Supervised Learning is Unsupervised & Supervised & Reinforcement Learning.

But the text coming from a human should be considered "manually" labelled, right?

Does it do only one round of retrieval?

> The grounding problem is a red herring because thoughts are events rather than physical objects.

What? If they are events they are physical as well. The problem with grounding is that LLMs don't get much of it. They a grounded in problem solving and code generation. But humans are in the real world, we get more feedback than a LLM.

So LLMs with real world presence would be more grounded and behave more like us. LLMs now are like dreaming people, but it is not their fault. We need to give them legs, hands and eyes so they wake up to the real world.

Well, you can't. Because it is really hard to extract any verbatim replications of training data from chatGPT. You need to put a considerable portion from the work as prompt, to put the model in the right place, and then sample your way ahead. Doesn't work for most stuff, like 99%.

If you go down to individual words or characters, everything is reused. If you go up, usually a random 10 word snippet is nowhere else in the internet. But boilerplate and basic things might be replicated in all shapes and forms.

> Honestly, kinda selfish. We'll all benefit from these powerful new tools and I don't appreciate you trying to hamper them.

They took their little pebble from the beach back home, that'll show them.

Diffusion models take about 1 byte of information from each training image - 5B images, 5Gb. So much less than a thumbnail.

So, all you need to do is use a source image you made yourself, and then the whole image belongs to you. Nobody can extract just the AI contribution.

That is glossing over the fact that nobody can actually demonstrate which of the source images were responsible for this derivation. Will you choose, or shall we pick one or ten at random, or just the closest by similarity score? We have no way of assigning merit.

And I suspect you think everything in a copyrighted work is protected by copyright. But it's not true. Only expression is protected, not the ideas. You can borrow ideas if you don't copy the exact expression. AI only learned basic concepts, it builds new images from first principles. By learning only ideas and not exact expression they can have free hand.

If you want to be 100% sure, then it is possible to train an AI with variations of the original works generated by another AI - this way only the ideas are transmitted and the new model has never seen copyrighted works, so it can never replicate them even by mistake.