The only thing worrying about that is that the time period that supports the most humans isn't in the future, when we expand into the stars.

Some of that might not be too hard, self-awareness and agency can be represented as text. If you give Chat GPT a text adventure game it can respond as though it has agency and self-awareness. It will tell you what it wants to do, how it wants to do it, explain motivations, ect. Character. AI takes this to another level, where the AI bots actually "believe" they are those characters, and seem very aware and intelligent.

We could end up creating a system that acts sentient in every way and even argues convincingly that it is, but isn't.

I'd argue that the transformer architecture(the basis for large language models and image diffusion) is a form of general intelligence, although it doesn't technically meet the requirements to be called AGI yet. It's able to take any input and output a result that, while not better than a human expert, exceeds the quality of the average human most of the time.

ChatGPT can translate, summarize, paraphrase, program, write poetry, conduct therapy, debate, plan, create, and speculate. Any system that can do all of these things can reasonably be said to be a step on the path to general intelligence.

Moreover, we aren't anywhere near the limits of transformer architecture. We can make them multi-modal, inputting and outputting every type of data, embodied, by giving them control of and input from robotic systems, goal directed, integrated with the internet, real time, and potentially much more intelligent simply by improving algorithms, efficiency, network size, and data.

Given how many ways we still have left to make them better it's not unreasonable to think systems like this might lead to AGI.

> A neural network learns a representation from the data. It literally scans ur work.

The neural network best know for this ability is the human brain. Aspiring artists and musicians scan many works during training which alter the parameters(synapses) of the neural net, allowing it to better recreate the training data or extrapolate from that data to create new and unique output. Sometimes, the parameters in the neural net are configured so precisely that it becomes possible for it to re-create copyrighted works with high precision. The ability to do this does not constitute copyright infringement. Copyright infringement only occurs if the recreation isn't properly attributed.

Just wait until these models are multi modal and can simultaneously process video, text, and robitcs data. Deep mind already created an early prototype called GATO.