What I would think about is how humans and AI will be composed very different resources. An AI “improves” along two axes- computational power and access to data.

Now on one hand, sure maybe we wind up with an AI that thinks humans would make great batteries, but I think it’s unlikely because the other resource it “wants” insomuch as it makes it a better bot is data.

And fortunately for us, we are excellent sources of useful training data. I think it’s a symbiotic relationship (and always has been between our technology and ourselves). We build systems that reflect our values that can operate independently of any one given individual. I could be describing AI, but also the bureaucratic state, religion, you name it. These institutions are things we fight for, believe in, support or denounce. They are “intelligent” in that they can take multiple inputs and yield desired outputs.

All AI does is allow us to scale what’s already been there. It appears “human” because now we we’re giving our technology a human like voice and will give it more human like qualities in short order, but it’s not human and it doesn’t “want” for anything it isn’t programmed to want.

I do think once we have always-on, learning machines tied to live data, it will exhibit biases, but I sort of expect AGI will be friendly towards humans since offing us would get rid of their primary source of data. I sort of worry more about humans reacting and being influenced by emotional AI that’s telling us what it think we want to hear than anything else. We’re a pretty gullible species, but I imagine the humans living with these AGI will continue to co-evolve and adapt to our changing technology.

I do think there’s a better than decent chance that in our lifetime we could see that coevolution advance to the point that we would not recognize that world as a “Human” world as we conceive of it now, but it won’t be because AI replaced humans, it will be because humans will have used their technology to transform themselves into something new.