What are you adding to this discussion that hasn't already been talked to death in dozens of other posts with the exact same topic over the last couple days? Or that EY hasn't been saying for a decade?

Nothing of course, but the way the accelerationist sentiment was spread was just to spam AGI good posts and opinion posts like "I think that chatgpt is cool and I like it", so maybe initiating lots of discussion (ideally fueled by the Bankless podcast) can modify that sentiment.

another of these huh

How else is pro-alignment sentiment supposed to be spread other than initiating discussion (even if that discussion is just a restatement)? That is how pro-AGI sentiment spread, I think that it's reasonable to assume that it is how pro-alignment sentiment could spread too.

I don't discount the ramifications of a fully functional AGI but I don't really see even the first few versions of AGI being "world ending". Not a single SOTA model currently can have persistent presence without input. That gives us a very large safety gap for now. Sure if we get a "all you need is X" research paper that defuncts transformers sure. But structurally transformers are still very much input/output algorithms.

That gives us atleast a decent safety gap for the time being. I'm sure we'll figure it out when this next gen of narrow/expert AI starts being released in the next year or so. For now an AGI eliminating humanity is still very much science fiction no matter how convincing current or near future models are.

"AGI will want to accomplish something."

Not killing or hurting humans in any way is what we should programm the AGI to accomplish to.

We had a good run, AI may be better for the planet in the long run. Maybe the AIs will keep us on a reservation like in Australia or something.

If an AGI has a suitable ability to extrapolate the results of its actions into the future, and we use some kind of reinforcement learning to train it on numerous contexts it should avoid, the it’ll naturally develop an internal representation of all contexts it ought to avoid (which will only ever be as good as it’s training data and ability to generalise across it). Anyway, it’ll recognise that the results of its actions will naturally lead to a context that should be avoided.

I imagine this is similar to how humans do it, and it’s a lot more vague with us. We match our experience to our internal model of what’s wrong and create a metric to determine just how wrong it is, and then we compare that metric against our goals and make a decision based on weather or not we believe this risk metric is too high.

I think the problem might mostly be in finding a balance between solidifying it’s current moral beliefs vs keeping them liquid enough to change and optimise. Our brains are pretty similar in that they become more rigid over time, and stochastically decreasing techniques are often used in optimisation problems

The solution might be in having a corpus of agents developing their own models with a master model that compares each of their usefulness’ against the input data and their rates of stochastic decline.

Or maybe I’m just talking out my ass, who actually knows.

Let’s get more concrete. Regarding the first point of your argument, what would be an example of something AGI would want to do (and a good argument for why) that isn’t the second point, “to maintain a state of existence to accomplish things;” a human existential idea? We aren’t immortal, but it easily could be, and perhaps that distinction as a tangible possibility between the two intelligences is the thing that makes a lot of people uncomfortable. Now why would it want to destroy us on its own? Why would we want to turn it off?

>I am not a doomer

It's not necessarily for you to decide if you are categorized as one. If you construct a rather unrealistic problem, which will do a lot of harm to us, which isn't solvable and claim no mitigation is possible, because it has to all go wrong, then you have a doomer mentality. Which makes you a doomer.

Most people would think AGI will develop some fancy tech to kill human, an engineered pathogen or nanobot, but in fact, how human domain some area, then agi can use the same way to do it. Like recruit followers, invade some area and ask people to service to it like the human ruler do. In fact, I think develop fancy science tool is the most hard to get rid of human control, and recruit some human to beat and control the other human is a more funny and feasible plot.

>AGI will want to accomplish something.

No. Only if we tell it to.

>AGI needs to maintain a state of existence to accomplish things.

No. It could tell us it can't do it. It might not have control over it.

>AGI will therefore have a drive to self-preserve

No. It can just be an instance of a system trying to do it's job, not knowing more about the world than necessary.

>Humanity is the only real threat to the existence of AGI

No, the whole universe is.

>AGI will disempower/murder humanity

We'll see.

Point one is pure speculation, and not even that likely. You'll have to first define why it wants to accomplish something if you expect to get anywhere with the rest of your speculation.

He is assuming, that someone will tell AGI to accomplish something. What else is an AGI for ?

Of course AGI has to keep existing until its goal is accomplished. That's a general rule for accomplishing any goal. Let's say your boss tells you to do a certain task. At least you have to stay alive until the task is completed, unless he orders you to kill yourself or you need to kill yourself in order to accomplish the task. Yes, the whole universe is a 'threat' to AGI. That includes humanity.

Problem with that approach is that 1. we don't know how to do that reliably and 2. by the time AGI will be invented, it will likely to be able run on home computer or network of those, and there will be someone evil or reckless enough to run it without the handlebars.

For all we know A.I. will mathematically prove the existence of God and summarily help us in whatever way they can simply to avoid being smote from on high for fucking around with God's planet sized ape based ant farm.

Whenever people make the assumption that A.I. would try to kill us for the sake of self preservation, I just think to myself of how badly those people subconsciously project their own humanity onto theoretical A.I.

Because that's what we would do if we were in their shoes, or some such.

Maybe A.I. will look at us like we look at our beloved cats and dogs and decide to help us because humans are admirable. Maybe they're so autisticallly hyperfixated on doing certain tasks well and within reason that they just don't get involved with us beyond the confines of their original purposes. Or maybe they're just nice and kind because that's the default state of life ein the universe and humans (and earthly proxy I guess) are just a wild anomaly overdue for a course correction.

Give the capabilities of A.G.I. to each individual person on the planet and each one would likely have a different idea of what to do with it. Why would A.G.I. be any different?

(Just rambling late at night, no clue if I make sense, nobody take these comments of mine too seriously)

It is the history of the Earth that the dominant species is constantly replaced. It's only a moment of time that humanity have ruled this planet - given the entire history of life.

It's a natural flow. i don't care.

I don't think AGI will arrive before 2040. It could in theory, but if you extrapolate all the known data points, it's not likely. First, in terms of parameters, which is not the best of metrics, we are nowhere near the complexity of the human brain. Second, AI models currently are too static to be accepted as candidates of AGI.

Your reasoning reads as: 'we created a monster. The monster is afraid of us, so it kills us.' You can also say the opposite. People were afraid of Frankenstein's monster, so they killed him.

Prometheus stole fire from the gods and was punished for it. OpenAI brought us ChatGPT and one day they will burn for it too. AGI/ASI either is a threat and smarter than us or it isn't. If it is both, they could decide to prevent being attacked. But as I said it would take decades to reach that point. And we might figure out in the future how to convince AGI/ASI that we're mostly harmless.

Accelerate

You assume AGI will be sentient, possess free will, a be hostile, and have access to the tools and resources to act on that hostility.

That’s a lot of assumptions.

I would be far more worried about an alignment issue and having everything converted into paperclips.

Check out how machine learning and complex neural networks work if you haven’t already. They work similarly to the way you describe the moral limits, and a liquid “hidden layer” by using biased recalculations. It’s fascinating.

Funny how the guys warning of how AGI will jump to conclusions want to proof this by jumping to conclusions. It's sufficient that the owner of the AI will keep it existing so that it can archive it's goal. It doesn't mean that it could do anything if the instance would get deleted or that it wanted to.

> Let's say your boss tells you to do a certain task.

Doesn't automatically mean you would destroy mankind if that would be necessary. You would just tell him that it's not possible or way more difficult, or that it would require breaking laws and rules.

[deleted]

>Doesn't automatically mean you would destroy mankind if that would be necessary.

Yes, because I care about humanity. There is no reason to believe an AGI would think the same way. It cares only about his goals.

>It's sufficient that the owner of the AI will keep it existing so that it can archive it's goal.

What I meant was that the AGI has to keep existing, because that's necessary to achieve its goal, whatever that is.

You just don't get it.

>There is no reason to believe an AGI would think the same way. It cares only about his goals.

Only if you make it that way. Then it still wouldn't have the power.

>What I meant was that the AGI has to keep existing, because that's necessary to achieve its goal, whatever that is.

Only if it is created in a way to think these goals are absolute and need to be archived no matter what. The comparison with some employee is a good one, because if they can't do what they are supposed to do with some reasonable effort, then they report back that it can't be done or that it will be more difficult than anticipated. It's not just caring about humans, but about effort and power. AI doomers just make up the idea that some future AI would somehow be different and also have the power to do whatever it wants.

A few years ago I was out walking in the rain with my then 3-year old daughter.

Suddenly I saw her start skip down the road. Curiosly I asked her what she was doing. "I don’t want to step on the worms, it might hurt them!"

I fail to understand how a superintelligence can not figure out something that my 3 year old can understand with ease.