Comments

You must log in or register to comment.

Kolinnor t1_j9gpzre wrote

He does have very good points, and he's very interesting, with brilliant ideas about alignment.

Overall, all the Lesswrong philosophy is a little too "meta" and confusing if you've not studied the jargon, so I'm a bit annoyed sometimes when I read it and realize, in the end, that they could have said the same ideas with less sophisticated phrasing.

Although, while I don't agree with the conclusions he reaches (and he reaches them with too much confidence to my taste), I've learned quite a number of things about alignment reading him. Definitely a must read for singularity and alignment even if you don't agree with him.

63

FestiveHydra235 t1_j9kdn6r wrote

Thank you for pointing out the jargon. His blog is incredibly difficult to read. And it’s not difficult to read because he’s so smart, it’s difficult because it’s such convoluted writing

4

CellWithoutCulture t1_j9noid8 wrote

Yeah the jargon and meta rambling is so annoying. It's like their first priority is to show off their brains, and their second priority is to align AGI. Now they are almost finished showing of their brains, so watch out AGI.

Sometime they behave in a silly fashion. Greek philosopher's had excellent logic and deduced all kinds of wrong things. These guys seem similar at times, trying to deduce everything with philosophy and mega brains. .

IMO they are at their best when it's said in short form and it's grounded by empirical data.

There is also a lesswrong podcast or two that will read out some of the longer stuff.

4

sticky_symbols t1_j9gwa67 wrote

He's the direct father of the whole AGI safety field. I got interested after reading an article by him in maybe 2004. Bostrom credits him with many of the ideas in Superintelligence, including the core logic about alignment being necessary for human survival.

Now he's among the least optimistic. And he's not necessarily wrong.

He could be a little nicer and more optimistic about others' intelligence.

46

GuyWithLag t1_j9j9l56 wrote

>He could be a little nicer and more optimistic about others' intelligence.

Apologies for sounding flippant, but the whole political situation since '15 or so has shown that he's too optimistic himself...

5

diabeetis t1_j9gob12 wrote

Very smart guy, huge douche, makes good doomer points regarding x risk

32

sticky_symbols t1_j9gvp91 wrote

I think slightly douchy is more fair. I've read a ton of his stuff and only a subset is offensive to anyone. But yeah, he's not as considerate as he probably should be.

11

FirstOrderCat t1_j9hxjzo wrote

I argued with him on hacker news, and he is very reactive when reading something he doesn't like.

9

sticky_symbols t1_j9i0yw8 wrote

Well, he's the father of a whole field that might determine the future of humanity. It would be tough to keep it cool the 1009th time you've seen the same poorly thought out dismissal of the whole thing. If I were in his shoes I might be even crankier.

9

FirstOrderCat t1_j9ifrw4 wrote

I don't know much about his practical achievements in this area.

12

sticky_symbols t1_j9itrli wrote

Founding a field is a bit of a rare thing

5

FirstOrderCat t1_j9j9qlg wrote

which field? ai danger awareness? It was in the terminator movie.

19

sticky_symbols t1_j9m3uus wrote

Good point, but those didn't convince anyone to take it seriously because they didn't have compelling arguments. Yudkowsky did.

3

FirstOrderCat t1_j9m6fj2 wrote

>but those didn't convince anyone to take it seriously

Lol, I totally got the idea that rogue robot can start killing humans long before I learn about Yudkowsky existance.

> Yudkowsky did.

could you support your hand-waving by any verifiable evidence?

4

sticky_symbols t1_j9m6t5d wrote

Well, I'm now a professional in the field of AGI safety. Not sure how you can document influence. I'd say most of my colleagues would agree with that. Not that it wouldn't have happened without him but might've taken many more years to ramp up the same amount.

2

FirstOrderCat t1_j9m8bhd wrote

> Not that it wouldn't have happened without him but might've taken many more years to ramp up the same amount.

happened what exactly? what are the material results of his research?

I think Azimov's with his rules produces earlier and much stronger impact.

> I'm now a professional in the field of AGI safety

Lol, you adding AGI makes my bs detector beeping extremely loud.

Which AGI exactly you are testing for safety?

2

sticky_symbols t1_j9m8yn3 wrote

Asimov's rules don't work, and many of the stories were actually about that. But they also don't include civilization ending mistakes. The movie I Robot actually did a great job updating that premise, I think.

One counterintuitive thing is that people in the field of AI are way harder to convince than civilians. They have a vested interest in research moving ahead full speed.

As for your bs detector, I'm don't know what to say. And I'm not linking this account to my real identity. You can believe me or not.

If you're skeptical that such a field exists, you can look at the Alignment Forum as the principle place that we publish.

1

FirstOrderCat t1_j9ma8lr wrote

> Asimov's rules don't work

you jump to another topic. Initial discussion was that Azimov rules brought much more awareness, and you can't point on similar material results from Yudkovsky.

1

sticky_symbols t1_j9mbzia wrote

Sorry; my implication was that Asimov introduced the topic but wasn't particularly compelling. Yudkowsky created the first institute and garnered the first funding. But of course credit should be broadly shared.

1

Melveron t1_j9gspdh wrote

He’s right about the dangers of AGI but too pessimistic about our ability to address them. He always moans about how leaders of major AI research firms are dismissive of his concerns which is not true. It’s not 2015 anymore, lots of good work is being done in alignment. Will it be enough? Maybe, maybe not, but the outcome is far from decided.

21

Present_Finance8707 t1_j9l2oq5 wrote

Any company serious about alignment would not pursue capabilities research, full stop. OpenAI is perhaps the most dangerous company on earth.

2

NoidoDev t1_j9ht1ur wrote

He uses thought experiments and unreasonable scenarios to get attention. If this is for commercial reasons, or his mentality, that I don't know. If it would be clear that these are just abstract thought experiments, it wouldn't be a problem, but he acts like these are real threats. He, and other similar "researchers" are building their scenarios on claims like

- AGI or ASI is going to be one algorithm or network, so no insights, no filters possible, ...

- someone will give it the power to do things, or it will seek these powers on it's own

- it will do things without asking and simulating things first, or it just doesn't care about us

- the first one build will be a runaway case

- it will seek and have the power to change materials (nanobots)

- there will be no narrow AI(s) around to constrain or stop it

- no one will have run security tests using more narrow AIs, for example on computer network security

- he never explains why he beliefs these things, at least he's not upfront in his videos about it, just abstract and unrealistic scenarios

This is the typical construction of someone who wants something to be true: Doomer mindset or BS for profit / job security. If he had more influence, then he would most likely be a danger. His wishes for more control of the technology show that. He would stop progress and especially proliferation of the technology. I'm very glad he failed. In some time we'll might have decentralized training, so big GPU farms won't be absolutely necessary. Then it's gonna be even more over than it is already.

Edit: Typo (I'm not a native English speaker)

18

Present_Finance8707 t1_j9l2vjp wrote

You really really really don’t understand the alignment problem. You really don’t know the field if you’re trying to understand by watching videos of Eliezer and not his writing. What a joke

2

NoidoDev t1_j9lptw1 wrote

His videos are where he can make his case. It's the introduction. If he and others fail at making the case, then you don't get to blame the audience. Of course I'm looking at the abstract first, to see if it's worth looking into. My judgement is always: No.

4

Present_Finance8707 t1_j9my8wl wrote

Like I said you really really don’t understand alignment. Imagine thinking a “filter” is what we need to align AIs or completely lacking understanding of instrumental convergence. You don’t understand even the utter basics but think you know enough to dismiss Eliezers arguments out of hand??? Thankfully I think you’re also too stupid to contribute meaningfully to capabilities research so thanks for that.

3

NoidoDev t1_ja17g56 wrote

> Like I said you really really don’t understand alignment.

What I don't understand is how you believe that you can deduct this from one or a very few comments. But I can just claim you just don't understand my comments, so you would first have to proof that you do understand them. So now spend the next few hours thinking about and answer, then I might or might not answer to that, and that answer might or might not take your arguments into account instead of just being dismissive. See ya.

Edit: Word forgotten

1

Present_Finance8707 t1_ja1em06 wrote

You’re literally saying “put a filter on the Ai”. That’s like “just unplug it lolz” levels of dumb. Give me a break.

1

obfuscate555 t1_ja0z055 wrote

Ok, if you do understand so we'll, then explain it.

0

NoidoDev t1_ja17niw wrote

>Thankfully I think you’re also too stupid to contribute meaningfully

Problem is, I don't need to. You doomers would need to convince people that we should slow down or stop progress. But we won't.

0

Present_Finance8707 t1_ja1f835 wrote

You’re mentally ill. Please remember this conversation when foom starts and you start dissolving into grey goo. Absolute degenerate.

0

tomorrow_today_yes t1_j9j0w44 wrote

He is a bit like Paul Ehrlich, the population doomster of the 1970’s. Ehrlich is very smart and a good arguer but not someone who can accept criticism easily. Ehrlich, like Yudkowsky, reached his conclusions based on simply extrapolating trends. Ehrlich refused to acknowledge that his simple extrapolation was made invalid by green revolution and improved technology, the possibility of this had been pointed out to him, notably by Julian Simon, but Ehrlich dismissed this as hopium. I really hope that we can ridicule Yudkowsky in the future for the same reasons, not because I don’t like him but because that would mean the AI alignment problem was solved.

15

Present_Finance8707 t1_j9l2f4b wrote

If you think Eliezer reached his conclusions by “extrapolating trends” you don’t have a single clue about his views.

3

Molnan t1_j9hoxnx wrote

I respect the fact that he started thinking seriously about AI security early on, but he got stuck in a dead end paradigm of AI security ("friendly AI") where we assume the AI will be all-powerful and our only hope is for its goals to be perfectly aligned with ours. As the flaws of this approach became increasingly apparent, instead of considering other AI security paradigms he just became increasingly pessimistic. He also seems to have a pretty big ego, which prevents him from reconsidering.

13

Present_Finance8707 t1_j9l32gz wrote

Your link completely fails to align an AGI. You aren’t offering anything interesting here

1

Molnan t1_j9l3q5s wrote

There's no need to "align an AGI". That's in fact the whole point, and you missed it.

2

Present_Finance8707 t1_j9l5tzs wrote

Two problems. It doesn’t work and the current models are already way down the Agent line and there’s no going back. Yawn. https://gwern.net/tool-ai

3

Molnan t1_j9lag3a wrote

From skimming through your blog post it's quite clear you really need to read and try to understand Drexler's FHI report. For instance, your claims about tool AIs Vs agent AIs are irrelevant because the idea is not to avoid agent AIs, only the "friendly AI" paradigm. Also, you'd know that Drexler's paradigm is a natural extension to current best practices in AI design, not just for some abstract big-picture AI security but also for expediency in development of AI capabilities and more mundane safety concerns. So it's the exact opposite of what you claim: the "friendly AI" paradigm is the alien, unwelcome newcomer that wants to turn the AI research community on its head for dubious reasons, while Drexler tells them to keep doing what they are doing.

2

Present_Finance8707 t1_j9myqs2 wrote

If you don’t even know who Gwern is I can’t really take you seriously about alignment. You can’t possibly have a deep understanding of the various arguments in play.

2

CellWithoutCulture t1_j9norym wrote

And if you don't know who Drexler is...

I know who all these people are, yet I don't know anything lol

4

Molnan t1_j9n7sql wrote

You don't have to take *me* seriously, but you should certainly read an FHI technical report before you take the liberty to yawn at it.

I don't keep up with every blogger who writes about AI alignment (which you stubbornly keep assuming to be the crux of all AI security) but I've been reading Eliezer and Nick Bostrom for long enough to know that their approach can't work, and now Eliezer seems to agree with that conclusion.

3

Present_Finance8707 t1_j9q8x1g wrote

Eliezers actual conclusion is that no current approach can work and there are none on the horizon that can.

1

Molnan t1_j9qd93i wrote

Yes, which implies he doesn't believe his approach would work, like I said.

1

Present_Finance8707 t1_j9qfavu wrote

His arguments don’t hold up. For one thing we already have powerful generalist agents. Gato is one and it’s clear that advanced LLMs can do all sorts of tasks they weren’t trained to. Prediction of next token seems as benign and narrow as it can get but if you don’t think a LLM can become dangerous you aren’t thinking hard enough. CAIS also assumes people won’t build generalist agents to start with but that cat is well out of the bag. Narrow agents can also become dangerous on their own because of instrumental convergence but even if you restrict building only weak narrow agents/services the profit incentive for building general agents will be too strong since they will likely outperform narrow ones.

1

Molnan t1_ja5zobe wrote

You say:

​

>CAIS also assumes people won’t build generalist agents to start with.

​

No, it doesn't. See, for instance, section 7: "Training agents in human-like environments can provide useful, bounded services":

​

>Training agents on ill-defined human tasks may seem to be in conflict with developing distinct services provided by agents with bounded goals. Perceptions of conflict, however, seem rooted in anthropomorphic intuitions regarding connections between human-like skills and human-like goal structures, and more fundamentally, between learning and competence. These considerations are important to untangle because human-like training is arguably necessary to the achievement of important goals in AI research and applications, including adaptive physical competencies and perhaps general intelligence itself. Although performing safely-bounded tasks by applying skills learned through loosely-supervised exploration appears tractable, human-like world-oriented learning nonetheless brings unique risks.

​

You say:

​

>if you don’t think a LLM can become dangerous you aren’t thinking hard enough.

​

Any AI can be dangerous depending on factors like its training data, architecture and usage context. That said, LLM as currently understood have a well defined way to produce and compare next token candidates, and no intrinsic tendency to improve on this routine by gathering computing resources or any similar instrumental goals, and simply adding more computing power and training data doesn't change that.

Gato and similar systems are interesting but at the end of the day, the architecture behind useful real-world AIs like Tesla's Autopilot is more suggestive of CAIS than of Gato, and flexibility, adaptability and repurposing are achieved through good old abstraction and decoupling of subsystems.

The advantages of generalist agents are derived from transfer learning. But this is no panacea, for instance, in the Gato paper they admit it didn't offer much advantage when it comes to playing Atari games, and it has obvious costs and drawbacks. For one, the training process will tend to be longer, and when something goes wrong you may need to start over from scratch.

And I must say, if I'm trusting an AI to drive my car, I'd actually prefer it if this AI's training data did NOT include videogames like GTA or movies like, say, Death Proof or Christine. In general, for many potential applications it's reassuring to know that the AI simply doesn't know how do certain things, and that's a competitive advantage in terms of popularity and adoption, regardless of performance.

​

You say:

>Narrow agents can also become dangerous on their own because of instrumental convergence

​

Yes, under some circumstances, and conversely, generalist agents can be safe as long as this pesky instrumental convergence and other dangerous traits are avoided.

There's a lot more to CAIS than "narrow good, generalist bad". In fact, many of Drexler's most compelling arguments have nothing to do with specialist Vs generalist AI. For instance, see section 6: "A system of AI services is not equivalent to a utility maximizing agent", or section 25: "Optimized advice need not be optimized to induce its acceptance".

0

Icanteven______ t1_j9hm0at wrote

Harry Potter and the Methods of Rationality was great

11

DandalfTheWhite t1_j9huwx9 wrote

Probably one of the most controversial fanfics out there. People either love it or hate it. Not much middle ground.

6

spryes t1_j9h2m5j wrote

It seems that he confidently believes we will all die once AGI/ASI is reached, but I don't see why *all* humans dying is more likely than only *some*. Why is it guaranteed it would cause catastrophic destruction rather than only minor destruction, especially since something can't be infinitely powerful.

For example, an analogy is that ASI::humans will be equivalent to humans::ants, and yet while we don't care if we kill ants to achieve our goals, we don't specifically go out of our way to kill them. Many ants have died due to us, but a ton are still alive. I think this is the most likely scenario once ASI becomes uncontrollable.

I also think it will leave our planet/solar system and pursue its goals elsewhere as Earth may not be adequate for it to continue, effectively just leaving us behind, and that humans as material won't be as effective as some other material it wants to use in space somewhere.

9

j4nds4 t1_j9hxcbb wrote

I'm not on board with everything he says, but to consider his probable retorts:

>It seems that he confidently believes we will all die once AGI/ASI is reached, but I don't see why *all* humans dying is more likely than only *some*.

Why would it opt to kill only some knowing that that would increase the probability of humans trying to turn it off or change its function? Concretely eliminating the risk requires concretely eliminating the only other sentient and conscious species that might be motivated to hamper it.

>Why is it guaranteed it would cause catastrophic destruction rather than only minor destruction, especially since something can't be infinitely powerful.

The idea is that a smart enough AGI (aka smarter than us at everything) won't cause destruction unless it's for a specific reason, and preventing risk of reduction of its reward probability (like us turning it off) would motivate it to eliminate us. And it doesn't need to be infinitely powerful, just better than just at everything relevant.

>For example, an analogy is that ASI::humans will be equivalent to humans::ants, and yet while we don't care if we kill ants to achieve our goals, we don't specifically go out of our way to kill them. Many ants have died due to us, but a ton are still alive. I think this is the most likely scenario once ASI becomes uncontrollable.

That's because there's no consideration of ants making a coordinated and conscious effort to disable us.

>I also think it will leave our planet/solar system and pursue its goals elsewhere as Earth may not be adequate for it to continue, effectively just leaving us behind, and that humans as material won't be as effective as some other material it wants to use in space somewhere.

If it's sufficiently smart and sufficiently capable and sufficiently calculated then it would presume that leaving us alive increases the risk of another sentient and conscious entity challenging its growth and reward potential.

It all comes down to the reward function - all ML programs are built with goals and rewards, and we need to be sure that, once it is exceedingly capable and calculating and generalized, its reward is sufficiently defined such that humans will never be seen as a threat to that reward and that actions taken toward that reward will never directly or indirectly affect us in a way that we would disapprove of. All of that has to be figured out before it reaches the point of superior intelligence; once it's at that point, there is otherwise no hope of ever having a say in its actions. We can't predict everything it could do because we are by definition dumber than it and will be by an increasingly unfathomable margin. We can't predict every loophole, every side effect, every glitch, every escape route it could conceive of or exploit; therefore to know with adequate confidence that we have all our ducks in a row before it takes off is literally impossible. The best we can do is continue to try to figure out and decrease the odds of those cracks ever forming or being problematic; and given how easily ML programs (most recently chatGPT and Bing Chat) engage in the kind of behavior their creators are actively trying to avoid, we are unambiguously very, very bad at that.

9

-emanresUesoohC- t1_j9hckqn wrote

Other ant colonies don’t seek us out for retribution when we destroy a colony. If they did I think we’d be a lot more thorough. We probably couldn’t eliminate them completely, because we’re not a super intelligence.

6

blueSGL t1_j9hu8f1 wrote

A developer does not give two shits about any nature or wildlife in the way of the worksite unless 1. it directly impacts it, or 2. forced to via regulation. (agreeably this could be seen as a subset of 1)

What makes you think ASI would be any different?

5

loopuleasa t1_j9hdhuo wrote

if you want a less douchy version of the guy, also on the topic of AI alignment, try Robert SK Miles

9

beachmike t1_j9k6jo7 wrote

Eliezer Yudkowsky is not an AI researcher, engineer, or creator. He's a writer on the subject of AI that gives commentary and opinions on the subject. Anyone can write about AGI "safety." He hasn't contributed anything practical to the field, or anything to advance AI.

8

Gimbloy t1_j9hi9qu wrote

I think he’s probably too far on the pessimist side, but we need people presenting both extremes. The truth is usually somewhere in the middle.

7

turnip_burrito t1_j9guttl wrote

He has good points, but during that interview that's posted around here, he takes too much time to explain them. It feels like he says something in 2 minutes that could be compressed down to 20 seconds without any loss of information. I get that it's an involved topic and difficult to explain on the spot, but still.

That said, I don't necessarily agree with the "we're doomed" conclusion.

6

Present_Finance8707 t1_j9l3o1u wrote

This thread is completely full of hopium and laymen saying “I disagree with his Views because their implications make me uncomfortable and here are 5 bad reasons that AI won’t kill us that people already squashed as useful 30 years ago.”

5

blueSGL t1_j9h0drv wrote

Should be listened to, and if anyone thinks he's wrong I'd say they could make a lot of money selling their working alignment tech (which you'd need in order to prove him wrong) to any of the big players.

4

Rep-love t1_j9h8q1g wrote

He is just a guy who really messes with the AI community. But the fact he has written AI doomsday stuff so much... It seems dishonest but I don't know how otherwise, he does not know as much about AI as he says he does. He is completely wrong about AI doomsday, it's just not connected to nearly as much as what he says.

3

SirZacharia t1_j9kf9jf wrote

Some of you might enjoy Neoreaction: A Basilisk. The author does a deep look into Yudkowsky among some other reactionaries. Menscius Moldbug and Nick Land.

3

vivehelpme t1_j9kqb1k wrote

The Donald Trump of early 2000's era of AI/futurology blogging, biggest and loudest opinion in the noble field of standing at the sidelines speaking as a confident expert about something that doesn't exist to start with, and that he is completely uninvolved in developing as a punchline.

Loud voice, huge ego, no substance. Seems to be enough to get a good following.

3

DonOfTheDarkNight OP t1_j9ksrre wrote

I just got to know that he is not an actual AI researcher or engineer.

2

CellWithoutCulture t1_j9nploc wrote

That's not right. He is a researcher.

You know Bostrom right? He wrote the book on the subject and credits Yud for many things. I mean what more do you want? Influential essays? Theories? They are all out there.

He's just no much of a coder from what I can see. Having released little code.

2

Ginkotree48 t1_j9h1g8e wrote

Hes smart he knows hes smart and I think hes right and that we are fucked.

2

WarAndGeese t1_j9zqih4 wrote

People should really focus on ideas. He is just a dude, and evidently a cult formed around him. I have stayed away in part from certain movements like effective altruism despite independently coming to the same logical inclusions long before hearing about them, because my suspicion that a lot of those in the movement were pursuing social status. That further seemed to develop into a cult. That's not to say that the effective altruist community is uniquely cult-y, it's probably less so than any other human community, but for a community that also calls itself rationalist you would think they would have disposed of that sort of behaviour long ago.

In short he's just a guy, people should stop focussing so much on people like that, and people should focus on the ideas like the potential impending threats of artificial intelligence, as well as other progress for humanity.

2

zkJdThL2py3tFjt t1_j9k22ou wrote

His central thesis just seems akin to the South Park underpants gnomes to me. Like (1) develop an AGI with superintelligence and then (2) "???" and then (3) humanity is doomed! Not in disagreement with the premise per se, but the "everybody suddenly falls over dead" thing is just so preposterous to me on a material or pragmatic level. But perhaps I just lack the imagination to see how physical agency could be achieved without us noticing it at all and then just "pulling the plug" if you will...

1

Environmental-Ask982 t1_j9hpdxi wrote

lmao bro, some guy dares question the machine god!

luddites need to burn!😍😍😍😍😍😍

−1

obfuscate555 t1_j9gybf2 wrote

I'm glad he's out there making his case, but watching the Bankless interview, it reminded me of the Joe Rogan episode where a pandemic researcher went on claiming millions upon millions of people we're going to die.

−8

3_Thumbs_Up t1_j9hb1um wrote

That seems like a very bad basis to dismiss something.

Someone else was wrong about a similar statement.

Everyone will be wrong about human extinction until someone is right. Your method of reasoning would never be able to distinguish the person who is right.

8

blueSGL t1_j9h04sj wrote

it's up to something like 6.5mil right now, so off by what? one order of magnitude?

6

obfuscate555 t1_j9h1v6n wrote

Pretty sure he was talking about the US alone.

0

obfuscate555 t1_j9h2ed5 wrote

And even if he wasn't 6.5 mil / 8 bil is still pretty small. My point is I think he's on an extreme end of the spectrum.

0