Viewing a single comment thread. View all comments

turnip_burrito t1_iv2u3xz wrote

It depends on what the initial AGI are tasked to do. Whatever their ultimate goal is, building an accurate world model and fast mind are instrumental goals needed to accomplish it.

Let's assume someone gives some AGI somewhere the freedom to just exist and collect experience. I expect an AGI to begin collecting real and virtual data until is it fairly good at modeling the physical world and human interactions. It will know cause and effect, and understand human intentions. It will also try to upgrade its intelligence (cloning, add more hardware, self code editing, etc) because faster processing power and better algorithms will make it better at achieving the ultimate goal.

Now we get to the tricky part of HOW it does these things. The ultimate goal of the AGI, its core impulse, will be determined by its builders. This will cause it to reach ASI level in different ways. I think its intelligence gathering phase will result in an AGI that is (surprisingly!) well-aligned to the expressed intentions of the builders. Let's look at four cases of the builder's expressed intentions:

  1. "You, the AGI, will always do as I/some human moral role model/human philosopher would intend". The AI's actions will be bounded by its internal models of the human. It will try to understand the humans better and refine its model of their intentions. It will likely not overact in a destructive way unless explicitly told to. Whether this is good or bad depends on whose ideals and words it is meant to follow. It is clear which person/people has control of the AI in this scenario. Summary: Good or bad ending (depends on humans in control)

  2. "Help humans reach their full potential while adhering to this literature/list of ethics". The AGI will understand the meaning behind these words and work with humans to increase its capabilities. It will take actions to improve only if not deemed harmful according to its ethics. As an ASI, it will reflect the same ethical constraints used on its ancestral AGI. It isn't quite as clear which human/group maintains control in this scenario. Summary: good or bad ending (depends on initial list of ethics)

  3. "Maximize my company's profits". The AGI will again understand exactly what this means. Profits are gained when revenue is higher than operating costs. The AGI will take underhanded and stealthy actions to increase this one company's profits (stocks, coercion) and basically lock humanity into a neverending corporate dictatorship. Even the owners of the company will not be safe, since logically they could change the company to thwart the AGI. Humans will live very restrictive lives by today's standards. Now consider if the company's industry doesn't require human consumers (not service-based). With no moral code except profits, the resulting ASI will force humanity into extinction as it creates automated routines to play the part of consumers. Basically, you get something like an everlasting paperclip factory or grey goo scenario. Summary: Very bad ending

  4. "Help my country/company/group/friends/me take over everything". It will do whatever it can to put you in a position of ultimate authority, no matter the cost. This would lead to widespread human suffering if the controlling human party doesn't specify otherwise. This AGI may, even as an ASI, be under control of the group of people, since it by definition is part of "everything". What happens next might still be up to the creators. Summary: Bad or good ending (depends on humans in control, but better to avoid)

Sorry for the essay. Hopefully you find something worth thinking about in this.

2