The article addresses a lot of topics on a very superficial level.

The one thing that did appeal to me as interesting was the idea that the advent of the deadly paperclip maker could be prevented by implementing „common sense“ in AI.

My opinion on that is that this would be already be possible with a LLM, as long as further scaling of processing power can be economically justified. Common sense depends on semantics and is tied to language and verbal reasoning. AGI would not be necessary for that.

The big problems I see here is the „mesa optimiser“ and its hidden goals as well as reward hacking (my pet peeve…).

Why?

Because common sense is overridden in humans by the pursuit of reward. Humans „wirehead“ and cheat all the time, and about 99% of the population is only partly able to apply common sense in ambivalent/ambiguous situations.