That's a design principle. What are you designing?

Moreover, the simplest explanation would be that they're telling the truth. They're afraid of getting killed.

Humans are extremely aligned compared to what's theoretically possible. We just generally focus on the differences rather than the similarities, because the similarities seem so obvious that we don't even consider them.

>We can test with things that are highly unlikely to be in the training data.

We can also test things where theres an infinite amount of alternatives so that memorization would be impossible.

If GPT could solve every arithmetic problem thrown at it, then it's obvious that it has developed some understanding of arithmetic, as it's simply impossible to memorize the answer for every possible problem.

However, the fact that it fails on arithmetic of large numbers could be an indication that it doesn't understand, but failure could also be caused by other factors, such as lack of enough working memory or similar (humans would fail at multiplying large numbers in their head as well).

So I think one could prove understanding, but proving lack of understanding seems harder.

You wouldn't need a billion robots to start a new society anymore than you'd need a billion humans.

Unnecessarily insulting people on the internet make you seem really smart. OP, unlike you, at least contributed something of value.

Have you heard of books?

Sometimes smart people can read one for hours straight. There are even some smart people who have read hundreds of books in their lives.

There is an infinite multitude of ways history might play out, but they're not all equally probable.

The thing about the singularity is that its probability distribution of possible futures is much more polarized than humans are used to. Once you optimize hard enough for any utility curve you get either complete utopia or complete dystopia the vast majority of times. It doesn't mean other futures aren't in the probability distribution.

When AI is better at creating new technologies for developing AI than humans.

If you are equally likely to be any one human throughout history, then you're most likely to be born in the time period that supports the most humans.

That seems like a very bad basis to dismiss something.

Someone else was wrong about a similar statement.

Everyone will be wrong about human extinction until someone is right. Your method of reasoning would never be able to distinguish the person who is right.

Once again not true.

From my perspective it has a cost, because I value other things than my own survival. As do most humans who are not complete sociopaths.

You're not counting the full cost of humanity dying. Humanity dying also means that all the future humans will never have a chance to exist. We're potentially talking about the loss of trillions+ of lives.