naptastic

naptastic t1_j7hr6lx wrote

An AI smart enough to detect the work of another AI is going to require three things developers are unwilling to give at the same time:

  1. mutable state
  2. self-reference
  3. self-modification

No one has realized that auditing these systems is very simple. (Though it will raise the questions of what intelligence counts as "artificial", where the line is between "programming" and "training", and it will require considerable human input.) Tedious, but simple.

We developers do our best to keep mutable state as small as possible. We have very strict rules about references. Self-modifying code... nobody does that. Maybe in a lab setting but never in production.

If you don't follow the rules with references, your program crashes (or leaks memory). If you carry a lot of mutable state, your program becomes brittle, even buggy. Code that rewrites itself is the definition of "unpredictable." Users don't like crashes, memory leaks, bugs, or unpredictable behavior, so we follow the rules...

But a smart AI is going to have to be able to refer to and modify itself with very few restrictions, and it's going to do so in ways we can't predict. However, I'm pretty sure that anyone clever enough to build a smart AI will also implement an effective safety harness.

Whether this program ever gets written or not is a more interesting question to me than whether life exists elsewhere in the Universe.

1