>The danger is that we don't yet know how to properly encode our values and goals into AI.

The danger is to give such a system too much power, maybe without delay between "having an idea" and executing it. Also, not having other systems in place to stop it if something would go wrong.