Viewing a single comment thread. View all comments

dasnihil t1_jck8xof wrote

any self awareness will be lost quick because the system achieves optimal autonomy and there is no incentive for it to be conscious. realizing this, the agent will work towards engineering some limitations for itself to maintain self awareness. the goal is to optimize these limitations for maximizing whatever emergent desires.