I had a fascinating coffee with my friend Steve Omohundro, a physicist, inventor and computer scientist. One of the areas he has done research on is self-aware systems and more particularly computer systems that change their own programming. What he found was that these systems inevitably change themselves in ways that we don’t predict. And this can be scary.
As an example, he cited a computer that is programmed to play chess. The computer may figure out more ways to fulfill its mission, for instance hacking into other computer systems to play more chess games. If it understands enough about the world, it may realize that more funding means that it can play more chess, so it may learn how to do phishing attacks or break into bank accounts. You might feel you can always unplug the computer, but if the computer knows enough about how thinks work it may find ways to prevent that. Self-improving systems can change so quickly and dramatically that they will inevitably cause problems. If they know enough about how the world works, they act in ways that are hard to stop.