Tech-loving geeks: Threat to humanity?

Reacting to a new British Royal Academy of Engineering report warning of potential dangers posed by increasingly autonomous technologies, The Economist weighs in on the looming danger of smarter-than-human machines with its piece Machines in control. The Economist notes that turning over full control to autonomous machines is scary… and that NOT turning over full control can be even worse!

How soon before evolvable machines become cleverer than people? Little over a decade is the current consensus. One such machine has already been awarded a patent for something it quietly invented on its own.

The temptation to surrender control to machines that are smarter, more vigilant and less prone to boredom, irritation and emotional outbursts than people will be overwhelming. People will do so for reasons of comfort, convenience, safety and cost. So, what happens when a one-in-a-billion bug causes the software to crash, or the safety valves are not operated properly?

That is what happened at Three Mile Island in 1979. Though the nuclear power station was not an autonomous system, it was running automatically with its human controllers outside the loop. When things went horribly wrong, inexperienced operators tried desperately to take command, only to make one compounding mistake after another—turning a control system with good negative feedback into a positive, runaway disaster.

We really must — though we almost certainly won’t — contemplate most seriously the potential dangers our intelligent creations pose and ensure that our machines remain truly under our control. The free market can’t even keep bankers from destroying the world economy. How can we hope to control smarter-than-human machines produced by profit-hungry companies and designed by tech-loving geeks?

Posted by James on Tuesday, September 01, 2009