Our naive technological optimism

I’ve long disliked unabashed technology enthusiasts, like Ray Kurzweil, who giddily rave about exponential technological “progress” and brush aside concerns that smarter-than-human robots and computers could turn destructive.

The nuclear and mortgage industries perfectly illustrate my fear that humanity consistently underestimates risk, esp. when short-term profit can be made by taking “small” long-term risks. As Tom Cochran, a senior Natural Resources Defense Council nuclear scientist, said:

The probability of a core melt had been estimated at about one chance in 10,000 reactor years of operation, he said. “We’ve had now three core melts in 30 years in less than 500 reactors, he said, referring to Three Mile Island, Chernobyl and, now, Japan. "So the probability of a partial core melt is one chance in several hundred instead of one chance in 10,000.”

His math eludes me, but since four Japanese reactors are apparently melting down, I count six core melts in 30 years. There are currently 442 nuclear plants in operation worldwide. Since the number of reactors has risen over time and some reactors are sometimes shut down for maintenance, let’s assume an average of 400 reactors over the past 30 years.

6 meltdowns / (400 reactors * 30 years) = 1 meltdown per 2,000 reactor years. Meltdowns have occurred five times more often than the “experts” estimated.

How many people would have accepted 1.5 meltdowns per 100 reactors over that 30 year period, had they known the true odds?

Even worse, nuclear power plant technology has been around 60+ years. It’s very well understood, unlike future computers that will be, according to Kurtzweil, “a million times more powerful in 20 years” (Bloomberg BusinessWeek, March 7-13, 2011, p. 38).

It’s virtually impossible to put any reasonable estimate on the likelihood that computers and robots with super-human intelligence will turn on us. Do we really want to take that chance, given our track record of underestimating the threat from known technologies?

Posted by James on Wednesday, March 16, 2011