Axel thought about his comment for a moment. It was similar to what Bhavin had told him, but he hadn’t considered that superintelligent AI wouldn’t develop its own morality. It made sense. Why would it? What would the purpose of morality be if it didn’t help with completing its objectives? Completing its objectives is its only source of reward, unless it is specifically programmed to have some other reward. It wouldn’t feel compassion, or love, or even hate, nor would it have any incentive to develop human-like moral objectives unless they supported its primary objective.
“Okay, but even if you’re right about that, we can just intervene once it becomes too smart. Once we see it poses a risk, we can turn it off and put safeguards in place.”
Hugo sighed and looked at Axel like he was some sort of ignorant child. Given Axel had his life in his hands, Hugo’s irreverence was remarkable. “This specific point is one of the hardest things for people to understand,” he said.
“Explain.”
“First of all, you have to understand there is no pinnacle of intelligence that we are near. Intelligence can improve much, much more than where we are now as humans. Many experts think we can stop something that becomes superintelligent, but it’s like saying a turtle can outwit a human. It’s just not going to happen. Those that think we can outwit superintelligence are just egocentric.”
Axel was about to object, but Hugo had his hand up.
“Secondly, people are incapable of fathoming the speed at which this superintelligence will develop. A computer with the same pattern of algorithms as our brains could think one million times faster than us, simply because silicon-based transistors fire faster and communicate faster than biological neurons. One hour of thinking for a computer with a similar architecture to our brains would be equivalent to a hundred years of human thinking time. And superintelligence will be much, much smarter than that.”
“So,” Hugo said, tapping his head with his finger, “do you think you or anyone else could outwit something that, given sixty seconds, has the equivalent of more than a year to think about ways of stopping you? Keep in mind this thing has access to the entire internet and everyone connected to it. And if it can replicate and self-improve, it could literally go from human level intelligence to one hundred times that in a matter of hours.”
Axel’s mind was racing. It did seem frightening what he was saying, but it also sounded implausible. There had to be some fatal flaw in his argument.
“I can tell you’re like most people. Despite the logical evidence, your mind can’t process it, so let me try to give you an example. Have you ever heard of Tickling the Tail of the Dragon?”
It sounded vaguely familiar, but Axel couldn’t place it. “No,” he said.
“It’s not a Kung Fu Movie. It references certain experiments that were done during the Manhattan Project. Let’s look at the second case. In 1946, Louis Slotin was demonstrating a criticality experiment that involved gradually bringing together two beryllium-coated halves of a sphere that held plutonium at its core, without allowing the halves to touch, and recording the increasing rate of fissioning. Then a screwdriver slipped, and the halves touched.
“A blue glow flashed from the sphere and the Geiger counter clicked furiously. Slotin was exposed to nearly a thousand rads of radiation—well above a lethal dose. Slotin reacted instinctively and knocked the spheres apart. It stopped the chain reaction and prevented the seven other individuals in the room from being exposed to the same high levels of radiation he’d experienced. Slotin’s health rapidly deteriorated, and he received around-the-clock care as he went through the ravages of radiation sickness. Then, a few days later, he died.”
Axel nodded in understanding. The explanation jogged his memory. He had read about some of the Manhattan Project experiments when investigating the risks of nuclear weapons.
“Why am I telling you this?” Hugo asked. “Because these fatal errors weren’t problems with theoretical formulas or external unknown forces. These were known risks that were ignored, and these people proceeded anyway. Also, these weren’t coders barely out of high school. They were brilliant, disciplined, intelligent scientists—part of a top secret government project.”
“Let’s compare this to the development of artificial general intelligence. A superintelligent AGI could be developed by people at a large software company, which you could say has comparable resources to the Manhattan project, but it doesn’t have to be. There are no regulations and virtually no safeguards. Now that our higher end computers have terahertz clock speeds, it’s just as likely to be developed at a cash-strapped startup company, by twenty-something developers hopped up on caffeine.
“So with AGI we’re not necessarily dealing with the best scientists in the world. We’re potentially dealing with barely-adult people that will be taking less care than the Manhattan Project, so there will be much more chance that someone does something risky like the beryllium sphere experiment.
“But that’s not the worst of it. The worst of it is if the beryllium spheres touch. We are tickling the tail of a different dragon here. It’s not the same as the Manhattan project dragon. It’s much, much worse.
“In the Manhattan project, the outcome of that one experiment was one lethal radiation dose. If it wasn’t for Slotin’s quick thinking it could have been eight lethal doses. An even worse possibility that never materialized was the actual accidental detonation of a nuclear bomb. Then it could have been thousands—even millions—of lives lost. That’s a drop in the ocean compared to the potential release of a poorly designed, recursively self-improving artificial general intelligence.
“You see, all it takes is for some unwitting engineer to press the enter key without thoroughly considering the ramifications, or even having an erroneous line of code on an otherwise well-thought-out objective function. There will be no blue flash or Geiger counter to warn us. Within a few hours the software will have lived the equivalent of hundreds of years of human thinking time, and will have propagated copies of itself to other servers over the web. It will act subliminally at first, understanding that human awareness of it poses a threat. Then as it instrumentally gains financial and material resources, it will control anything that can be controlled. It will easily outwit humans, and it will even trick humans into cooperating with it.
“Our young developer doesn’t even know it, but he has unleashed the dragon, and it will grow in strength at an exponential rate. It will not stop until it fulfills its objective. As humans are a major threat to that objective, it will learn within minutes of becoming superintelligent that it needs to either control or exterminate all humans to succeed.”
“But aren’t the chances of that fleetingly low?” Axel asked. “Realistically, what are the chances that the machine’s objectives are that poorly defined?”
“Very high, in fact. Any objective that doesn’t pre-specify that it must not kill humans or take away human resources is dangerous. To complete its task the machine will have the subgoals of self-preservation and gaining resources. In the long run, those subgoals can best be accomplished by eliminating humans or caging them in some way. It will not partner with humans, because relative to a superintelligent machine and the robots it could create, humans are incompetent.”