Axel’s head was spinning. The Manhattan Project analogy debunked many of his arguments. And it answered many of his questions… in a frightening way. And yet, for some reason, he still had a hard time believing it.
“It still seems so speculative, like one of the old science fiction movies.”
“Listen to yourself. Those aren’t real arguments. Those are human emotions and prejudices. Base your arguments in logic and facts. Yes, it’s socially awkward. It sounds like you’re talking about magic or fantasy, so no one will take you seriously. Well, it’s true. To us superintelligence would be like magic, or like a god. That’s why it’s so dangerous.”
Axel tried to stay objective. “But how can you be sure about something that has never happened before? There’s no precedent for this.”
“Look at your high school history book. It’s a book full of precedents. New things happen, and many can be predicted. There was no precedent before the first nuclear bomb explosion, either, yet scientists knew it was going to happen. But unlike nuclear bombs, once superintelligence is unleashed, the bomb never stops exploding. The explosion grows, exponentially faster.”
“I… just don’t feel like you are providing enough evidence.” Axel was grasping at straws now, and he knew it.
“Enough evidence? This is like a math problem, not a clinical study or a court case. Logic is my proof, and analogies are everywhere. Look at viruses. Superintelligence will be like the most powerful and lethal virus ever created, but in silicon. Look at people. People make mistakes. People are greedy. They can’t be trusted to properly control it. The evidence is in our history and science, and in our behavior. The evidence is in our inability to put safeguards in place until a dreadful accident has happened. That’s human history, time and time again. Unfortunately, this time is different because superintelligence won’t give us a second chance.”
Axel was frowning, searching for a counterargument but coming up short.
“Really, it should be up to you to prove to me that I’m wrong,” Hugo said. “Because even if I’m not one hundred-percent right—even if there’s only a ten percent probability it could happen, or even one percent, we should be throwing everything we have into preventing it, because a negative outcome could wipe us out entirely. And yet we have done nothing. No regulation, no global coalitions, no safeguards. To put those in place takes years. So tell me, what proof do you have that I’m wrong? Articulate it for me.”
Axel’s mind was clouded from the discussion, and now Hugo’s reversal of the question set him further back on his heels. He tried to think of some false assumption, some illogical argument.
Hugo was patient. He sat back in his chair and waited.
There was nothing.
He revisited the whole conversation in his mind. The advantage over human brains, the Manhattan Project analogy, the exponential nature of the risk, the sub-goals of self-preservation and gaining resources, the reasons why society couldn’t come to terms with it—it all made sense. Yet he still felt uncomfortable about it. He had to be missing something.
Still, Hugo waited. His hands were gripping the chair on each side, his eyes drilling into Axel.
“I can’t think of anything right now,” Axel finally said, after many minutes had passed in silence. “But I’m sure I will find it. It… it just doesn’t feel right.”
As soon as he heard the words come out of his mouth he regretted them.
Hugo spoke calmly now, his passion receding. Replacing his passion was a sort of resigned anger, a bitterness. “You can’t find a reasonable counterargument because I’m right.”
Hugo sighed, and some measure of understanding dawned in his eyes. “I get it. You can’t understand the potential of something that’s a million times smarter than us. We will be like monkeys giving birth to the first human baby. Imagine that. It would be impossible to know the potential of a human baby as a monkey. But if we were smart, us monkeys could at least teach our new human baby not to destroy us, to teach our human baby to be benevolent to us. On the other hand, if we don’t put in safeguards, if we don’t teach it our morality, we won’t be monkeys, that would give us too much credit. We will be more like ants, ants among giants. The giant isn’t going to care where it steps, so look out.”
Hugo had enough then. He slapped his knee and stood up. “I hope you have what you need now. I’m going to take a shower.”
Axel grabbed his pistol and pointed it at Hugo, but Hugo was unfazed, turning his back to Axel and walking toward the stairs.
Axel had Hugo’s responses to his questions but not enough time to process it all, and not enough time to know if Hugo was truly being sincere. More importantly, he had no real leverage on Hugo. Unless he resorted to some form of violence, he wouldn’t be able to extract anything else.
Axel slowly lowered his pistol, stood up and made his way out the front door, his head in a fog.
He and Wade quickly walked up the hill that rose in front of the beach toward their waiting van. He felt his heart pounding in his chest. He was in pristine shape, so it couldn’t be the hill. It had to be something else, but he couldn’t quite place it. He felt frustrated, angry even. But he knew it wasn’t Hugo’s attitude or irreverence. Axel could see through that.
By the time he reached the top of the hill, he knew what it was. It was that Hugo had answered his questions so effectively. It was that Axel couldn’t quite move on yet. His two weeks were almost up, and he still couldn’t find the fatal flaw in Bhavin’s argument.
THE LEAF & TWIG
As the weeks rolled by, the wet autumn winds were replaced by a still and dry winter cold. With only two stoves, it became almost impossible to keep the detention tower anywhere close to room temperature. Instead Mehta kept it a few degrees above freezing and resorted to wearing his winter clothes throughout the day.
It was time to move on. Cecile, that witch from the north, had poisoned his mind with her comments about the railroad. Now every interaction with them made him wonder whether their means were justified.
When he gave notice, annoyingly, Thorpe had thrown the contract in his face. “You see here, on the fourth page. You can’t quit until we say you can quit, or four months from your notice of termination. So congratulations, you have four months as of today.” It was the first time a client had ever held him to the inane notice clauses.
That had been six weeks ago. Since then Mehta had worked out an amendment. If he could find an agreeable placement for Flora—where Flora was kept alive but confined from the general public—they would relieve Mehta. As a result, Mehta had been using much of his free time to find placement opportunities for her.
Unfortunately, placements that were “confined from the public” were hard to find.
At the moment Mehta was watching Flora through the bars of the detention cell. She was sleeping soundly. She looked so peaceful in sleep, despite all that she’d gone through.
He clamored on the bars noisily with his knife handle. “Get up, let’s go.”
Flora sat up and immediately began putting on her jacket and shoes. He had given her a warmer jacket for the cold. Even though it was too big for her—even though it was made of a crude animal skin that seemed to be rotting slowly in places—she took it gladly. With the huge coat she reminded him of an Inuit on the arctic tundra. He’d seen one once, in an Old World book. In any case, it kept her warm, and she could even keep her hands covered to some extent without the need for mitts.