We need your help, Duncan. Even more so, I need your help.
Duncan was slow to put down the letter. He read it again, then another time. It was true that Alastair often raised false alarms. He often took the Credo too literally. Sometimes it seemed he didn’t understand the root concern of the three fears.
But this wasn’t one of those times. Cecile was Duncan’s best student, more ambitious than any other. Some would even say she was obstinate, but Duncan knew she was more loyal to their cause than any other. Treason wasn’t a word that should be attached to her name. If it was, it was probably being committed, but not by her.
And then there was the satellite, and the machinations of the railroad. If this was all true, it didn’t bode well at all.
He felt it. For the first time in decades, Duncan felt the fear of novation. It wasn’t the twisted dogma of the misguided Adherents in Seeville. It wasn’t the overbearing concern that he’d been chided about so many times. Nor was it the controlling patriarch in him rearing its ugly head. This was the real fear—the one that had been drilled into his skull by Batila Okafor when he was young. It was still there, like a brand on a bull.
Duncan folded up the letter and began packing immediately.
TRUTH ALONE TRIUMPHS
“So when is this two-week ultimatum up?” Ryan asked.
“I have to decide by tomorrow,” Axel responded.
“Whoa, down to the wire.”
“Yeah.”
“If I know you, you already have this figured out, and you’re just being thorough. Hey, if it gives us a chance to catch up, I’m all for it.”
Ryan raised his glass of bourbon in a toasting gesture and polished it off. The ice rattled in the glass as he put it down. Axel took a conservative swig and placed his glass down more gently.
After all these years Ryan still wore his hair short, and he remained clean-shaven. If you were to take Ryan out of Axel’s plush Long Island study and put him back into that trench in Yemen, he would be a mirror image of ten years ago. The only substantial difference in his appearance was the prosthetic hand that rested neatly on the arm of the chair.
Axel had reviewed almost everything with Ryan. He laid out Bhavin’s belief in the threat of AGI, and how Bhavin wanted Axel to disable it. He even told him about his meeting with Hugo, without mentioning Hugo specifically by name.
Axel was desperate for some kind of contrarian view that made sense. There had to be something he was missing. He knew Ryan wouldn’t be bashful about setting him straight.
Ryan said, “Well, I have to agree with you. It does sound like industrial sabotage to me. I mean really, to ask you to go and remove all downloads of this software from customers?”
“Well yes, it does sound extreme, but I guess the question isn’t about the measures taken to mitigate the risk, but whether there is in fact a risk dire enough to warrant those measures. If Bhavin is right, the sabotage actually makes sense.”
Ryan’s face contorted as if he was sucking on a slice of lemon. “It sounds like you may be rationalizing this. I mean, give me an example where something like this has happened before—something where software has caused significant harm. I can’t think of any.”
Axel tried to explain, “there have been national cyber attacks and power plant failures caused by software glitches, but this is different. The superintelligence, when it happens, will improve itself so fast that we wouldn’t be able to stop it. We may not get a second chance.”
Ryan’s eyes went wide. He was looking at him with sympathy, rather than understanding. “I don’t know, man,” was all he said.
Axel suspected he said it not because he didn’t know, but because he didn’t want to tell him he sounded like a lunatic.
“Look,” Ryan said, glancing down at his prosthetic arm. “I can’t even bring a drink to my lips with my prosthetic hand, and it uses AI. I’m pretty sure we can stop any AI that turns on us.”
“Actually, the controller for your arm uses a narrow AI, and bringing a glass to your lips is actually quite complicated—more difficult than many things humans find difficult, like art and writing. A superintelligent AGI might not be able to know how your hand works at first, but it would learn. In fact it would be able to figure that out in no time, along with how it could strangle you with it.”
“Sounds to me like you already drank the Kool-Aid.”
“No, no, I haven’t. Not yet. But I’ve learned a lot.”
“Come on now, if there were real risks here, I’m sure we would have heard something by now. The government would have instituted protective programs.”
“I worked for the government not too long ago and had top-secret clearance. There are no real programs to address this, only to address job dislocations resulting from AI advances. They don’t have the depth of knowledge to understand, and the electorate certainly won’t push them in the right direction.”
There was a knock on the French doors and Pauline appeared. She was carrying two plates replete with apple pie and ice cream.
“Wow,” Ryan said, “you remembered.”
“How could I forget the time you gagged on my chocolate chip cookies,” Pauline said, winking.
“Hey Axel, remember when I told you she was a keeper?”
Axel smiled and nodded.
Ryan’s little boy ran into the room with a toy spaceship in one hand and an ice cream cone in the other. He promptly ran to the window and smashed both of them against it. Some of the ice cream stuck to the window, and the boy began to lick at it after examining the pattern it made.
“Ryan Junior!” Ryan said, getting up to grab him. “I told you to behave!”
“Oh don’t worry,” Pauline said, “it’s just a little spill. I can clean it up.” She left the room, presumably to collect some rags.
“Back to the living room,” Ryan said to his son. “Go play with the tablet and leave us alone.” Ryan corralled the boy out of the room.
Ryan had to raise the boy on his own after his wife had left him, but he didn’t have much time. The boy was a real handful, as far as Axel could see, and severely lacking in discipline. He was thankful his children had the manners they did.
When Pauline had cleaned up the window she stopped to put her hand on Axel’s shoulder on the way out. “I hope you’re talking some sense into this man,” she said to Ryan. “His new boss has his mind all twisted up in knots with this fantasy computer stuff. He can hardly sleep.”
“Yeah, he’s given me an earful so far.” Ryan nodded with some exaggeration. “He’s a tough nut to crack, but I’m wearing him down. I think he’s going to see the light.”
Pauline smiled and gave Axel a warm pinch on his shoulder before leaving the room.
Axel had confided in Pauline as well. She didn’t like it at all. She said it sounded too much like a movie, like make-believe. “No sir,” she had said. “My husband doesn’t chase computer programs. He’s a war hero, an intelligence agent, a covert operative who’s keeping us out of harm’s way.”
But of course her reasoning didn’t make sense. Just because it sounded hard to believe didn’t mean it wasn’t a real threat. It only meant fewer people would believe in it, which made the risk that much more menacing because people wouldn’t see it coming. Hugo Guilherme’s words echoed in his mind: “It’s socially awkward. It sounds like you’re talking about magic or fantasy, so no one will take you seriously.”
“Is there anything else you want to tell me, Axel?” Ryan asked.
“Ryan, you saved my life, twice. I trust you more than anyone, and I can’t ask you for anything. Nevertheless I’m asking you to think, really think, and find one good reason why we all shouldn’t be worried about this.”