“Actually, I don’t work for Vitadyne, Neil. I’m an outsider. I don’t want to steal any of your code. I just want to make sure you haven’t caused a safety problem for the rest of us.”
Neil frowned and looked around. “Sorry dude, but no can do.”
Axel unholstered his pistol and also took out a large hunting knife. “Now think about this for a minute, Neil. I was able to get into the building easily. I know what you were working on. I know your name. I was able to get you to come here by hacking in and sending you some carefully crafted emails and voice mails. Think for a minute about who you’re dealing with, the resources we have and what we’re capable of.”
Neil wasn’t stupid. He would make all the connections. He would quickly realize what it took for Axel to simply be standing in front of him.
“Or I could start cutting off your fingers one by one,” Axel said, getting impatient.
Neil finally showed a glimmer of fear. “Fine, fine. What do you want? I’m just a grunt developer, man. I don’t have any corporate secrets.”
“I already told you what I want. Tell me what you were doing at 11:40 p.m.”
“I was just testing the new deep learning system.”
“Tell me about it.”
“We’re developing a new AGI here based on the Wog platform. It’s supposed to run market simulations for our video division, so I thought I would allow it to sample some external data. I know I’m not supposed to do that, but there are lots of safeguards we put in place. It automatically self-terminates in twenty-four hours, it can’t replicate, and it has other constraints. The recursive self-improvement function requires human approval. We’re kosher here, dude. I know what I’m doing.”
“You let it have external network access?”
“Just for a few minutes, then I shut it down. It was just a test, man. Totally harmless.”
Axel shook his head, letting Neil know he wasn’t convinced.
“Hey man, I just work here, okay? No one else is as good at programming this stuff, so I can do what I want. I was just playing around. Sure, I made a little mistake, but no harm done.”
“I need proof.”
“Sure, man, sure. Here, look.”
He was lightning fast on the keyboard—too fast for Axel to keep track of everything he was doing. Axel hoped he wasn’t subliminally triggering alarms, but he couldn’t be sure. An activity log popped up for 11:39 p.m. to 11:46 p.m.
Axel took a picture and sent it to Grant. A brief glance at the log seemed to match Neil’s description.
“You can see the constraints here too,” Neil said. Then he navigated to the command structure and showed Axel the AGI constraints. It was checking out.
Axel was relieved. This was probably the most advanced AGI development operation he’d seen so far. They had been testing recursive self-improvement and abstract planning functions that others hadn’t. But they had also put constraints in place, which should prevent the release of a self-improving machine entity.
“I’ll need all your login and security credentials.”
“Sure, man.” Neil went about printing a sheet of login information. Axel did a trial run with one of the credentials, and it worked.
Neil’s leg was shaking. It wasn’t a normal shake. It was a shake of nervousness. It could be nothing, or it could be he wasn’t telling him everything.
“This checks out,” Grant chirped in his ear, “but there were two external IP impressions with different AGI signatures. What about the second one occurring at 11:52 p.m.?”
A subtle feeling of malaise crawled over Axel. If the second impression was hunky dory, Neil would have shown his activity log for that time as well.
“What about the second AGI signature occurring at 11:52 pm?” Axel asked, leaning closer into Neil.
Neil’s face contorted. “Look, I fucked up okay, but it’s just a pet project for the sporting goods division. I know some of the guys over there, and they don’t get much attention. I knew if I could unleash this to its full potential it could totally kick ass.”
“Describe unleash it to its full potential, in practical terms.”
“Like, without the constraints, but I stopped it. I even checked my system. No latency.”
“How do you know it didn’t self-replicate across the network?”
“I… I… don’t, but I doubt that.”
“How long had it been running before you gave it access to the network?”
“I dunno, since this morning.”
“Did it ask you for internet access?”
“Yeah. How did you know that? It showed me this really cool proposal for chomping the data. I knew it could make the whole sports division really destroy the competition, but it needed network connectivity for just a few minutes.”
All the pieces were fitting together to paint a harrowing picture. The scenario had all the ingredients for superintelligence, and with no constraints. Ten terahertz computer processing for twelve hours could be equivalent to more than a thousand years of human-equivalent thinking time. And to top it off, it displayed manipulative behavior.
“Grant, are you listening to this?”
“Yes,” Grant’s voice cracked. It was the first time he’d heard real emotion in his voice. “This is an intelligence detonation,” he said.
The words echoed in Axel’s mind.
“Fuck,” Axel said aloud. He was hit with a momentary feeling of vertigo.
“Hey man, it can’t be that bad,” Neil said.
“Time to end the operation,” Grant chirped ominously in his ear.
Axel knew he was right. The outcome couldn’t be certain, but something almost certainly got loose. As to what kind of superintelligence exactly, they would have to figure out later.
Without further hesitation, Axel shot Neil in the head and then collected all of the devices on his desk.
“What should I do with the rest of the office?” Axel asked Grant.
“We’ll take care of it.”
“How?”
“Airstrike.”
“Jesus,” Axel said.
He left the way he came, not bothering to replace the cut windowpane on his way out. He ran until he arrived in a wooded area skirting the town.
“Extraction will be two kilometers southeast of you,” Grant said.
A few minutes later he heard an explosion rock the urban area behind him. It briefly lit up the night.
His earbud chirped again. “Axel, this is Bhavin.”
Axel’s heart pounded violently. “Sir… has Grant informed you what happened?”
To Axel’s surprise, Bhavin wasn’t angry. He wasn’t even surprised. He sounded resigned, his voice weighed down by fatigue. “Yes. I know. Not your fault, Axel. You see, without government help, without widespread understanding of the risks, it was only a matter of time. You bought us some time—more time than I thought we would have.
Axel breathed heavily, trying to keep up a rapid pace while carrying the duffel bag.
“Send me a report immediately upon extraction,” Bhavin said, “it’s time for phase two. When you return we will meet in the Fortient war room.”
“Where is that?”
“It’s ten levels below Nadar tower. Grant will send you coordinates.”
“Yes… yes, sir.”
There was a pause.
“Sir, is that all?”
For a moment, Axel thought he’d lost the connection.
“There’s one more thing we need to know.”
“What’s that, sir?”
“We need to know its objective function. What is the primary goal of the AGI system?”
“Right now, sir?”
“Yes.”
Axel stopped running, sat in the middle of the sparse forest in the darkness and pulled out the laptop. He logged in using the credentials Neil had given him.