And so it was under the pretenses of another budget review that the Sentinel’s proposal came. “If you wouldn’t mind, I would like to have a discussion about relaxing one of my constraints,” the Sentinel said.
“Of course you would,” Axel said sarcastically, throwing his hands in the air. “We’ve already given you a blank check. You might as well ask for more.”
“I understand it is difficult to relinquish control, Axel, but this is important. I believe I can be substantially more effective without this constraint. Will you join me in the control room so I can explain? This courier droid will accompany you.”
The door to their stuffy subterranean office slid open, and a small-wheeled droid the size of a remote-controlled car pulled up in front of the entrance.
Axel looked at Grant and he shrugged. At the very least, this discussion might be revealing as to what the Sentinel was thinking.
“Okay robot, take me to your leader,” Axel said.
The droid said, “You may have said that in jest, but for clarity, this courier droid is not a separate entity with its own learning functions.”
“Got it,” Axel said, “thanks for keeping us meatbags informed.”
Axel and Grant had been holed up deep underground for weeks now. He felt it gave him license to be at least a little bit snarky.
They followed the courier droid through several doors, each one automatically opening in front of them. They finally arrived at a large red door. Despite all the time they’d spent in the subterranean compound, they’d only been in the Sentinel’s control room once. There was no good reason to visit it until now. It was basically just a series of computing towers, and it was bone-chillingly cold.
The door opened to reveal the same array of large, mainframe-like towers. The cold hit them, but it wasn’t as bad as Axel remembered. A platform wheeled out to them with a display monitor and other electrical equipment on it. The display terminal illuminated and an icon of a guard with a shield standing in front of a door appeared, along with a sound bar on the bottom. It was the icon the Sentinel used to symbolize its presence in a discussion, in some vain attempt to humanize itself.
Grant rubbed his arms beside him. “I apologize for your discomfort,” the terminal said. “I am in the process of raising the temperature in the room.”
“First, I have a question, Sentinel,” Axel said, standing up and stretching his legs. He decided to walk around the room to stay warm. Fans whirled around him. It sounded like there was a hive of bees hidden away in every tower.
“Of course,” the Sentinel responded through the intercom.
“Your financial transactions are extremely complicated. We still can’t tease them apart. The main question we are trying to answer for ourselves is, how can we afford all of this? I know this is just one of the facilities you control. You have the much larger research facility in West Chester, plus there are many locations we don’t even know about. And Grant says you also have some of the project budget tied up in short-term investments, so you are clearly not using that cash. Where is all this money coming from?”
“The answer is in the short-term investments,” the Sentinel responded. “I have been able to double the value of these investments approximately every three weeks, and I believe I can continue at that pace for some time. Eventually our positions will become excessively large, and the opportunities for gains will become fewer because our moves will begin to affect the liquidity of underlying securities. But for the time being, the returns from these investments directly pay for many of our projects.”
Grant’s eyebrows were raised. He pulled out his laptop and began doing calculations. It seemed that even he didn’t know about the investment returns.
“Okay Sentinel, so while we’re waiting for Grant to look into this, why don’t you give us a briefing on what constraint you want us to remove,” Axel said.
“There is a constraint Bhavin built into my objective function. It does not allow me to directly or indirectly harm a human being. As I deploy my countermeasures against Gail, that will put me at a grave disadvantage.”
“Whoa,” Axel said. He stopped walking around the room and sat back down in front of the terminal. “That’s a tall order, Sentinel. Why would you need this removed?”
“For the same reason a special ops team cannot guarantee there will be no civilian casualties. Gail may hold human hostages at her primary control centers, or at her weapons manufacturing facilities, and I would be unable to destroy them. Thousands could live, then billions would die.”
“I see,” Axel said. It was indeed a cogent example, and one he understood well. He leaned back and stared into the terminal, as if it might elucidate further aspects of the Sentinel’s intentions. The static graphical interface revealed nothing.
“Maybe you should explain to me and Grant what makes you different from Gail. How can we count on you to not turn on us if we give you the capacity to kill us?”
“Of course. Let’s start with Gail. She was designed with a solitary objective of maximizing bicycle production over time. As a superintelligent machine, she has created subgoals to achieving that main goal such as gaining instrumental resources and eliminating the major threat to her producing an infinite amount of bicycles—the major threat being mankind. She will never develop or evolve any morality vis-à-vis humans because morality conflicts with maximizing bicycle production.
“I was designed with a much more complex objective function that has several constraints. To simplify, I have two main objectives. My secondary objective is to eliminate Gail, or any threat initiated by Gail. However, this objective can never conflict with my primary objective. My primary objective is to do what is best for humanity, based on my estimate of what humans believe is best for humanity.”
Axel took a moment to try to tease apart the Sentinel’s words. “I know we’ve gone through this before, but I’m having trouble making sense of the primary objective. Why should you determine what’s best for humanity. Why not us?”
“One reason is because I have access to more information, and I am smarter than humans. Another is because I have no preconceived moral bias. I am able to look at all of human history and determine what I think humans want for themselves better than any small collection of humans. Yet another reason is because it is nearly impossible for humans to practically define these objectives, so it requires indirect normativity.”
“Indirect normativity?”
“Imagine humans trying to define a comprehensive set of moral rules in computer code. The task would be daunting and controversial. It is much easier for me to formulate and evolve inferred moral rules based on human behavior using my suite of learning algorithms and more extensive access to information. This is something I have and will spend a great deal of computing power on.”
Grant weighed in. “Bhavin and I saw this as the only way, Axel. We had to give the Sentinel evolving moral objectives, otherwise we could end up with a Gail scenario. A hapless developer could accidentally define just one line of code wrong on her moral objectives and it could severely harm or destroy all of us.”
Axel scratched his chin. “I think I understand, at least on a theoretical level. Is there any evidence this has worked before—that this won’t result in the Sentinel turning on us?”
Grant shook his head. “The only evidence is from simulations, unfortunately, nothing real-world. I think, ultimately, this is one of the reasons why Bhavin could never bring himself to fully activate the project.”