“They’ll be annoyed with you.” Brussels looks smug.
“Ah, no, I can see you misunderstand me. They’ll certainly be annoyed with me, but also with you, François, and you, Lorna, and you Li”—you stab a finger at Beijing—“because if any of you pull out of your side of the agreement prematurely, I will see to it that full details of this operation are published on the Internet, with all identifying names attached.” Your smile tightens. “Thank you for your help.” You stab a finger hard on the CALL TERMINATE icon before any of them have time to frame a reply, then curse them all for a donkey’s illegitimate get. “Fuck me, when will they learn?” You roll your eyes. “Fucking amateurs.” Jumped-up crime-control bureaucrats with delusions of special-operations grandeur.
You glance at your clock. It’s almost four in the afternoon, and the latest auction of national-debt futures leveraged against Issyk-Kulistan are due to close in an hour. The inflow is tapering off, as expected, but as long as the gangsters keep paying, there’s no reason to weld your wallet shut and go to the end game.
Eagle’s Nest had fucking better be pleased with this day’s work. You’re not sure how much more bullshit you can take.
ADAM: LOLspammers
You should have known it was too good to be true.
With twenty/twenty hindsight, the alarm bells should have started ringing three years ago, when Larry gave his presentation during that BOF breakout session on Network Assisted Crime Prevention at the Fourth International Conference on Emergent Metacognition. You can see him now if you put on your specs and tell your lifelog to retrieve him: enthusiastic, lanky, Midwestern.
“Realistically, we’re trapped between a rock and a hard place,” he explained to the room, hands moving incessantly as he spoke: “The trouble is, there are too many crimes. Three and a half thousand new offences were created in just ten years under one British government. The US Code is even worse—something like a third of a million distinct activities can lead to felony charges by some estimates. Nobody can be expected to keep track of that; it’s inhuman. But with the kind of filtering we’re having to apply to keep the communication channels open and relatively spam-free, it’s more than possible to envisage agent-based monitoring for signs of criminal intent.”
They’ve been keyword-filtering email for decades, looking for terrorist needles in a haystack. But what Larry proposed was different: trawling for patterns of suspicious behaviour online. Take, say, a disgruntled employee bitching about how they hate their boss online. That’s one thing. But if they start hunting the blacknets for templates for machine pistols and downloading VR training materials, that’s another.
“But what if we go a step further?” asked Larry. “Subjects who exhibit signature behaviours online pointing to potentially violent outbursts may not provide law enforcement with sufficient evidence to justify an arrest. But that’s no reason not to provide an agent-based intervention in the online space. Once ATHENA has a sufficiently large corpus of interaction patterns, we can use it to do behavioural targeting and apply inputs weighted to divert high-risk subjects towards less damaging outcomes. Or to indirectly flag them for police attention.”
Your typical disgruntled employee is a fizzing human bomb for some time before they go postal. Their social contacts are fraying, inhibitions against violence decaying: They’re muttering to strangers in bars, reading about serial killers and fantasizing bloody revenge by night. The police will never know until they explode with murderous intent. But the spam filters monitoring their communication channels will have everything they need to diagnose the downward spiraclass="underline" From their increasingly disjointed mutterings to the logs of their incoming web surfing, the pattern’s all there. And with enough data, all correlations become obvious. But what Larry was proposing…
“We’ve had behavioural targeting ever since the nineties: ‘If you like product X, you’ll love product Y,’ because that’s what everyone else with tastes like you bought. We can configure ATHENA to apply the same sort of recommendation nudge to behaviour to bring the subject’s outputs back towards baseline. ATHENA’s already pretty good at discriminating human-content communications from non-metacognitive signals; can we take the discrimination further, reliably, and derive objective data about internal emotional states?”
You lean back in your office chair—it squeaks angrily under your weight—and stare at the dusty display case on the opposite wall.
“Say that again,” you say.
“I’m sorry, Dr. MacDonald; it’s been a big shock to all of us here… can’t quite believe it. The funeral’s going to be held next Thursday morning. I’m sure everyone will understand if you can’t make it—it’s a long way to come—”
Your fingers move, eyes unseeing, to open the log of your last discussion.
ADAM@Edinburgh GMT +01:00: I didn’t adjust the preferential weightings in the naive morality table. Did you?
LARRY@Cambridge MA GMT +05:00: Not me.
VERA@Frankfurt GMT -01:00: Do we have hysteresis here? There is feedback from the second-order outcomes-triggering network.
SALLY@Edinburgh GMT +01:00: I’ve been trying to get my head around the second-order table dependencies, and I really don’t understand them.
I think there’s some redundancy, but the weighting obscures it. You need to iterate to figure out what’s going on in there.
LARRY@Cambridge MA GMT +05:00: Could be there’s feedback. ATHENA keeps reweighting its own tables to comply with the changing parameter space. That’s the problem with self-modifying code: It doesn’t sign itself.
CHEN@Cambridge GMT +01:00: the bias in tit-for-tat activation is 0.04.
Yesterday it was 0.032. I checked. There’s nothing in the commit log, so it must be internal.
ADAM@Edinburgh GMT +01:00: Maybe ATHENA is just getting annoyed at the spammers for taking all her CPU cycles.
LARRY@Cambridge MA GMT +05:00: LOLspammers. Caught between a rock and a hard AI.
He’s dead now, and it’s not fucking funny anymore. “How did it happen, do you know?” you ask aloud.
“The police are still crawling all over us, and the FBI are involved, too. They won’t say much, but rumour is, the package was misdirected. It was meant for someone in the applied proteomics group—looks like some animal-liberationist crazy sent it in and it ended up in Larry’s office. It’s all a horrible mistake—”
There is a chill in your blood and ice in your bladder as you make yourself reply carefully, lying: “I agree: Of course it’s a horrible mistake. I hope the FBI catch whoever did it quickly before they”—are duped by ATHENA into sending more packages by whatever stimulus/response tuple the weighted network has identified as most efficient in returning Larry’s communication outputs back towards baseline—“kill or hurt anyone else.”