But, it turned out, the denizens of 47 Ursae Majoris had beaten us to the punch, sending what had been dubbed the Reticulum—their alien counterpart of our World Wide Web.
“My predecessors at my lab pioneered automated picture captioning,” Emily said. “It’s such a common feature in cameras now, we tend not to think about it. But the techniques they developed are a big part of what’s letting us make sense of the Reticulum.” She was standing next to a wall monitor at the Interstellar Communications Society, an image of three of the aliens filling it.
Hannah Plaxton looked down at the caption. “‘A doctor treating a patient,’” she read aloud.
“That’s right,” said Emily. She gestured, and text labels appeared over the image, identifying the specific objects in it: Doctor. Patient. Individual. Tray. Equipment.
Hannah pointed at the “individual.” “Why isn’t this guy mentioned?”
Emily nodded. “The captioning is done with neural networks; they learn as they go along. The more pictures they looked at, the better they got at picking out what was relevant. Look at the eyestalks on the two aliens who are mentioned in the caption. On the doctor, they’re facing toward the patient. She is looking at the patient; the patient is the focus of her attention. And see the patient’s eyestalks? They’re turned toward the doctor. Those two aliens are regarding each other. But the third guy? The one not mentioned in the caption? Sure, the software realized that he’s there, but he’s not looking at either of the other two; his eyes are turned to look at something out of frame. And so the algorithms decided he was just a bystander accidentally caught on camera, not part of the action being depicted.”
“Interesting,” said Hannah. “In most of our own pictures—ones taken by humans—the people in the frame are looking not at each other, but at the camera.”
“Right. Which is so artificial, when you think about it. In fact, if that ever happens on a TV show or in a movie, we’re freaked out; an actor almost never turns directly toward the camera. But it doesn’t seem that any of the pictures in the Reticulum are posed photographs. And that says something about their society, I’m sure. In any civilization, cameras are going to become dirt cheap—they’ll be everywhere. And storage will become so cheap, as well, that you’ll record everything.”
“But what happens to privacy?” asked Hannah.
“Maybe the aliens never valued it in the first place. Look at what little clothing they wear. It’s all functionaclass="underline" sometimes a sash with storage pouches, protective gear, ornamental ribbons. No one part of the anatomy is always covered, so there are clearly no nudity taboos. And there are plenty of photos in which the individuals in the background seem to be having sex.”
“And they don’t desire privacy for that, at least?”
“Maybe the reason we started desiring it was because, while doing it, we’re particularly vulnerable to sneak attack. But with cameras recording everything, you’re probably perfectly safe all the time—so, what the heck. For us, any picture in which people were having sex, whether it was the foreground of the image or the background, that’d be the thing we focused on. But it’s clear from the eyestalks that that particular act is given no special importance. It’s not ignored—it’s not that every alien demurely swivels his eyestalks away whenever he sees a couple of others going at it; there are plenty of pictures in which others happen to be looking at what’s going on. But it’s not disproportionate.”
“Well, that explains why the Reticulum is smaller than our World Wide Web: no nudity or sex taboo equals no pornography. You could shave a zettabyte off our own Web if it didn’t have all that stuff.”
“Exactly. And there’s more. The aliens have three arms. Males have two on the left and one on their right, and females have the opposite configuration. It’s a weird dimorphism, but you can almost see how evolution selected for it. They sent us numerous pictures of family groups walking as woman-child-man. They don’t hold pincers, but the woman puts her left pincer on one of the child’s shoulders, and the man puts his right pincer on the other, and the child is protected, while the adults each have two arms on the outside to do things with, right?”
“Ah, Okay,” said Hannah.
“The algorithms were able to divine some additional meaning from that. Whether it’s conscious or not, the aliens clearly place things that are precious to them adjacent to their single hands. We’re calling them ‘twoside’ and ‘oneside,’ and anything that’s of great value or needs protection seems to be on the oneside.”
“The algorithms did that? Starting with nothing?”
“They didn’t start with nothing. They started with hundreds of millions of images, and looked at each one with a patience and a depth that no humans could have.”
“Data mining for the win,” said Hannah.
“Exactly.”
It was finally Emily’s turn to take the stand. Hannah Plaxton asked her a series of rehearsed questions about the data mining that had discovered the Reticulum. But then it was Piotr Sudeyko’s turn to cross-examine her.
“Dr. Chiu,” he said, “I’ve been very impressed by the naturalness of Ursula’s speech.”
“Thank you.”
“You really have done a remarkable job, producing an alien AI avatar that makes the Reticulum as easy to query as our own World Wide Web.”
“Thank you. It was a team effort.”
“It’s astonishing, really. I was given to understand that the sort of universal translation Ursula performs is impossible. Can you explain to the good women and men of the jury how it was accomplished?”
“Certainly. We had our first big breakthrough in spontaneous speech recognition in 2010, using fully connected deep neural networks, or DNN. For deep learning, you keep throwing more and more samples at the neural nets, and, by comparing one to another, the nets eventually figure out word semantics, sentence semantics, and knowledge modeling.
“One of the keys was realizing that semantic intent is better defined at the phrase/sentence level, rather than at the word level. After all, the meaning of a single word is often ambiguous—is a bat a flying mammal or a sporting club? But a phrase, or a sentence, or even a whole document, contains rich contextual information that we leverage. And, of course, the whole web, whether it’s ours or theirs, contains countless Rosetta stones. There are only a handful of ways to lay out the periodic table, for instance, and any technologically advanced civilization is going to have some sort of representation of it.
“So, our neural nets just kept looking to see which alien words were juxtaposed frequently with which objects in accompanying illustrations. It’s a statistical game, but if you play it long enough, you win.”
“I see. And I suppose for simple things it was fairly easy to come up with a translation table, no?”
“That’s right. For instance, they sent thousands of pictures of mineral specimens, and by examining text linked to those pictures, not only were the neural nets able to figure out the alien words for specific types of minerals—their term for ‘quartz,’ say, or for ‘diamond’—but eventually to figure out general terms, including ones with fine distinctions that even most humans are unaware of, such as the difference between a ‘rock’ and a ‘mineral.’”