As von Neumann described over a half century ago, the brain is extremely slow but massively parallel. Today’s digital circuits are at least 10 million times faster than the brain’s electrochemical switches. Conversely, all 300 million of the brain’s neocortical pattern recognizers process simultaneously, and all quadrillion of its interneuronal connections are potentially computing at the same time. The key issue for providing the requisite hardware to successfully model a human brain, though, is the overall memory and computational throughput required. We do not need to directly copy the brain’s architecture, which would be a very inefficient and inflexible approach.
Let’s estimate what those hardware requirements are. Many projects have attempted to emulate the type of hierarchical learning and pattern recognition that takes place in the neocortical hierarchy, including my own work with hierarchical hidden Markov models. A conservative estimate from my own experience is that emulating one cycle in a single pattern recognizer in the biological brain’s neocortex would require about 3,000 calculations. Most simulations run at a fraction of this estimate. With the brain running at about 102 (100) cycles per second, that comes to 3 × 105 (300,000) calculations per second per pattern recognizer. Using my estimate of 3 × 108 (300 million) pattern recognizers, we get about 1014 (100 trillion) calculations per second, a figure that is consistent with my estimate in The Singularity Is Near. In that book I projected that to functionally simulate the brain would require between 1014 and 1016 calculations per second (cps) and used 1016 cps to be conservative. AI expert Hans Moravec’s estimate, based on extrapolating the computational requirement of the early (initial) visual processing across the entire brain, is 1014 cps, which matches my own assessment here.
Routine desktop machines can reach 1010 cps, although this level of performance can be significantly amplified by using cloud resources. The fastest supercomputer, Japan’s K Computer, has already reached 1016 cps.8 Given that the algorithm of the neocortex is massively repeated, the approach of using neuromorphic chips such as the IBM SyNAPSE chips mentioned above is also promising.
In terms of memory requirement, we need about 30 bits (about four bytes) for one connection to address one of 300 million other pattern recognizers. If we estimate an average of eight inputs to each pattern recognizer, that comes to 32 bytes per recognizer. If we add a one-byte weight for each input, that brings us to 40 bytes. Add another 32 bytes for downward connections, and we are at 72 bytes. Note that the branching-up-and-down figure will often be much higher than eight, though these very large branching trees are shared by many recognizers. For example, there may be hundreds of recognizers involved in recognizing the letter “p.” These will feed up into thousands of such recognizers at this next higher level that deal with words and phrases that include “p.” However, each “p” recognizer does not repeat the tree of connections that feeds up to all of the words and phrases that include “p”—they all share one such tree of connections. The same is true of downward connections: A recognizer that is responsible for the word “APPLE” will tell all of the thousands of “E” recognizers at a level below it that an “E” is expected if it has already seen “A,” “P,” “P,” and “L.” That tree of connections is not repeated for each word or phrase recognizer that wants to inform the next lower level that an “E” is expected. Again, they are shared. For this reason, an overall estimate of eight up and eight down on average per pattern recognizer is reasonable. Even if we increase this particular estimate, it does not significantly change the order of magnitude of the resulting estimate.
With 3 × 108 (300 million) pattern recognizers at 72 bytes each, we get an overall memory requirement of about 2 × 1010 (20 billion) bytes. That is actually a quite modest number that routine computers today can exceed.
These estimates are intended only to provide rough estimates of the order of magnitude required. Given that digital circuits are inherently about 10 million times faster than the biological neocortical circuits, we do not need to match the human brain for parallelism—modest parallel processing (compared with the trillions-fold parallelism of the human brain) will be sufficient. We can see that the necessary computational requirements are coming within reach. The brain’s rewiring of itself—dendrites are continually creating new synapses—can also be emulated in software using links, a far more flexible system than the brain’s method of plasticity, which as we have seen is impressive but limited.
The redundancy used by the brain to achieve robust invariant results can certainly be replicated in software emulations. The mathematics of optimizing these types of self-organizing hierarchical learning systems is well understood. The organization of the brain is far from optimal. Of course it didn’t need to be—it only needed to be good enough to achieve the threshold of being able to create tools that would compensate for its own limitations.
Another restriction of the human neocortex is that there is no process that eliminates or even reviews contradictory ideas, which accounts for why human thinking is often massively inconsistent. We have a weak mechanism to address this called critical thinking, but this skill is not practiced nearly as often as it should be. In a software-based neocortex, we can build in a process that reveals inconsistencies for further review.
It is important to note that the design of an entire brain region is simpler than the design of a single neuron. As discussed earlier, models often get simpler at a higher level—consider an analogy with a computer. We do need to understand the detailed physics of semiconductors to model a transistor, and the equations underlying a single real transistor are complex. A digital circuit that multiples two numbers requires hundreds of them. Yet we can model this multiplication circuit very simply with one or two formulas. An entire computer with billions of transistors can be modeled through its instruction set and register description, which can be described on a handful of written pages of text and formulas. The software programs for an operating system, language compilers, and assemblers are reasonably complex, but modeling a particular program—for example, a speech recognition program based on hierarchical hidden Markov modeling—may likewise be described in only a few pages of equations. Nowhere in such a description would be found the details of semiconductor physics or even of computer architecture.
A similar observation holds true for the brain. A particular neocortical pattern recognizer that detects a particular invariant visual feature (such as a face) or that performs a bandpass filtering (restricting input to a specific frequency range) on sound or that evaluates the temporal proximity of two events can be described with far fewer specific details than the actual physics and chemical relations controlling the neurotransmitters, ion channels, and other synaptic and dendritic variables involved in the neural processes. Although all of this complexity needs to be carefully considered before advancing to the next higher conceptual level, much of it can be simplified as the operating principles of the brain are revealed.