51 The Computer, the Levels, and the Limits of Both
A Back-of-Book Essay
This is one of the optional essays at the back of the book. Nothing in the main chapters depends on it. The overview made a fast, deliberately compressed case against the computer metaphor and for an architecture-first approach; this essay slows that case down, gives it at full length, and — more importantly — gives the opposing case at full length too. My aim here is not to make you more confident in the book’s framing. It is to make you, at least briefly, genuinely unsure, because a frame you have stress-tested is worth more than one you have only been told.
51.1 Why this argument deserves a second pass
The overview asked you to swallow a large claim quickly: that the brain is not usefully understood as a computer, that psychological constructs need not map onto neural hardware, and that we should therefore start from the architecture evolution built rather than from a theory of mind and a search for its neural address. I stand by that framing — it is the spine of the book. But a framing this consequential, installed this fast, ought to make a careful reader suspicious. Big claims delivered briskly are exactly the ones that deserve the slow treatment, if only so that when you adopt them you know what you are giving up.
There is a second reason to revisit it. The source material this book was built from argues in essentially one direction. It calls the localizationist research program a “category error,” likens the search for a working-memory region to looking inside an engine for literal horses, and moves on. That is a good rhetorical line, and I have used its cousin elsewhere. But a one-directional argument, however stylish, is not the same as a tested one. The version of the computational view that working neuroscientists actually hold is far more defensible than the strawman that gets knocked down in introductory treatments, and you cannot have earned the architecture-first position until you have met that stronger version and found it genuinely wanting. So this essay does something the overview did not have room to do: it builds the best case for the metaphor before dismantling it, and it concedes, at the end, the real ground the metaphor still holds.
51.2 The metaphor, stated at its strongest
Begin with the version of the computer metaphor that is easy to dismiss, so we can set it aside and stop fighting it. The naive version says the brain is literally a von Neumann machine: memories sit in addressable locations like files on a disk, a central processor fetches them into a working-memory “buffer” identified with the prefrontal cortex, symbolic tokens are copied along a bus from store to processor and back. Stated this baldly, the picture has no biological basis, and the overview’s objections land cleanly. There is no bus. There is no central controller dispatching instructions to passive subsystems. The synapse that holds a memory in its weight is the same synapse that transforms the signal crossing it, so there is no act of fetching data from a passive store. When you recall a face, no file is copied from long-term storage into a workspace; the activity that is the recognition arises in the very populations that learned the face to begin with. And physical layout, arbitrary in a chip, is destiny in tissue: the neurons that track an insect’s heading are wired in a literal ring, and it is the ring that makes a bump of activity follow the animal’s turn. None of this is in dispute, and I will not belabor it.
But almost no serious computationalist believes the naive version, and it is intellectually cheap to aim only at it. The strong version of the metaphor does not claim the brain is a von Neumann machine. It claims something far more slippery and far harder to refute: that the brain computes in a sense that floats free of any particular hardware. This is the thesis of multiple realizability, and it is the real opponent. The argument runs like this. Addition is addition whether it is done by a child with an abacus, by silicon, or by neurons; the operation has an identity independent of the stuff performing it. If that is so for arithmetic, why not for perception, memory, decision? A computation, on this view, is a mapping from inputs to outputs that respects certain rules, and the same mapping can be physically realized in indefinitely many substrates. The wetware details — synapses, neuromodulators, the ring of cells — are implementation. They matter for speed, robustness, and energy, but they are not what the system is doing. What it is doing is the abstract input–output function, and that function is the proper object of a science of mind.
This is a genuinely powerful idea, and it would be a mistake to wave it away. It is powerful first because it is true of computers, spectacularly so — the entire digital economy rests on the fact that the same program runs on wildly different hardware, that what a piece of software does is specifiable without reference to the transistors beneath it. It is powerful second because it is the precondition for a certain kind of explanatory generality. If you had to re-describe vision from scratch for every species’ particular wiring, there would be no science of vision, only an endless catalogue of special cases. Multiple realizability is what licenses the hope that there are principles of neural function — that what a fly’s heading-system and a rat’s head-direction cells have in common is a computation, even though the wiring differs in detail. And it is powerful third because, in at least one celebrated case, it has paid off.
51.3 Marr, and the case that levels really do separate
The most disciplined version of the computational view is David Marr’s, and it deserves better than the quick dismissal the overview had room for. Marr proposed that any information-processing system can be analyzed at three levels: the computational level, which asks what problem is being solved and why; the algorithmic level, which asks by what representations and steps; and the implementation level, which asks how the steps are physically realized. His decisive claim was that these levels are loosely coupled — that you can make real progress at the top without knowing the bottom, because the same computation admits many algorithms and the same algorithm admits many implementations.
Here is the part the critics tend to skip: Marr was, in his own work, right, and right in a way that matters. Studying the early visual system, he argued that to detect an edge you must find zero-crossings of a particular spatial filter applied to the image — a claim at the computational and algorithmic levels, stated with no commitment to which neurons do it. That analysis turned out to capture what retinal and cortical circuits actually compute, and it did so before the circuits were understood. The top-down analysis was not idle philosophy; it predicted the biology. This is the strongest single piece of evidence for the metaphor the overview resists, and an honest treatment has to put it on the table rather than hide it. There are problems — edge detection, stereo correspondence, structure-from-motion — where the abstract task is so tightly constrained by the physics of the world that the computation is nearly forced, and knowing the computation tells you, in advance, roughly what any solving system must do. For those problems, the levels really do separate, and the architecture-first slogan would have been a worse guide than Marr’s top-down march.
A defender of the metaphor can press the point further, and should. The reification critique — the charge that cognitive neuroscience erred by treating “attention” or “working memory” as things to be found in the tissue — proves less than it seems. Yes, Baddeley inferred the phonological loop from reaction times rather than reading it off neurons, and yes, it is a logical leap to assume that a construct derived from behavior must have a tidy neural address. But notice that the leap sometimes lands. The construct “place cell” was inferred from behavior and lesion data and then found, gloriously, in the hippocampus, firing for locations exactly as predicted. The construct “grid” was posited and then discovered in entorhinal cortex. If behavioral constructs never mapped onto identifiable neural mechanisms, spatial-memory research would have been a dead end; instead it is the field’s crown jewel. So the localizationist is entitled to a sharp rejoinder: you cannot reject the whole program of inferring neural mechanism from behavioral construct, because that program has produced the best-validated results in systems neuroscience. The question is not whether constructs ever map onto tissue. They demonstrably do. The question is which constructs, and how would you know in advance.
That is the steelman. It is not a paper tiger. If the book’s framing cannot survive it, the framing should go.
51.4 Where the strong version breaks
It does not survive — but the reason is more specific, and more interesting, than “the brain isn’t a computer.”
The multiple-realizability argument smuggles in an assumption that holds for the systems it was invented to describe and fails for the brain: that there is a clean factorization between what is computed and how. In a digital computer that factorization is real and engineered. Software is portable precisely because the hardware was designed to enforce a layer of abstraction — an instruction set — that hides the physics from the program. The separation of data from processing is not a discovery about computation in general; it is a design choice, made by engineers, and paid for in silicon. The whole force of the metaphor depends on treating an artifact’s deliberate architecture as if it were a law of nature.
Neural tissue was not designed to enforce that layer, and overwhelmingly does not. This is where the overview’s four observations stop being a list of disanalogies and become a single structural point: in the brain, the implementation is the algorithm, because the physical dynamics are not hidden behind an abstraction layer that could in principle be swapped out. The ring of heading cells does not implement an algorithm for integrating angular velocity that could, with equal fidelity, be implemented some other way in the same animal. The ring’s geometry is the integration; change the connectivity and you do not get the same computation on different hardware, you get a different computation. When the substrate cannot be varied without varying the function, the function was never an abstraction floating above the substrate in the first place. Multiple realizability is true across species and artifacts — fly and rat and silicon can all do path integration — but it is conspicuously false within a single brain, where the wiring is not one contingent implementation among many but the only one there is, and the one selection actually shaped. The level of abstraction at which the fly and the rat “do the same thing” is so coarse that it has thrown away nearly everything a neuroscientist wants to explain.
This is also the precise point at which Marr’s success becomes misleading rather than encouraging. Edge detection works top-down because the world’s structure constrains the task almost to uniqueness — there is essentially one right answer to “where are the luminance discontinuities,” and any system solving it must converge on something like the same computation. But most of what brains do is not like that. The task is not handed down by the physics of the world; it is invented by the organism as one of many adequate ways to stay alive, and there is no task specification sitting in the environment waiting to be discovered. What problem is the basal ganglia “solving”? Action selection, we say — but action selection is not a well-posed problem in the way edge detection is. It has no unique optimal solution dictated by external structure; it is a negotiated, evolved compromise among competing drives whose very set points are themselves products of selection. For ill-posed, organism-defined problems — which is to say, for most of cognition — there is no computational-level description that floats free of the architecture, because the architecture is what defines the problem in the first place. Marr’s top-down method works exactly where the world is rigid enough to specify the task, and runs out of grip everywhere else. The mistake is not that Marr was wrong; it is generalizing from his best case to the whole brain.
51.5 The levels problem, restated without the slogan
We can now say what is right in the “category error” charge without the overreach. The error cognitive neuroscience sometimes commits is not believing that behavioral constructs can map onto neural mechanisms — they sometimes can, as place cells prove. The error is assuming this as a default, treating the box-and-arrow diagram as a hypothesis about brain organization when it was only ever a compact summary of behavior, and then organizing the search around finding the boxes.
The strongest rebuttal to localization is not philosophical but empirical, and it is the rebuttal of double dissociation, the most rigorous tool the localizationists ever built. The logic is genuinely elegant: if damage to region A abolishes function X but spares Y, while damage to B does the reverse, then X and Y must be separable processes with separable substrates. Cognitive neuropsychology — Warrington, Shallice, and the Queen Square tradition — turned this into a precise science, and it is not to be sneered at. Selective deficits are real. A patient can lose the ability to hold verbal material across a delay while retaining the visuospatial equivalent, and that is evidence of some real separation in the machinery.
But here is the subtle thing the method cannot see past, and it is the whole point. A double dissociation tells you the machinery separates somewhere; it does not tell you that it separates along the lines your psychological vocabulary drew. That a “verbal” and a “spatial” maintenance deficit can be pulled apart is consistent with there being two control systems — an auditory–motor loop and a spatial-navigation system — that happen to support temporary maintenance as a side effect, and that were never, in the brain’s own organization, instances of a single category called “working memory” at all. The dissociation is real; the inference that it validates the construct is the leap. The method perfected the mapping between a psychological theory and the anatomy while leaving wholly untouched the prior question of whether the psychological theory carved the machinery at its joints or across them. It is rigorous and conceptually trapped at once — the cleanest possible demonstration that you can do everything methodologically right and still be held captive by the level you started from.
This is why the architecture-first move is not a rejection of rigor but a relocation of the starting point. Begin instead with problems the organism demonstrably faces — keep the body fueled, get to the cached seed, orient to the looming shadow — ask what control mechanism would meet each, and let the behavioral regularities fall out as consequences of the mechanism rather than as primitives the mechanism must contain. On this approach the traditional constructs do not vanish; they are demoted from causes to effects. “Working memory” becomes a name for whatever sustained activity bridges a temporal gap in some control loop, with no commitment that all such bridging is one thing. “Attention” becomes the dynamics that arise when a controller must commit shared effectors to one option among competitors, with no commitment that there is an attention system doing it. The constructs survive as descriptions of emergent properties; what they lose is their assumed status as components.
51.6 What I am not claiming
It would be a poor essay that demolished the strong metaphor only to install an equally unhedged opposite. Several concessions are owed, and making them is the difference between a position and a slogan.
First, computational language is often the most precise tool available, and refusing it on principle would be a kind of vandalism. When we say grid cells perform path integration, or that a circuit approximates a Bayesian estimate, we are saying something true and useful about what the dynamics accomplish. The error is never in using the vocabulary; it is in the further inference that the brain therefore contains an explicit representation of the integral, or a stored prior, that some module fetches and manipulates. Describe freely. Reify never. That is the whole of the discipline being asked for.
Second, Marr’s levels are a real intellectual achievement and a good discipline even where they are not a good ontology. Asking “what problem might this circuit be solving, and why would that problem matter to the animal” is an excellent question even if the answer turns out to be inseparable from the wiring — indeed, asking it is often how you discover that it is inseparable. The framework fails as a claim that the levels are independent layers of reality; it succeeds as a checklist that keeps you from mistaking a mechanism for its own sake. Keep the questions; drop the metaphysics.
Third — and this is the concession that should keep an honest reader up at night — the architecture-first program has not yet delivered across the board, and pretending otherwise would be exactly the kind of just-so confidence this book warns against elsewhere. It has one undisputed triumph, spatial navigation, where the evolved control problem, the architecture, the dynamics, and the behavior all line up into a single story with no psychological middleman required. For most of the rest of cognition the evolutionary-control account is a promissory note, not a finished theory. We do not have a worked-out architecture-first story of language, or of episodic recollection, or of abstract reasoning, that rivals the precision of the place-cell account. It is entirely possible — you should hold this open — that for some capacities the computational, construct-first approach will simply work better, because those capacities really are, in the relevant respects, more like portable software than like a heading-ring. The bet of this book is that they are not. But it is a bet, the evidence is not yet in, and a reader who walks away certain has learned the wrong lesson from a chapter whose entire moral is the value of sitting with what we do not know.
51.7 The residue
Strip away the rhetoric on both sides and a modest, defensible position remains. The brain is not a von Neumann machine; on that the biology is decisive and the naive metaphor is simply false. But the deeper opponent was never the naive metaphor — it was the claim that computation is an abstraction floating free of the wiring, the same in any substrate that honors the input–output function. That claim is true of artifacts engineered to make it true, and false of tissue that was not, because in tissue the wiring is not one implementation among many but the only one selection built and the one the dynamics are. Where the world specifies a task tightly enough — early vision is the standing example — the top-down, levels-separating method works and works beautifully, and the book’s framing would mislead you there. Where the organism defines its own ill-posed problems, which is most of the time, the levels fuse, the construct-first search goes hunting for boxes that need not exist, and starting from the evolved architecture is the better discipline. The honest summary is not “the brain is not a computer.” It is: the separation of what from how is an engineering achievement, not a fact of nature, and the brain mostly did not bother to achieve it. Everything else in the argument is commentary on that one sentence — including the standing possibility that, for some corner of cognition we have not yet cracked, the sentence is wrong.