35  What It’s Worth

How Consequences Reweight Future Action

35.1 From selecting now to learning for next time

The preceding chapter (Chapter 34) described how basal-ganglia loops regulate access to action. They help one set of cortical and brainstem controllers gain influence, keep incompatible actions from interfering, scale the vigor of the selected action, and reorganize control when circumstances change. That account explains how several current possibilities become one course of behavior. It leaves open a second problem: why the competition has the weights it does.

An action changes the body and world. A reach obtains the cup or knocks it over. A turn reveals the source of a sound. Opening one door produces food; opening another produces nothing. Some consequences restore a regulated variable, some cause injury, and some provide new information. Once those consequences have occurred, the same actions should not enter the next competition with their original weights.

The change can be specific. Finding food in the pantry should make opening the pantry more likely when the same need returns. It should not strengthen every action that occurred during the evening. The change must also remain flexible. The food can still be remembered when satiety makes it temporarily unimportant. A familiar route can still be avoided after the destination becomes dangerous. Repeated practice can eventually produce the opposite problem: an action may be triggered so reliably by its context that it persists after the outcome that established it has lost its value.

These are problems of learning from consequences. Reinforcement occurs when a consequence increases the future probability of the behavior that produced it. Reinforcement-learning models explain how experience updates predictions and action selection. The biological problem includes learning what an action produces, assigning credit to the relevant activity, and using that knowledge under changing bodily conditions.

Consequences reweight future action. Dopamine-related prediction errors help update recently active corticostriatal circuits; current bodily state changes how learned outcomes matter now; and repeated experience can shift relative control from outcome-sensitive action toward cue-triggered habit.

Discussions of reward and value involve several experimentally distinguishable variables:

Term The question it answers
Outcome prediction What is likely to happen?
Reward-prediction error Was the outcome better or worse than expected?
Incentive value How worth pursuing is that outcome in the organism’s present state?
Hedonic impact How pleasurable or aversive is the outcome when it is experienced?
Learned action weight How strongly should this action be favored in this context?

These variables interact, but they are not interchangeable. A cue can predict food accurately when an animal is sated. An action can still be known to produce an outcome that is no longer wanted. A dopamine transient can teach without being the pleasure produced by the outcome. A habitual action can remain strong even when the outcome’s current incentive value has fallen.

The chapter begins with the conserved vertebrate circuitry in which dopamine modulates action. It then develops the teaching signal, the synapses it changes, and the behavioral tests that distinguish learned prediction, current motivation, and habit. Together, these mechanisms explain how experience changes the conditions under which an embodied controller will act again.

35.2 Dopamine modulates an ancient action system

The lamprey evidence in Chapter 34 established a conserved basal-ganglia scaffold with access to tectal and brainstem controllers. Dopamine participates in that scaffold. The comparative question is how far back its effects on the two principal striatal pathways can be traced.

In mammals, direct-pathway spiny projection neurons are enriched in D1 dopamine receptors and indirect-pathway neurons in D2 receptors. Work from Sten Grillner’s group identified corresponding populations in lamprey. Dopamine increased the excitability of direct-pathway neurons and decreased the excitability of indirect-pathway neurons [@ericssonetal2013dopamine]. This receptor organization and its opposing physiological effects were present long before the expansion of mammalian neocortex.

Lamprey dopamine neurons also reach the motor hierarchy by more than one route. An SNc/VTA-like population projects to striatum and directly to the optic tectum, where dopamine alters the visuomotor transformations supporting orienting and avoidance [@perezfernandezetal2017tectum]. Some individual neurons branch to both structures [@vontwickeletal2019corelease]. Dopamine can therefore influence the basal-ganglia circuits regulating access and the tectal circuitry organizing the response.

Mammalian elaboration linked dopaminergic modulation to expanded cortical, limbic, and mnemonic networks. Ventral, associative, and sensorimotor striatal territories receive different combinations of information about objects, contexts, rules, actions, and physiological state. Dopamine changes their responsiveness and plasticity, affecting both current performance and later behavior.

The comparative evidence establishes ancient modulation of action-related circuits. Reward-prediction-error coding, developed below, is a computational account tested in other experimental preparations. The connection between them is the chapter’s organizing principle: dopamine changes how information and experience influence action.

Figure 35.1: Dopamine modulates a conserved vertebrate action system. A: Lamprey dopamine neurons in an SNc/VTA-like population project to striatum and optic tectum; some individual neurons innervate both. Dopamine increases the excitability of D1-enriched direct-like striatal neurons and decreases that of D2-enriched indirect-like neurons. Striatal activity influences inhibitory pallidal/nigral output, while direct dopamine projections modulate tectal visuomotor circuitry. These routes regulate access to orienting, avoidance, and locomotor controllers. B: Mammalian elaboration links ventral, associative, and sensorimotor striatal territories to expanded cortical networks and to amygdalar, hippocampal, thalamic, and bodily-state information. Basal-ganglia output retains access to superior-collicular and brainstem systems. The comparison establishes conservation of action-related modulation; reward-prediction-error coding is tested in the learning experiments described later. Blue arrows denote dopamine projections, conventional arrowheads excitation, blunt-ended lines inhibition, and green and red symbols increased and decreased excitability. The mammalian connections summarize interacting systems rather than individual monosynaptic pathways [@ericssonetal2013dopamine; @perezfernandezetal2017tectum; @vontwickeletal2019corelease].

35.3 Three things an animal can learn

Learning from an outcome requires more than remembering that something good or bad occurred. The nervous system must preserve different kinds of relation. Three are especially important here.

35.3.1 A cue can predict an outcome

In Pavlovian learning, an event predicts an outcome whether or not the animal’s action produces it. A sound may precede food. A place may predict danger. A light may indicate that water will soon become available. Once the relation has been learned, the cue can orient attention, evoke anticipatory physiology, and prepare actions appropriate to the expected outcome.

Prediction does not imply desire. A sated animal can continue to recognize a cue as a reliable predictor of food. A person can know that an alarm predicts an aversive event without wanting the event. Pavlovian learning establishes what the cue forecasts.

35.3.2 An action can cause an outcome

In instrumental learning, the animal learns a relation between what it does and what follows. Pressing one lever produces sucrose; pressing another produces a food pellet. Turning left reaches shelter; turning right reaches exposed ground. The consequence depends on the action.

This relation contains more information than a generalized history of reinforcement. It can preserve the identity of the outcome and the causal structure of the action. An animal can therefore reduce the action producing one devalued food while preserving another action that produces a still-desirable food. That result will become the chapter’s principal test of flexible action.

35.3.3 The current body changes what an outcome is worth

The learned identity and probability of an outcome do not determine its present incentive value by themselves. Water has a different claim on action during dehydration. Salt can change from aversive to intensely attractive during sodium depletion. Food loses much of its immediate pull after sensory-specific satiety while remaining recognizable and remembered.

The nervous system therefore combines learned outcome representations with the current state of the organism. Hypothalamic and brainstem signals, hormones, visceral feedback, and insular and amygdalar activity change how cues and outcomes influence action. The regulated body helps determine which learned opportunities deserve pursuit.

The three relations can change independently. A cue may continue to predict an outcome after the outcome has been devalued. An animal may retain an action–outcome relation while no longer performing the action because the outcome is unwanted. A habit may continue after both the current incentive value and the action–outcome contingency have changed. The behavioral preparation matters because each pattern supports a different conclusion.

35.3.4 Credit must be assigned across more than one timescale

Consider a hungry animal that leaves its nest, crosses a room, searches an empty location, opens a container, and finally finds food. Several temporal problems are embedded in that episode.

At the local synapse, cortical and thalamic activity in the striatum may last only briefly. If a dopamine signal arrives a second later, the recently active connections need a temporary state that marks them as eligible for change. At the level of the sequence, predictive significance must propagate from the eventual food toward earlier places and actions over repeated experience. Across still longer intervals, the physiological consequences of eating unfold over minutes or hours and must be related to remembered episodes, internal-state changes, and intermediate predictors.

No single mechanism solves all three delays. A synaptic eligibility trace helps bridge a short interval between recent neural activity and a teaching signal. Temporal-difference learning describes how predictive information can propagate across a sequence of states. Long-horizon learning also depends on memory for events and contexts, reactivation of prior experience, hierarchical organization of action, and repeated contact with intermediate consequences.

The mechanisms complement one another. A brief eligibility trace preserves a local opportunity for plasticity; predictive learning links successive events; memory retains the larger episode. Together they allow consequences at different delays to influence later action.

35.4 Unexpected outcomes revise prediction

Simple co-occurrence cannot explain what animals learn. The decisive demonstration is blocking, introduced by Leon Kamin [@kamin1969predictability].

An animal first learns that a light predicts an outcome. The outcome may be a shock in the original aversive preparation or food in an appetitive version. Once the light is a reliable predictor, a tone is added. Light and tone now occur together before the same outcome. The tone has been paired with the outcome repeatedly, yet it acquires little predictive control. The established light already accounted for what happened. The new tone supplied no error that required the prediction to be revised.

Blocking shows that associative change depends strongly on what the outcome adds to the current prediction. It is not enough that two events occur together. A redundant cue can be present on every trial and still acquire little control.

Rescorla and Wagner formalized this principle by proposing that associative strength changes according to the discrepancy between the outcome obtained and the outcome predicted by all cues present [@rescorlawagner1972theory]. A larger positive discrepancy produces greater strengthening. An outcome smaller than expected produces a negative discrepancy. When prediction and outcome match, little updating is required.

The prediction error is not subjective astonishment. It is a computational difference. A tiny, unnoticed change can generate an error if it violates a precise expectation; a dramatic event can generate little new learning if it was fully predicted.

The Rescorla–Wagner model treats each trial as a unit. It does not represent the successive moments between cue and outcome. Temporal-difference learning extends the same principle through time [@sutton1988temporal]. At each moment, the system compares its present estimate of future outcome with the reward and prediction available at the next moment. A cue that improves the forecast generates a positive error before the final outcome arrives. With repeated experience, predictive significance can move toward earlier states in the sequence.

Opening a container and seeing food therefore matters immediately. The sight of food raises the estimate of what is about to occur, even before ingestion changes physiology. Earlier cues can acquire predictive significance because they lead to that improved state. Learning proceeds by successive changes in expectation rather than by one signal spanning the entire delay.

Temporal-difference models can learn the expected return associated with a state, often written as V, or the expected return associated with taking an action in a state, often written as Q. Applying the model to an organism requires specifying what information defines its current state, which actions it distinguishes, and what consequences it predicts. Those representations determine what the prediction error can teach.

The Rescorla–Wagner update can be written in simplified form as:

\[ \Delta V = \alpha\beta(\lambda - \sum V) \]

The change in associative strength, \(\Delta V\), depends on learning-rate terms and on the difference between the obtained outcome, \(\lambda\), and the outcome predicted by all cues, \(\sum V\). Blocking occurs because the established cue already predicts the outcome, leaving little error for the added cue.

A temporal-difference error can be written as:

\[ \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \]

The error at time \(t\) compares the current prediction with immediate reward, \(r_t\), plus the discounted prediction at the next state. A positive error means that prospects improved; a negative error means that they worsened. The same framework can be written for action values, \(Q(s,a)\), and combined with eligibility traces that allow several recently visited states or actions to be updated.

Here \(V\) denotes the learned prediction used by each model; in the temporal-difference equation it is the expected discounted return from a state. The discount factor, \(\gamma\), sets the weight given to future return. The earlier chapter How a Synapse Learns develops the synaptic and computational machinery in greater detail.

Reward-prediction error explains a major class of learning from consequences. Perceptual learning, motor adaptation, imitation, and learning about environmental structure involve other errors and teaching relations. An animal can, for example, learn the layout of a place while exploring it before any food is found there. The reward-based account developed here explains how valued consequences revise predictions and action weights within that broader capacity to learn.

The models now make a mechanistic demand: some neural activity must reflect the discrepancy at a time when it can change learning. Midbrain dopamine recordings provided a striking candidate.

35.5 Dopamine can supply a teaching signal

The computational account became a neural account when recordings from midbrain dopamine neurons revealed a striking temporal pattern. Wolfram Schultz and colleagues trained monkeys in tasks in which a sensory cue predicted a drop of juice. Many dopamine neurons in substantia nigra pars compacta and ventral tegmental area responded according to three conditions [@schultzdayanmontague1997].

When juice arrived unexpectedly, the neurons produced a brief increase in firing. After learning, the main increase occurred at the earliest reliable cue rather than at delivery of the now-predicted juice. When expected juice was omitted, firing fell below its ongoing level at the time the juice should have arrived.

These responses resemble a signed reward-prediction error:

  • better than expected produces a positive response;
  • as expected produces little outcome response;
  • worse than expected produces a negative response.

The cue response does not literally travel backward from the reward. Learning changes when the discrepancy occurs. Early in training, the juice improves the forecast. Later, the cue produces the improvement, while the predicted juice adds little.

The correspondence extends beyond the three-part illustration. In monkey blocking experiments, learning and dopamine-neuron responses depended on whether an outcome supplied new information [@waeltietal2001blocking]. If dopamine supplies a teaching signal, adding activity when an outcome is already predicted should make an otherwise redundant cue learnable.

35.5.1 Supplying the missing teaching signal

Steinberg and colleagues tested that prediction in rats. An auditory cue first signaled that sucrose was available at a reward port. During compound training, a visual cue was added while the sucrose remained unchanged. The established auditory predictor normally blocked learning about the new cue [@steinbergetal2013causal].

The investigators used optogenetics to activate VTA dopamine neurons briefly during sucrose delivery. Later, the visual cue was tested alone, without sucrose or optical stimulation. Rats that had received stimulation paired with sucrose spent more time at the reward port during that cue. The same stimulation delivered between trials did not overcome blocking.

The crucial result appeared after the intervention had ended. Dopamine activation at the outcome changed what the rat learned about the cue; it did more than increase movement during stimulation. The timing of the added signal supplied the learning opportunity missing from ordinary blocking.

Dopamine-related errors also extend to instrumental sequences. Hollon and colleagues found that nigrostriatal dopamine signals depended on violations of sequence-specific action–outcome predictions [@hollonetal2021sequence]. This connects the teaching account to the chapter’s central problem: consequences change the future recruitment of particular actions in particular circumstances.

35.5.2 A brief signal can modify recently active synapses

The teaching signal must affect the connections involved in the preceding event. The three-factor rule introduced in How a Synapse Learns supplies a mechanism: presynaptic activity and postsynaptic activation create a local eligibility state, and a subsequent modulatory signal determines whether lasting change occurs.

Yagishita and colleagues tested the timing directly in mouse nucleus-accumbens slices. They examined dendritic spines on D1-receptor-expressing projection neurons, using two-photon glutamate uncaging to stimulate a selected spine, paired with postsynaptic spikes evoked through a recording electrode. They independently stimulated dopamine axons with light [@yagishitaetal2014window].

Dopamine promoted spine enlargement when it followed the glutamatergic input within a window of approximately 0.3–2 seconds. Delaying the same dopamine stimulation beyond that window greatly reduced its effect. Recent local activity had created a brief opportunity for dopamine to stabilize structural plasticity.

This supplies a cellular mechanism for short-delay credit assignment. Dopamine can reach many connections, while recent local activity determines which connections are eligible for this form of plasticity.

Dopamine acts through receptor-specific intracellular pathways whose effects depend on the activity and signaling state of the receiving neuron. Experiments in identified D1- and D2-expressing striatal neurons show that plasticity can be bidirectional and is regulated differently in the two populations [@shenetal2008plasticity]. Eligibility therefore marks which recent connections can change, while local activity and receptor signaling shape whether they strengthen or weaken. The spine experiment demonstrates one form of strengthening in D1 neurons. The broader learning problem requires selective changes that can increase or reduce the future influence of particular inputs.

The two experimental levels fit together. The blocking experiment shows that timed dopamine activation changes later cue control. The spine experiment shows how a delayed modulatory signal can act selectively on recently active connections. Temporal-difference learning extends the account across repeated encounters, as earlier predictors acquire control over expectation.

35.5.3 Dopamine signals differ across place and time

Reward-prediction-error coding provides a strong account of learning in the tasks just described. Recordings across dopamine populations also reveal how the system supports ongoing action. Neurons differ in their inputs and targets, and local striatal circuitry shapes release from their terminals.

During learning, dopamine release in nucleus accumbens can shift from an unexpected reward toward its predictive cue [@dayetal2007shifts]. In mice moving through a virtual environment, different dopamine axons carried rapid locomotion-related and reward-related signals [@howedombeck2016]. Recordings of dopamine and acetylcholine across striatal territories likewise reveal regionally distinct timing during reward-guided behavior [@duhneetal2024mismatch]. The location and measurement therefore matter: firing at the cell body, calcium in an axon, and dopamine concentration at a target describe linked steps in transmission.

The preceding chapter developed the contribution of nigrostriatal dopamine to initiation and vigor (Chapter 34). Experiments also show that striatal dopamine signals integrate reward benefit, effort already invested, and motivational state [@esheletal2024cost]. These influences can change pursuit on the current occasion, while plasticity changes the response on later occasions.

Dopamine helps turn experience into learning and expected benefit into action. The teaching-signal account explains how timed changes modify predictions and action weights. The movement and motivational findings explain how dopaminergic modulation helps express those weights in behavior.

Figure 35.2: Prediction errors and synaptic eligibility solve complementary learning problems. A: An unexpected reward produces a transient increase in dopamine-neuron firing. B: After learning, the earliest reliable cue produces the principal increase, while the predicted reward adds little response. C: Omission of the expected reward produces a decrease at its expected time. These schematic responses illustrate signed reward-prediction error: better than expected, as expected, and worse than expected [@schultzdayanmontague1997]. D: Within a trial, recent pre- and postsynaptic activity leaves selected connections eligible for plasticity. A subsequent dopamine signal can alter their weights, \(\Delta w\). In D1-expressing accumbens neurons, dopamine promoted spine enlargement within approximately 0.3–2 seconds after glutamatergic input [@yagishitaetal2014window]. E: Across repeated trials, temporal-difference learning allows predictive significance to develop at earlier states in a sequence. Seeing food, opening its container, and encountering an earlier predictor can successively improve expectation. The prediction changes through learning; the signal does not travel backward in time. Local eligibility preserves a short-lived opportunity for synaptic change, while learning across successive states connects more distant predictors to the outcome [@sutton1988temporal].

35.6 Learning is distributed across a striatal gradient

Dopamine modulates striatal territories with different connectional biases. These territories form gradients through which information about outcomes, context, and bodily state can influence associative and sensorimotor control.

The ventral striatum includes the nucleus accumbens and adjoining portions of caudate and putamen [@zahmbrog1992accumbens]. The accumbens is commonly divided into core and shell. The boundary is anatomically useful, and the two territories differ in their inputs, outputs, neurochemistry, and contributions to behavior. The shell is especially connected with medial prefrontal, amygdalar, hippocampal, hypothalamic, and brainstem systems. The core has strong relations with anterior cingulate, orbitofrontal, amygdalar, thalamic, and motor-related systems. Their contribution becomes clearer when experiments separate the behavioral information each task requires.

Farther dorsally and laterally, striatal input becomes increasingly biased toward associative and sensorimotor cortex. In rodents, posterior dorsomedial territories are especially important for using action–outcome relations. Dorsolateral territories receive dense sensorimotor input and become prominent when familiar contexts trigger well-practiced actions. The transition is graded. Cortical and thalamic projections overlap, and the same action can engage ventral, associative, and sensorimotor territories at different phases of learning and performance.

This organization is suited to the chapter’s problem. An odor can predict a particular food. The hippocampus can represent where that food was found. The amygdala can represent its biological significance. The insula can represent its sensory and visceral consequences. Hypothalamic and brainstem systems can signal current metabolic and hormonal state. Cortical systems can represent the available actions and the rules relating them to outcomes. These influences converge in partially overlapping striatal populations, allowing the significance of an opportunity to change with context and need.

The major output of nucleus accumbens reaches the ventral pallidum, which in turn communicates with mediodorsal thalamus, hypothalamus, lateral habenula-related circuitry, midbrain, and brainstem. Accumbens neurons also project to midbrain dopamine regions and other targets. Ventral-striatal activity can therefore influence cortical action systems, autonomic and endocrine control, orienting, locomotion, and the dopamine signals that reshape later learning. The circuit is recurrent from the beginning.

The distinction between core and shell becomes experimentally useful when current outcome value and the influence of predictive cues are tested separately. The devaluation and transfer experiments below show how damage to these territories changes different sources of control over the same instrumental behavior.

The ventral circuit also modifies the dorsal pathway arrangement introduced in Chapter 34. In dorsal striatum, D1-enriched neurons project mainly toward output nuclei and D2-enriched neurons mainly toward external pallidum. In nucleus accumbens, both receptor classes project substantially to ventral pallidum [@kupchiketal2015coding]. D1 and D2 identity therefore remains important for dopaminergic modulation but does not specify the same direct–indirect division of output routes.

Ventral striatum is a major interface through which learned predictions, current bodily state, context, and reward-related cues alter the control of action. Associative and sensorimotor striatal territories contribute other information and become increasingly influential as action–outcome relations are established and behavior is practiced. Learning is distributed across the gradient because natural behavior requires the whole relation: what is present, what can be done, what will follow, and what that outcome means now.

35.8 The decisive test: does action depend on its outcome?

A reinforced action can be strong for at least two reasons. The organism may perform it because it represents the outcome and currently wants that outcome. Or the context may trigger the action because repetition has strengthened the relation between situation and response. Ordinary performance often looks the same in both cases. The distinction appears only when the outcome or contingency changes.

The clearest test uses two actions and two outcomes. A rat learns that pressing one lever produces sucrose and pressing another produces a food pellet. Both actions become frequent. One outcome is then devalued, commonly by allowing the animal to consume it to satiety before the test. The animal is returned to the chamber, but the levers are tested briefly in extinction so that no new food is delivered.

An outcome-sensitive animal selectively reduces the action that had produced the devalued food. It continues performing the action that produced the other outcome. This result is more informative than a general decline in responding. The animal must preserve the identity of both action–outcome relations, update or retrieve the current incentive value of one outcome, and use that combination to guide action without receiving the outcome during the test [@balleinedickinson1998goal].

The procedure separates knowledge from present motivation. Selective satiety does not need to erase the memory that a lever produces sucrose. The animal may continue to approach or consume sucrose later when hunger returns. What changes during the test is the current claim of that outcome on action.

Devaluation can also be produced by pairing one outcome with illness. The sensory identity of the outcome then predicts an aversive consequence. Again, selective reduction of the corresponding action shows that performance depends on an internal model of what the action produces, not merely on the number of prior reinforcements.

Several brain systems are required to make this flexibility possible. Posterior dorsomedial striatum is important for acquiring and expressing action–outcome control. Lesions there impair sensitivity to both outcome devaluation and changes in the causal contingency [@yinetal2005dms]. Insular cortex is important for retrieving the current incentive value of a specific sensory outcome. Animals with insular lesions can learn the action–outcome relation yet fail to adjust performance after the outcome is revalued through selective satiety [@balleinedickinson2000insula]. Accumbens lesions further separate the use of outcome value from the influence of predictive cues.

Corbit, Muir, and Balleine trained rats on two lever–outcome relations and tested the effect of selectively devaluing one outcome. Rats with accumbens-core lesions failed to reduce selectively the action producing the devalued outcome. Rats with shell lesions retained this selectivity [@corbitetal2001accumbens]. The result links core function to the use of current outcome value in instrumental performance. A different test, developed below, exposed an important contribution from shell.

These results identify different requirements for flexible action. The action is represented in corticostriatal networks. The sensory identity and remembered consequences of the outcome involve insular, amygdalar, orbitofrontal, and related systems. Current physiological state changes the response to that representation. Dorsomedial striatum helps bring the relation back into the competition among actions.

35.8.1 State can change a cue’s pull before new experience

Sodium appetite supplies a complementary test of how bodily state changes learned motivation. Concentrated salt is normally aversive, but sodium depletion makes obtaining salt biologically urgent. Can the new state change a learned cue’s influence before the animal tastes salt again?

Tindell and colleagues trained rats with auditory cues followed by intraoral infusions of either sucrose or concentrated salt. They recorded neurons in posterior ventral pallidum. After sodium depletion, responses to the salt-predicting cue increased on its first presentation, during an extinction test before any salt was delivered in the new state [@tindelletal2009dynamic]. The physiological need changed the neural response to a learned predictor before new experience with its outcome.

Robinson and Berridge demonstrated the behavioral transformation using a briefly presented lever as the Pavlovian cue for concentrated salt. Rats initially avoided the cue. On first encountering it during sodium depletion, they approached and engaged it, including gnawing, before experiencing the salt in that state [@robinsonberridge2013instant]. The lever predicted salt; pressing it was not the action required to obtain salt. Its changed attractiveness therefore exposed a change in cue-triggered motivation.

The two studies establish complementary results: a change in the neural response to the predictor and a change in attraction to the predictor itself. Learned information about salt remained available while current need changed its motivational effect.

This extends the devaluation argument. The lever-choice experiment tests whether instrumental action uses an outcome’s updated value. The salt-cue experiments show that a physiological change can transform Pavlovian motivation before the outcome is experienced again. Both demonstrate how the current body changes the use of past learning.

35.9 When control shifts toward habit

With repetition, behavior can become less dependent on the current outcome. This is the defining property of an instrumental habit [@dickinson1985actions].

A habit is an action whose performance has become relatively insensitive to changes in the current value of its outcome or to changes in the causal relation between action and outcome.

The definition concerns the information controlling performance. A person may execute a practiced movement fluently while remaining sensitive to its outcome. Conversely, a response can persist after devaluation even when its execution is awkward. Skill, awareness, and habit must therefore be measured separately.

35.9.1 Devaluation and contingency test different relations

Outcome devaluation asks whether the action still depends on the present desirability of its consequence. Contingency degradation asks whether the action still depends on causing that consequence.

Suppose a lever press initially produces food. Food is then delivered just as often without the press, weakening the causal advantage of acting. Goal-directed performance declines because the action no longer improves the probability of the outcome. Habitual performance can persist because the context continues to trigger the response even after the causal relation has been degraded.

An omission schedule reverses the causal relation: the outcome is delivered only if the animal refrains from the previously reinforced action. Persistent responding now postpones or prevents the outcome. If the action continues, its control cannot be explained by a currently useful action–outcome relation.

The diagnostic comparison is selective persistence. Two actions and two outcomes allow a researcher to distinguish loss of outcome sensitivity from a general change in movement or motivation. Tests of outcome discrimination and consumption establish that devaluation worked and that the animal still distinguishes the consequences.

35.9.2 Dorsomedial and dorsolateral striatum have different biases

Rodent dorsal striatum contains a corresponding anatomical gradient. Posterior dorsomedial striatum receives associative cortical input and is strongly involved in action–outcome learning. Dorsolateral striatum receives dense sensorimotor input and becomes increasingly important when familiar stimuli and action sequences control performance.

Lesions of dorsolateral striatum can preserve knowledge of an expected outcome while preventing the normal emergence of outcome-insensitive responding after extended training [@yinetal2004dls]. Reversible inactivation of dorsolateral striatum can restore sensitivity to a degraded action–outcome contingency in animals whose behavior had become habitual [@yinetal2006contingency]. Conversely, disrupting dorsomedial striatum impairs goal-directed control even when the animal can still move and obtain reinforcement [@yinetal2005dms].

The evidence supports a shift in relative control. Dorsomedial and dorsolateral circuits participate during the same periods of training, while their influence on performance changes. Gremel and Costa showed that orbitofrontal and striatal activity changed with the strategy supported by the task, and circuit manipulations altered the balance between outcome-sensitive and habitual performance [@gremelcosta2013shift].

The transition also depends on molecular regulation of plasticity. Histone deacetylase 3 (HDAC3) helps regulate gene expression by modifying the histone proteins around which DNA is packaged. Malvaez and colleagues found that reducing HDAC3 function in either dorsomedial or dorsolateral striatum accelerated habit formation, whereas increasing its expression prevented habit formation under the tested training conditions [@malvaezetal2018habits]. Molecular changes in both territories can therefore alter the emergence of habitual control. This provides a further connection between plasticity within the circuits and the changing influence those circuits exert on behavior.

The balance remains reversible. Context changes or circuit perturbations can restore outcome-sensitive performance. Learning can preserve action–outcome information even when a practiced response dominates behavior; changing the conditions changes which information gains control.

35.9.3 Sequence organization and outcome sensitivity are different properties

The preceding chapter described the emergence of sequence-boundary activity in nigrostriatal circuits (Chapter 34). Related recordings in rats learning a maze found that activity became concentrated near the beginning and end of the practiced sequence, forming a task bracket [@jogetal1999habits]. This organization can help recruit several movements as a unit.

Whether the sequence is habitual requires a further test. A pianist can execute a fluent phrase while adjusting it to the intended sound. A driver can perform a practiced gear change while responding to traffic. Chunking concerns the organization of the sequence; habit concerns its sensitivity to outcome value and causal contingency. The same learned sequence can participate in either form of control.

Human experiments can apply the outcome test directly. Tricomi and colleagues used extended instrumental training and outcome devaluation, finding reduced outcome sensitivity alongside increasing engagement of posterior putamen [@tricomietal2009human]. The combination of behavior and imaging links the animal literature to human action. Posterior-putamen activity alone would identify engagement during practice; devaluation establishes what information governs the practiced response.

Human experiments also place practiced responses under the influence of instructions and explicit knowledge. These can sustain outcome-sensitive performance, while laboratory training usually samples a much shorter history than years of everyday repetition. Interpreting a human habit test therefore requires attention to both the training history and the information available at test.

Computational reinforcement learning often distinguishes model-based control from model-free control. A model-based agent uses a representation of transitions and outcomes to evaluate possible action sequences. A model-free agent learns cached action values from prediction errors without explicitly simulating the transition structure.

The distinction resembles the behavioral contrast between goal-directed action and habit. Outcome-sensitive action normally requires knowledge of what the action produces, whereas a habitual response can be supported by a cached relation between situation and action. The pairs are not exact synonyms. A model-based computation need not be conscious or slow. A model-free value can remain sensitive to some motivational changes. Behavioral habit is defined by performance after devaluation or contingency change, not by fitting one computational model.

The two vocabularies answer related questions at different levels. The behavioral tests establish what information controls the action. The computational models propose how that information might be learned and used.

Figure 35.4: Behavioral tests distinguish goal-directed action from instrumental habit. A: Two actions earn different outcomes. One outcome is then devalued by selective satiety or, in an alternative procedure, by pairing its consumption with illness. A brief extinction test withholds both outcomes. Selectively reducing the action that previously earned the devalued outcome demonstrates use of its updated value. Habitual performance is relatively insensitive to that change, provided that devaluation worked and the animal still distinguishes the outcomes [@balleinedickinson1998goal]. B: An omission schedule tests sensitivity to causal consequences: pressing now delays food, while withholding the response permits delivery. Continued responding works against obtaining the outcome. C: Posterior dorsomedial striatum (pDMS, blue) and dorsolateral striatum (DLS, red) summarize the differing biases identified chiefly in rodent experiments. pDMS contributes strongly to action–outcome control; DLS contributes strongly to practiced, outcome-insensitive performance. Training and task conditions change their relative influence [@yinetal2004dls; @yinetal2005dms; @yinetal2006contingency; @gremelcosta2013shift]. D: Sequence-boundary activity can become prominent with practice, forming a task bracket around the component movements [@jogetal1999habits; @jincosta2010startstop]. This concerns sequence organization; devaluation or contingency tests are needed to establish habit. All bars and traces are schematic. Awareness, skill, movement speed, and outcome sensitivity are distinct properties.

35.10 Learning, wanting, and liking can separate

The word reward can refer to several operations. An outcome can teach the actions and cues that precede it. A cue can make the outcome worth pursuing now. Consumption can produce positive hedonic reactions. These processes normally cooperate, but experiments can separate them.

35.10.1 Learning changes future weights

The teaching experiments explain how outcomes change the predictions and connections governing later behavior. The resulting knowledge can remain available across changes in appetite and opportunity. A food predictor can still be recognized during satiety; an action–outcome relation can still be retrieved after a period without performing the action. Learning supplies a history that current conditions can put to different uses.

35.10.2 Cues can acquire motivational pull

A reward-predictive cue can attract attention, evoke approach, and invigorate actions that obtain its outcome. Berridge and colleagues call this cue-triggered motivational property incentive salience, or wanting [@berridgerobinson1998].

A useful assay is Pavlovian-instrumental transfer, or PIT. A cue is paired with an outcome independently of the animal’s action. Separately, the animal learns an instrumental response. During a later extinction test, the cue changes performance of that response even though the outcome is withheld. Prior learning has given the cue access to ongoing action.

Wyvell and Berridge trained rats with a sucrose-predictive cue and a sucrose-earning lever response. Infusing amphetamine into nucleus accumbens increased the cue’s ability to provoke lever pressing during an extinction test. The manipulation did not increase positive taste reactions to sucrose [@wyvellberridge2000wanting]. The experiment separated stronger cue-triggered pursuit from greater hedonic impact during consumption.

The core–shell comparison introduced earlier becomes informative here. In Corbit and colleagues’ study, shell lesions spared selective outcome devaluation but abolished the tested transfer effect; core lesions produced the opposite pattern [@corbitetal2001accumbens]. Using current outcome value and responding to a predictive cue placed different demands on the accumbens.

Corbit and Balleine subsequently separated two forms of transfer. Rats learned that three sounds predicted three different foods, and independently learned two actions earning two of those foods. In extinction, a cue could preferentially increase the action earning its own predicted food: outcome-specific transfer. A cue predicting the third food could increase responding more generally: general transfer [@corbitballeine2011transfer].

Shell lesions eliminated the specific effect while sparing general invigoration; core lesions eliminated the general effect while sparing the specific one. Inactivating the regions just before testing produced the corresponding dissociation. Thus a predictive cue can both direct action toward a particular consequence and increase the readiness to act. The more precise task explains why a broad statement that either core or shell simply mediates motivation is insufficient.

Current state regulates these influences. Hunger increases the pull of food cues; sodium depletion can transform the salt cue before new consumption. The vigor account in Chapter 34 describes how motivation reaches movement: it changes the probability, latency, speed, and persistence of pursuit. Wanting is expressed through action.

35.10.3 Hedonic impact has partly different causal machinery

Liking refers to the positive hedonic impact of an outcome during experience. In rodents, taste-reactivity experiments measure patterned orofacial responses to sweet and bitter infusions. Positive tongue movements and aversive gapes provide behavioral measures that can be compared with food intake and pursuit.

Peciña and Berridge mapped the effects of stimulating mu-opioid receptors within nucleus accumbens shell. In a small rostrodorsal part of medial shell, stimulation increased positive reactions to sucrose. Stimulation elsewhere could increase eating without the same hedonic enhancement [@pecinaberridge2005hotspot]. Smith and Berridge identified a corresponding opioid-sensitive hedonic hotspot in posterior ventral pallidum, with different effects at more anterior sites [@smithberridge2005pallidum].

The anatomical contrast is the important finding. Changes in consumption and changes in positive taste reactions had different spatial distributions. These experiments localized mechanisms that amplify hedonic impact rather than treating all increased eating as increased pleasure.

Mahler, Smith, and Berridge extended the accumbens result to an endocannabinoid. Microinjections of anandamide into medial shell increased positive sucrose reactions, with the strongest enhancement in a dorsal medial-shell hotspot [@mahleretal2007endocannabinoid]. This establishes endocannabinoid-sensitive hedonic amplification in accumbens shell; the pallidal study above established opioid-sensitive amplification.

The hotspots operate within a wider network that includes brainstem taste circuits and forebrain systems processing the sensory and bodily consequences of consumption. Taste reactivity gives one experimentally accessible measure of hedonic impact. Its value here is the causal separation it permits among three operations:

  • Learning changes predictions and the weights governing future behavior.
  • Wanting gives cues and outcomes motivational control over present action.
  • Liking is the positive hedonic reaction during experience.

Dopamine contributes strongly to teaching and pursuit. The hotspot experiments identify additional causal machinery for hedonic amplification. An outcome can therefore become more intensely pursued without a corresponding increase in the pleasure it produces.

Olds and Milner discovered that rats would repeatedly perform actions that produced brief electrical stimulation at certain brain sites [@oldsmilner1954reinforcement]. Stimulation could serve as a powerful reinforcer, establishing a direct experimental link between neural activation and the later probability of an action.

Electrical current recruits mixed neuronal elements, including axons. The behavior established the reinforcing effect of that activation; identifying its components required further experiments. Separating cue learning, pursuit, and hedonic reactions subsequently made it possible to ask which of those processes a manipulation changes.

35.11 Addiction recruits several forms of learning and control

Addiction reveals how the mechanisms developed in this chapter can sustain behavior despite escalating costs. Drugs alter neural signaling; their consequences reinforce actions; their predictors acquire motivational pull; and repeated seeking changes control over performance. Dependence adds bodily states in which taking the drug produces relief. These processes can reinforce one another.

35.11.1 Drug pharmacology enters learning circuits

Cocaine blocks monoamine transporters. Amphetamine reverses or disrupts monoamine transport. Opioids act at opioid receptors and strongly alter pain, brainstem, striatal, and stress circuits. Nicotine activates nicotinic acetylcholine receptors. Alcohol changes several transmitter and membrane systems. Their acute subjective and physiological effects are not interchangeable.

Many addictive drugs nevertheless increase or dysregulate dopamine in mesolimbic striatum, either directly or through upstream circuitry [@dichiaraimperato1988drugs]. This allows drug delivery and its predictors to engage the same plasticity that ordinarily relates cues and actions to biologically important outcomes. Drug pharmacology directly changes the signals through which learning and motivation are implemented.

The consequence can be unusually persistent cue control. Places, people, odors, paraphernalia, internal states, and action sequences become predictors of drug availability. Those predictors can capture attention and provoke approach long after withdrawal. Human PET studies show that cocaine-predictive cues can increase dopamine in dorsal striatum in relation to craving [@volkowetal2006cues]. The learned cue thereby gains a route into present motivation and action.

35.11.2 Repeated seeking can recruit dorsal habit circuitry

Early drug use can be flexible and explicitly outcome directed. With repetition, the actions used to obtain a drug can become increasingly controlled by familiar cues and sequences. In rats, prolonged cocaine self-administration can produce seeking that becomes less sensitive to outcome devaluation and increasingly dependent on dorsolateral striatum [@zapataetal2010habitual]. The ventral-to-dorsal interactions described earlier supply a route for this recruitment (Figure 35.3).

Other preparations reveal persistence through flexible control. Rats can develop addiction-like behavior while repeatedly solving changing action requirements, without the predicted recruitment of dorsolateral dopamine control [@singeretal2018habits]. Human drug seeking can likewise involve planning and prolonged pursuit. The practical distinction is which information controls the seeking: a familiar response can be triggered habitually, or flexible action can be organized toward obtaining a harmful outcome.

35.11.3 Relief from a bad state can become the reinforcer

As dependence develops, taking a drug can remove or reduce withdrawal, stress, dysphoria, pain, or another aversive state. When that consequence increases future drug taking, it is negative reinforcement. Negative refers to removal of an aversive condition; reinforcement refers to the increased probability of the action.

The prediction-error account still compares obtained and expected consequences. Relief that is greater or earlier than expected can provide a positive error. Fully expected relief can remain an important reason to act while producing little new error at its arrival. A consequence can sustain pursuit after its occurrence is well predicted; prediction error signals that the expectation needs updating.

Ahmed and Koob compared rats with short and extended daily access to cocaine. Extended access produced escalating intake and altered patterns of self-administration, which they interpreted as a change in hedonic regulation [@ahmedkoob1998escalation]. The behavioral result established that prolonged access can change subsequent consumption; identifying the contribution of withdrawal and relief requires measurements of the accompanying state.

The control loop is clear. The drug changes the body; adaptation changes the state produced by its absence; relief gives taking the drug a renewed function. The same action can consequently be maintained by anticipated positive effects, cue-triggered wanting, habitual responding, relief, or a combination that changes over time.

35.11.4 Persistence despite harm exposes a further control problem

A habit is identified by relative insensitivity to outcome devaluation or contingency change. Compulsion concerns persistence despite punishment, conflict, or known harm. An action can be both, but the tests examine different failures of adjustment.

In animal models, only a subset of subjects develops persistent drug seeking across tests of punishment, high effort, or drug unavailability [@derochegammonetetal2004addiction]. Pascoli and colleagues isolated a mechanism for punishment-resistant reinforcement using optogenetic self-stimulation. Mice pressed a lever to activate dopamine neurons; after electric shock was introduced as an adverse consequence, some stopped and others continued [@pascolietal2018compulsion].

Persistent responding was associated with strengthened transmission from orbitofrontal cortex to dorsal striatum. Potentiating those synapses in mice that had stopped produced renewed punishment-resistant pressing. Depotentiating them in persistent mice reduced that behavior. The bidirectional manipulation linked a specific corticostriatal change to persistence despite punishment.

This experiment used direct dopamine-neuron stimulation as the reinforcer. It isolated a circuit mechanism that a drug exposure model would combine with drug-specific pharmacology and bodily effects. The result advances the chapter’s argument: changing synaptic influence can alter whether an adverse consequence successfully reconfigures an ongoing course of behavior.

The wanting–liking distinction provides another route to understanding persistence. Drug cues can acquire intense motivational pull even when the expected experience offers less pleasure or carries serious costs. Incentive sensitization describes increasing responsiveness of systems supporting cue-triggered pursuit. Habit and relief can further sustain the action, each through the mechanisms already developed.

Human exposure and vulnerability depend on development, stress, pain, social conditions, availability, and the properties of the drug. These conditions change the opportunities for learning and the alternatives available to the person. They belong within the explanation of persistent use because they help determine what consequences and competing actions the nervous system encounters.

Addiction can combine altered teaching signals, powerful cue-triggered wanting, habitual seeking, relief from aversive states, and persistence despite harm. These mechanisms explain different parts of the disorder and identify different points at which control can change.

Figure 35.5: Learning, wanting, and liking provide several routes to persistent drug seeking. A: Learning changes predictions and future action weights. Wanting gives learned cues motivational influence over present action, expressed in attention, approach, and vigor. Liking is the positive hedonic reaction during experience. Dopamine contributes strongly to teaching and pursuit. Opioid stimulation can amplify positive taste reactions within small hotspots in medial accumbens shell and posterior ventral pallidum [@pecinaberridge2005hotspot; @smithberridge2005pallidum]. Endocannabinoid-sensitive amplification has been demonstrated with anandamide in accumbens shell [@mahleretal2007endocannabinoid]. The hotspot labels summarize these distinct anatomical and pharmacological findings. B: Several processes can sustain drug use. Drug-specific pharmacology alters neural signaling. Predictive cues acquire motivational pull that interacts with stress and bodily state. Repeated seeking can recruit dorsal-striatal habit mechanisms. Withdrawal and other aversive states can make relief reinforcing. These processes can combine, with their relative importance changing across drugs, individuals, and stages of use. In the matrix, habit concerns reduced sensitivity to outcome value or causal contingency, while compulsion concerns persistence despite punishment, conflict, or harm. Either can occur without the other. The diagram summarizes mechanisms whose contribution is tested through the behavioral and neural experiments described in the text.

35.12 How experiments identify what controls action

Explanations in terms of value require evidence beyond the behavior being explained. If lever pressing establishes that an outcome is valuable, and its value is then used to explain the pressing, the argument is circular. Labeling an active brain region a value region does not resolve the problem. A useful experiment must change a proposed source of control independently and ask whether the action changes as predicted.

The strongest experiments change a relation while preserving enough of the behavior to identify the effect. Blocking changes how much information an outcome supplies. Devaluation changes the desirability of a particular consequence. Contingency degradation changes the advantage gained by acting. Transfer tests change the predictive cue while withholding the outcome. Each comparison asks which information governs performance.

Neural recordings then identify candidate mechanisms at different scales. Single-neuron electrophysiology measures spikes with millisecond resolution. Microdialysis measures slower changes in extracellular transmitter, often across minutes. Fast-scan cyclic voltammetry detects subsecond local dopamine changes and helped establish the development of cue-related release [@dayetal2007shifts; @willuhnetal2012hierarchical]. Genetically encoded fluorescent dopamine sensors provide another rapid measure of extracellular signaling [@patriarchietal2018sensor]. These methods follow different stages between neuronal activation and the effect of transmitter at its target.

Perturbations establish causal influence. A manipulation during learning can alter what is acquired; a manipulation confined to a later test can alter how existing knowledge controls performance. Delivering stimulation at the outcome and delivering it between trials are different tests, even when the amount of stimulation is identical. Lesions, reversible inactivation, and receptor manipulations likewise become informative when the design distinguishes learning, retrieval, motivation, and movement.

Anatomy identifies the routes available for those effects. Anterograde and retrograde tracers establish the direction and topography of projections, as in the primate striato–nigro–striatal spiral [@haberetal2000spiral]. Establishing synaptic contacts or functional transmission requires additional methods. Human diffusion MRI estimates white-matter orientations at a coarser scale and does not supply transmitter identity or direction of signaling.

Human experiments connect the behavioral account to distributed brain activity. Model-derived prediction errors correlate with ventral-striatal and orbitofrontal BOLD signals during learning [@odohertyetal2003td]. Pharmacological experiments relate dopaminergic manipulation to changes in learning and prediction-error signals [@pessiglioneetal2006human]. PET measures receptor availability and can infer changes in endogenous dopamine from competition with a radioligand, as in studies of cocaine cues and craving [@volkowetal2006cues]. An fMRI prediction-error correlate links a computational variable to a hemodynamic signal; it does not directly measure dopamine. PET displacement infers task-related dopamine changes over an interval much longer than a phasic burst. These measurements connect the behavioral account to human brain function at different spatial and temporal scales.

Across these methods, the explanatory sequence is the same: identify the information controlling the action, measure the candidate neural process, and test whether changing that process changes learning or performance in the predicted way. The task gives the neural measurement its behavioral meaning.

35.13 Coda: when action becomes history

Unit 5 began with controlling a body. Motor pools turn neural activity into force; spinal and brainstem circuits organize rapid responses, posture, orienting, and recurring movements. Cortical systems relate action to objects and rules. Cerebellar loops support prediction and calibration. Basal-ganglia loops regulate access, restraint, and vigor. These contributions interact throughout behavior.

This final chapter adds history to that control system. An action changes the body and world, and its consequences change what happens next time. Prediction errors revise expectations. Dopamine-dependent plasticity changes the influence of recently active connections. Cues acquire the capacity to attract and invigorate. Current bodily state changes the use of those learned relations. Practice can make familiar contexts sufficient to recruit an action with less dependence on its present outcome.

The experiments distinguish these processes by separating variables that usually occur together. An outcome can arrive without supplying new information. A predictive cue can gain motivational force before its outcome is experienced in a new state. Pursuit can increase without greater hedonic impact. A practiced action can remain strong after its consequence loses value. Each dissociation identifies a different way that experience governs control.

This organization makes behavior efficient and creates vulnerabilities. Learning makes useful actions easier to recruit; it can also give a cue excessive influence or sustain a response after its consequences have changed. Addiction shows how altered teaching, powerful wanting, habit, relief, and persistence despite harm can combine.

Unit 6 extends the same account to delayed and uncertain possibilities, imagined outcomes, symbolic rules, and consequences involving other people. The cortical, striatal, limbic, and bodily systems developed here remain part of that explanation. Future-oriented control builds on the capacity to learn from what action has already produced.

Action becomes consequence. Consequence becomes learning. Learning changes the next action.

Established findings. Blocking demonstrates learning governed by predictive information. Many dopamine neurons exhibit signed reward-prediction-error responses, and timed activation can alter learning. Dopamine can promote synaptic plasticity within a short interval after local activity. Devaluation and contingency tests identify outcome-sensitive and habitual control. Cue-triggered pursuit and hedonic reactions can be changed separately. Striatal territories and their connections contribute differently to these effects.

The organizing interpretation. Consequences reweight future action through interacting changes in prediction, synaptic influence, and behavioral control. Current bodily state determines how learned opportunities matter now. Practice changes the relative influence of outcome knowledge and familiar contexts. This framework connects reinforcement learning to the embodied controller developed throughout the unit.

Open questions. How are teaching, movement, and state signals distributed across dopamine populations and release sites? How is credit assigned over long natural sequences? Which representations allow a new need to change behavior before further learning? How do the mechanisms identified in laboratory habit and compulsion tasks combine over years of human experience?