32 Acting on the Seen World
Visual Guidance, Action Goals, and Control Beyond the Hand
32.1 From force to a target in the world
The preceding chapter followed action through motor cortex, descending pathways, spinal circuits, motor neurons, and muscle. Those systems make force available and organize it into postural adjustments, withdrawal, locomotion, and flexible voluntary movement. They do not, by themselves, specify which cup to reach toward, where its handle lies relative to the hand, or how the fingers should change when the cup begins to slip.
Those are relational problems. A target is visible at one place on the retina, but the eyes can move within the head, the head can turn on the trunk, the trunk can lean, and the arm can begin from many configurations. The hand is not a point that merely has to arrive at a coordinate. It has a size, orientation, joint limits, possible contact surfaces, and a history of forces acting upon it. The target can move. The animal can move. The intended outcome can change while the movement is already underway.
Visually guided action is a recurrent control process. Visual, proprioceptive, tactile, vestibular, and motor-related signals continually update the relation among the target, the body, and the action in progress. The same visible object can be reached for in different ways because the body’s starting state and the intended outcome help determine what the movement must achieve.
Acting on the seen world means continually keeping a changing body related to a changing target.
Vision is prominent in this chapter because it often identifies distant opportunities before the body contacts them. It is not sufficient. A person reaching in darkness can combine a remembered target location with proprioceptive evidence about the arm. A climber adjusts grip when tactile afferents signal incipient slip. Vestibular and postural systems stabilize the body from which the arm moves. What is commonly called a visuomotor transformation is therefore embedded within a multisensory and biomechanical control system.
32.2 Orienting comes before reaching
Before an animal can manipulate an object, it must usually orient toward something that matters. This problem is older than hands and older than neocortex.
Across vertebrates, the optic tectum—called the superior colliculus in mammals—contains spatially organized sensory signals and contributes to orienting the eyes, head, and body toward relevant events. Work in living lamprey makes the ancestral organization especially visible. Different regions of the tectal map can recruit brainstem pathways that turn the animal toward a possible target or away from a possible threat. These circuits do not produce a primate reach, but they solve a more basic version of the same control problem: relate a mapped event in the world to a coordinated change in the body [@suzuki2019tectum; @grillner2021evolution].
Evolution did not replace that midbrain system when forelimbs became available for manipulation. It added control above control. In primates, the superior colliculus is best known for its contribution to gaze shifts, but its activity is also related to coordinated eye–hand behavior. Collicular neurons can be active before and during arm movements toward visual targets [@werner1993arm]. Behaviorally, gaze commonly reaches an object before the hand and can remain anchored near it while the hand approaches [@neggersbekkering2000]. Orienting and reaching are normally coordinated components of one action. This coordination remains flexible: people can also reach toward peripheral, remembered, or briefly viewed locations.
Expanded posterior parietal and frontal systems added capacities that a tectal orienting map alone could not provide. A primate can select one object among several, delay a reach, use an arbitrary cue, choose either hand, route the arm around an obstacle, rotate the wrist, and shape individual digits for a precision grip. Those newer cortical contributions still act through the postural, brainstem, spinal, and muscular systems described in Chapter 31. They also remain coupled to the older orienting system.
The evolutionary progression builds a hierarchy of interacting controllers. The tectum or superior colliculus helps align sensory surfaces and the body with events in space. Parietofrontal networks add flexible limb-specific and object-specific control. Descending pathways recruit older brainstem and spinal machinery. The changed body then generates new visual, proprioceptive, vestibular, and tactile evidence.
32.3 A target has no single coordinate
The cortical contribution to visually guided action is often summarized as the dorsal visual stream, extending from occipital visual cortex into posterior parietal and frontal motor-related regions. Several occipitoparietal routes interact recurrently with frontal cortex, the superior colliculus, somatosensory systems, and ventral visual networks. Posterior parietal cortex helps maintain changing relations among sensory events, the body, and possible actions [@goodalemilner1992; @goodalewestwood2004].
A target five degrees to the right of fixation has a well-defined retinal location. It does not, by itself, specify where an object lies relative to the head, shoulder, hand, or room. The same retinal location can be produced by many combinations of eye position and object position. Turn the eyes to the left and an object straight ahead can fall on the same retinal region as an object to the right when gaze is centered.
The ambiguity multiplies as action approaches the body. The eyes rotate within the head. The head rotates relative to the trunk. The shoulder translates when the trunk leans. The elbow and wrist change the location and orientation of the hand. A reach also begins from a particular posture, and the nervous system must estimate that posture from proprioception, touch, vision of the limb, and motor-related signals. Each source carries noise and delay. The current hand location is therefore not simply read from one sensor.
Researchers describe these relations using reference frames. A neuron or population may be characterized as more eye-centered, head-centered, body-centered, hand-centered, or object-centered when its activity varies most systematically with one of those descriptions. Posterior parietal neurons commonly combine several of these variables. A cell may be strongly tuned to the retinal location of a target while its response is modulated by gaze direction, hand position, intended effector, or current task. Nearby cells can express different mixtures [@andersenmountcastle1983; @chang2010idiosyncratic].
Human imaging likewise reveals multiple reference frames within parietofrontal cortex. During reach preparation, posterior parietal and dorsal premotor areas show gaze-centered integration of target and hand position, whereas regions closer to primary motor cortex show body-centered hand-position effects [@beurze2010reference]. A network can therefore preserve several descriptions at once. That may be advantageous. Eye-centered information remains useful when the eyes move again; hand-centered error is useful for directing the limb; object-centered geometry is useful for shaping a grasp; and body-centered information helps keep the action compatible with posture and balance.
A reference frame describes how neural activity relates to variables in an experiment. Distributed populations can support action by jointly relating the current target to the current body. The mixed responses do not imply a row of anatomical coordinate converters or require one final master map.
A classic posterior parietal neuron may respond when a visual target appears in a particular retinal region. Its visual receptive field can remain anchored to the retina while the magnitude of the response changes with eye position. The target is the same distance from the fovea, but the neuron fires more strongly when the eyes point left than when they point right. Eye position has altered the neuron’s gain [@andersenmountcastle1983].
A population containing many such combinations can distinguish situations that a retinal signal alone cannot. One neuron prefers a target to the right of gaze and is amplified when gaze is leftward; another has a different retinal preference and a different eye-position modulation. Taken together, the pattern can support an estimate of target location relative to the head or body.
Gain fields provide a population mechanism for combining several signals. An individual neuron can retain a largely retinal receptive field while contributing to a readout that relates the target to the head or body. Its response therefore need not be a complete head-centered coordinate. The combined representations remain mixed and depend on the behavior being performed [@chang2010idiosyncratic].
32.4 Reaching and grasping are coupled controllers
A reach toward a cup can be divided descriptively into two components. Transport moves the hand toward the object. Grasp formation changes hand aperture, finger configuration, and wrist orientation so that appropriate surfaces can be contacted. The distinction is useful because these components impose partly different demands and are associated with relative biases within parietofrontal cortex [@jeannerod1984prehension].
A dorsomedial network that includes medial and superior posterior parietal regions and dorsal premotor cortex is strongly engaged by reaching. A more lateral network that includes the anterior intraparietal region and ventral premotor cortex is especially associated with hand shaping and grasp. Human imaging reveals corresponding reach- and grasp-related gradients, and macaque recordings show neurons sensitive to target location, gaze, wrist orientation, grip type, and combinations of these variables [@konen2013parietal; @murata2000aip; @fattori2009v6a].
Transport and hand shaping are coordinated throughout the approach. The hand begins to open and rotate before transport is complete. Maximum grip aperture is timed relative to the approach of the hand. A change in object orientation alters the wrist trajectory; an obstacle can alter both the route of the arm and the side from which the fingers approach. Reaching and grasping are prepared concurrently and continually constrain one another. The anatomical terms describe functional biases within overlapping networks.
Neither component is specified by visual geometry alone. Imagine lifting a familiar mug. Its visible shape helps determine where the thumb and fingers can contact it, but successful lifting also depends on the initial posture of the arm, joint limits, expected weight, surface friction, and what the mug is being lifted for. A sip, a pass to another person, and placing the mug upside down can begin with similar transport and diverge in wrist orientation, contact points, and forces. Visual object information associated with ventral temporal cortex interacts with parietofrontal systems rather than remaining sealed in a separate stream.
The body also contributes constraints that are easy to omit from a diagram of cortical pathways. Shoulder rotation changes the feasible range of elbow and wrist positions. The trunk can extend the reachable workspace. Gravity alters the torques required along different trajectories. The fingers must contact the object without destabilizing it, and the arm must preserve the body’s balance while lifting the load. A neural controller works with these mechanics rather than calculating a movement for an abstract, weightless hand.
This is another expression of motor abundance. Many joint configurations can bring the hand to the same region, but not all are equally stable, economical, comfortable, or compatible with the intended grasp. Parietofrontal activity helps constrain the available solutions; spinal feedback, muscle properties, and the physical interaction with the object continue to shape which solution is realized.
32.4.1 Object geometry prepares the hand
The anterior intraparietal area, or AIP, provides a particularly clear view of how object information enters the grasping system. A narrow stem and a broad cylinder call for different hand configurations. Their shapes matter because the fingers must establish useful contact with them.
Murata and colleagues recorded from macaque AIP while monkeys manipulated objects in the light and in darkness, and while they looked at objects without manipulating them. Comparing these conditions separated responses to visible object information from activity accompanying the action. Some neurons depended strongly on vision; others responded during manipulation in both conditions, with an additional visual contribution; still others remained active during manipulation in darkness without responding to the sight of the object alone. Many visually responsive neurons also distinguished object shape, size, or orientation [@murata2000aip].
These response profiles connect the geometry of an object with the configuration of a hand. The comparison with darkness is especially useful: movement and its somatosensory consequences remain when ongoing visual information is removed. A response that survives in darkness therefore has a different evidential meaning from one that appears during object viewing alone. It need not be a purely outgoing motor command.
AIP interacts with ventral premotor cortex, including macaque area F5, within the grasping network. Visual dimensions, possible contact surfaces, and the developing hand configuration can influence one another through this recurrent circuit. This gives object shape a practical significance before contact: it helps establish how the object can be taken hold of.
The next question is what the animal will do with that grasp. Picking up the same object can begin very different actions.
32.5 The action is corrected while it unfolds
A reach continues to be shaped after it begins. Moving the target during the reach makes this continuing control visible.
In a target-jump experiment, a person begins reaching toward a visible target. After the movement starts, the target is displaced. The hand often changes course rapidly, sometimes even when the displacement is difficult to report. The correction does not require vision of the hand itself; information about the new target can be combined with an estimate of the moving limb [@goodale1986adjustments; @pelisson1986doublestep]. The trajectory that finally appears was therefore not wholly specified before movement onset. It emerged from preparation plus continuing sensory control.
Posterior parietal cortex is important for these updates. Brief disruption by transcranial magnetic stimulation can impair correction to a displaced target, and parietal lesions can produce a related inability to redirect the hand efficiently [@desmurget1999updating]. These updates recruit a distributed controller. Visual signals pass through several cortical and subcortical routes. Premotor and motor cortex alter descending output. Proprioceptive feedback reports the changing limb. Spinal and brainstem circuits respond to the resulting forces. The cerebellum helps estimate state and calibrate correction. The limb’s own inertia, elasticity, and interaction with the environment determine how any neural change affects the trajectory.
Correction also continues after contact. When the fingertips first touch an object, cutaneous afferents signal contact location, pressure, vibration, and local slip. If a smooth object begins to move within the grasp, grip force increases rapidly to restore a safety margin. Experience with the object also shapes the forces applied before slip occurs. Precision grip therefore combines anticipatory scaling with tactile correction rather than choosing between feedforward and feedback control [@johanssonwestling1984; @johanssonwestling1987].
Proprioception supplies another route. A mechanical perturbation that displaces the arm changes muscle length, joint configuration, and skin deformation. Early spinal responses are followed by later responses that incorporate more of the task, as described in Chapter 31. Visual and proprioceptive evidence can be given different weights depending on their reliability and on which variable the movement must control. When the hand is hidden, proprioception becomes especially important; when limb vision is clear, the two sources can be combined. Neither is a perfect copy of the limb’s state [@sobersabes2005].
The useful unit is a recurrent relation:
- a target is estimated relative to gaze, body, and hand;
- descending activity changes the limb;
- the movement changes retinal, proprioceptive, and tactile input;
- the estimated relation is updated;
- output changes again where the remaining error matters.
These stages overlap. Sensory processing continues during movement, and motor-related activity changes how incoming sensory evidence is interpreted. The nervous system is not repeatedly starting the action over. It is preserving some task variables—hand near target, stable contact, upright body—while allowing many lower-level details to vary.
Some target displacements produce rapid hand corrections even when participants do not give a reliable verbal report of the change. Findings of this kind helped motivate the idea of an automatic pilot for the hand [@pisella2000pilot]. The phrase captures an important fact: visual evidence can influence an unfolding action without waiting for a deliberate decision or a verbal description.
Perceptual report and rapid visuomotor correction place different demands on a distributed visual system. Their timing can differ, and a person may become aware of a change after the correction has begun. Correction speed also depends on visibility, attention, movement phase, and displacement size. The contrast establishes different access to visual information; failure to report a displacement alone does not identify a wholly separate unconscious pathway.
32.6 Affordances depend on the current body
James Gibson used the term affordance for an opportunity for action available to an animal in an environment [@gibson1979ecological]. An affordance is not simply a visual feature stored inside an object. It is a relation.
A ledge affords sitting only for a body of an appropriate size and mobility. A gap affords passage for one animal but not another. A cup handle may afford a precision grip from one direction, a whole-hand grip from another, and no comfortable grasp when the wrist is already near its limit. A target just beyond reach can become reachable when the trunk is free to lean or when a tool extends the hand. Fatigue, injury, current posture, and what the animal is trying to accomplish all alter the available action.
Object shape, location, body configuration, available effectors, and current goals jointly constrain what can be done. The same object can invite several actions, and the same action can often be achieved with several effectors.
32.6.1 Peripersonal space is a multisensory control interface
Some neurons in parietal and premotor cortex respond both to touch on a particular body region and to visual events near that region. A neuron responsive to touch on the face may also respond when an object approaches the face; another may combine touch on the hand with vision near the hand. In macaque ventral premotor area F4, such visual receptive fields can move with the associated body part rather than remaining fixed at one location in the room [@fogassi1996peripersonal; @grazianocooke2006].
These findings are often described as representations of peripersonal space, the immediately reachable or body-relevant region around the animal. The represented region changes with posture, task, threat, and potential contact. Visual–tactile convergence can support reaching and grasping, but also avoidance, defense, and protection of the body [@fogassi1996peripersonal; @grazianocooke2006]. It is better understood as a flexible multisensory interface between events near the body and the actions those events may require.
A visual event near the hand can therefore be represented differently from an identical event far from the body because it has different consequences. It may be contacted, grasped, deflected, or avoided with little delay. The geometry of the environment is being evaluated relative to a body capable of acting within it.
32.6.2 Several actions can be available at once
A cup can be grasped by the handle, lifted around its body, pushed aside, passed to another person, or ignored. One influential proposal—the affordance competition hypothesis—is that parietal and frontal populations can specify several potential actions in parallel, while goals, rules, expected outcomes, and selection systems bias the competition [@cisek2007affordance; @cisekkalaska2005]. On this view, perception does not have to finish constructing a neutral description before action possibilities appear.
The world normally permits several actions, and selection can alter sensorimotor processing before one movement has been released. The competition hypothesis gives this interaction a specific computational interpretation. Activity related to several candidate reaches supports parallel preparation; establishing how those candidates compete requires additional evidence about their interactions.
This provides a natural link to the basal ganglia. Parietofrontal systems help specify where and how the body could act; basal-ganglia and frontal systems contribute to whether one candidate is selected, withheld, or replaced. The division is not absolute. Selection signals influence parietal and premotor activity, and the sensory consequences of an emerging action can change what remains available.
32.7 The goal organizes the movement
An action goal can specify where the hand should arrive, what contact it should establish, or what the animal will accomplish afterward. These are related levels of control. Reaching a location, securing an object, and bringing food to the mouth all constrain movement, but they constrain it in different ways.
The distinction matters when interpreting neural activity. During an ordinary reach, a visual cue, the intended destination, and the direction of movement often coincide. During grasping, finger closure and successful capture of the object often coincide. Experiments become informative when these normally correlated events are separated.
32.7.1 A visual location becomes a destination for an action
A location can be the target of a gaze shift or of a reach. Snyder, Batista, and Andersen compared these possibilities in macaque posterior parietal cortex using delayed eye and arm movements. During the waiting period, activity depended on the action being prepared: neurons in the lateral intraparietal area, LIP, were more strongly associated with planned saccades, whereas neurons in the parietal reach region, or PRR, were more strongly associated with planned reaches [@snyder1997intention]. The target’s spatial location was therefore only part of the information expressed by the population. The intended effector mattered as well.
PRR is a functional designation for reach-related posterior parietal territory, not a claim that every neuron within it serves reaching alone. The delayed task reveals preparation before the movement and its sensory consequences occur. In this setting, intention means an experimentally specified forthcoming action.
Gail and Andersen separated the visible cue from the reach destination more directly. Monkeys maintained central gaze while a peripheral cue briefly appeared. A rule instructed either a pro-reach, toward its remembered location, or an anti-reach, toward the opposite location. After a delay, a go signal released the movement. Thus the same cue location could require opposite reaches. During the delay, PRR population activity predominantly reflected the upcoming reach goal rather than the location where the cue had appeared [@gailandersen2006].
This is a substantial transformation. A retinal event has been interpreted through a rule and related to a future action. It also gives a precise meaning to goal-related activity: the experiment dissociates the instructed destination from the sensory event that specifies it. The result does not require a neuron to contain a verbal intention or a complete plan for the muscles.
32.7.2 The same goal can require opposite hand movements
A tool can separate what the hand does from what the action achieves. Umiltà and colleagues trained macaques to grasp objects with ordinary pliers and with reverse pliers. Closing the hand closed the tips of ordinary pliers; opening the hand closed the tips of the reverse tool. Both actions captured an object, but they required opposite finger movements.
Recorded F5 neurons maintained their relationship to the phases of tool grasping across the two implements. A neuron associated with securing the object could discharge during hand closure with ordinary pliers and during hand opening with reverse pliers. Primary motor cortex contained both goal-related responses of this kind and responses more closely tied to the hand movement [@umilta2008pliers].
The distal effect of the tool had become a useful organizing variable for the trained motor system. This extends the motor equivalence introduced in Chapter 31: different bodily movements can serve a common outcome. Here the outcome is the tool taking hold of the object. The study examined extensively trained animals, so it establishes what the practiced system can do rather than how that organization first develops.
32.7.3 A grasp is prepared for what comes next
A grasp belongs to a larger action. Fogassi and colleagues recorded neurons in macaque inferior parietal cortex while the animal grasped an item to eat it or to place it in a container. Some neurons were more active during grasping-to-eat, others during grasping-to-place, and others responded similarly in both conditions. The difference was present during the grasp, before the subsequent act was completed [@fogassi2005actions].
The future use of the object was already expressed in the activity accompanying its acquisition. The grasp was being organized as part of an action sequence. This makes biological sense: the wrist orientation, contact, and forces appropriate for bringing something to the mouth need not be those appropriate for putting it down.
The word goal now refers to the larger action in which the grasp participates. The recorded activity could draw on the context, anticipated next movement, and expected outcome that distinguished eating from placing. The experiment establishes sensitivity to that action context; assigning the response exclusively to an abstract intention would require further dissociations.
Together, these experiments show why a motor description limited to joint displacement is incomplete. The planned effector changes the response to a location. A rule changes the destination specified by a cue. A tool changes the hand movement needed to capture an object. The larger action changes the activity accompanying a grasp. At each level, the nervous system relates the body to something the action is intended to accomplish.
I regard this as an important extension of the book’s control perspective. A controller must organize movement around consequences that matter. The consequence can be as local as stable fingertip contact or as extended as placing an object where it will be needed. These goals constrain lower-level activity while remaining dependent on the mechanics and feedback through which they are achieved.
32.8 Mirror neurons: seeing an action in a motor system
Watching another individual grasp an object can activate neurons that also participate when the observer grasps. This finding brought action observation into the physiology of motor control and gave rise to the term mirror neuron. The defining observation concerns the same recorded neuron under two conditions: performing an action and observing a corresponding action performed by someone else.
Gallese and colleagues described such responses in macaque ventral premotor area F5. Some neurons discharged during the monkey’s own grasping and during its observation of an experimenter’s object-directed action. For some cells, the effective observed and executed acts matched closely; for others, the correspondence was broader. The sight of an object alone or a similar movement without an object often failed to produce the effective response [@gallese1996mirror].
This distinguishes mirror responses from the object-related activity described earlier. A cell responsive to the sight of a graspable object can help specify an available action. A mirror neuron responds to another individual acting. Both connect vision with motor organization, but the visual information is different. A hand taking hold of a cup contains information about an unfolding interaction, not just the cup’s geometry.
32.8.1 Observation includes the action’s context
Mirror responses can extend beyond the immediately visible part of an act. Umiltà and colleagues briefly showed monkeys whether an object was present, then positioned a screen to hide the final hand–object interaction before the experimenter began to reach. The approach remained visible. A subset of F5 mirror neurons still responded when the monkey had seen that an object was present. A comparable movement behind the screen without an object did not produce the same response [@umilta2001hidden]. The earlier scene helped determine how the now-hidden action was represented.
The eating-versus-placing experiment also included observation. Some inferior parietal neurons distinguished the two action contexts while the monkey watched the experimenter grasp, before the later eating or placing act revealed the completed sequence [@fogassi2005actions]. Motor and observational responses could therefore be sensitive to what came next.
These findings make action observation an active process. An observer can use the object, the scene, and the developing movement to anticipate an outcome. For an animal preparing to receive food, avoid a competitor, or coordinate with a partner, that anticipation can guide its own behavior. This is a functional interpretation of the findings: the recordings reveal context-sensitive activity, while its particular contribution to behavior must be tested separately.
32.8.2 Why the discovery reached beyond motor neuroscience
Mirror neurons offered a compelling answer to an old problem in psychology and philosophy: how does one individual understand another’s actions? Simulation accounts propose that an observer uses aspects of its own action machinery to model what another individual is doing. The discovery supplied a concrete neural candidate for such a mechanism. Gallese and Goldman explicitly connected mirror-neuron physiology to the simulation theory of understanding other minds [@gallesegoldman1998].
The appeal is substantial. An observed reach could be understood through a system already capable of organizing reaching. The meaning of the act would be grounded in the observer’s capacities for action. This fits naturally with an embodied account of cognition and helps explain why mirror neurons became important well beyond the laboratory tasks in which they were first studied.
The explanatory reach of the idea depends on what understanding requires. Recognizing a grasp, predicting that an object will be lifted, inferring that someone is clearing a table, and judging whether that person is being helpful are different achievements. A shared response to performed and observed grasping identifies a link between seeing and performing an act. Establishing how that link supports recognition, prediction, or judgments about beliefs and social motives requires evidence about the observer’s performance. Empathy adds another distinction: understanding what someone is doing is not the same as sharing or caring about that person’s emotional state.
The most informative challenge to a strong simulation claim is a test of necessity. Vannuscorps and Caramazza studied five people born with absent or severely shortened upper limbs. Across several tasks, these participants could recognize and interpret upper-limb actions with performance comparable to that of people with typical limbs [@vannuscorpscaramazza2016]. Efficient understanding therefore does not universally require the observer to possess and reenact the corresponding hand action. Development can support other routes to that knowledge. The result leaves room for motor experience to contribute when it is available.
I think the lasting importance of mirror neurons lies in bringing the observer’s action system into the explanation of action perception. That is a substantial contribution. A complete account must also explain how visual evidence, learned context, motor experience, and knowledge about other people work together. Naming a mirror mechanism identifies a candidate contribution to that process; it does not complete the explanation.
32.8.3 Evidence in humans comes at several scales
Human action observation can be studied with fMRI, EEG, TMS, and, in selected clinical settings, recordings from individual neurons. These methods answer different questions. An fMRI region active during both observation and execution establishes overlap at the scale of a population. It does not establish that the same cells generated both responses. EEG mu-rhythm suppression, a reduction in sensorimotor rhythmic activity, and changes in TMS-evoked muscle responses provide other measures of sensorimotor engagement. Neither is a direct count or recording of mirror neurons.
Mukamel and colleagues obtained more direct evidence from patients undergoing intracranial monitoring. They recorded neurons in medial frontal and temporal regions while patients observed or performed hand actions and facial gestures. Some neurons responded in both conditions, including cells in supplementary motor cortex. Responses also included different signs of modulation during observation and execution [@mukamel2010mirror]. Human evidence therefore includes recordings at the cellular level, although these clinically determined recording sites did not provide a comprehensive sample of the lateral parietofrontal regions commonly discussed as the human mirror system.
The distinction between observation and execution is itself a control problem. Seeing a grasp usually does not make the observer’s hand grasp. Shared activity must be embedded in a network that preserves whose action is occurring and whether the observer should imitate, cooperate, withdraw, or remain still. For this chapter, that is the productive connection: seeing another’s action changes the possibilities for one’s own.
32.9 When visual guidance fails: optic ataxia
The term optic ataxia describes a disproportionate impairment in using vision to guide the hand toward a target. A patient may see and identify an object yet misreach, misorient the wrist, or form an inappropriate grasp, despite having enough strength and elementary movement to perform the action under other conditions. The syndrome is most strongly associated with damage to posterior parietal and occipitoparietal systems [@pereninvighetto1988].
The deficit is not uniform. Errors are often greater for targets presented away from fixation, in particular regions of the visual field, or with one hand. Some patients show a strong contralesional field effect; others show hand-related or fixation-related patterns. Reaching to the remembered location of a target, reaching with central vision, or using proprioceptive and tactile information can produce a different level of impairment. Individual lesions also differ in their cortical and white-matter involvement [@pereninvighetto1988; @pisella2000pilot].
Optic ataxia should not be confused with weakness. The arm can generate force and follow a trajectory. Nor is it identical to cerebellar ataxia, in which prediction, timing, coordination, and error correction can be disturbed across visual and nonvisual tasks. The historical word ataxia identifies disordered coordination in both names, but the disorders arise from different failures within the control hierarchy.
Bilateral parietal injury can produce optic ataxia as part of Bálint syndrome, together with disturbed visual attention and difficulty directing gaze. Unilateral lesions can produce more restricted forms. It is therefore inaccurate to define optic ataxia only as one component of the full bilateral syndrome.
As discussed in the third vision chapter, patient D.F. became an important contrast case because some immediate, visually guided actions were less impaired than her explicit judgments of object form. Optic ataxia appears to reverse that emphasis: perception can be relatively successful while online guidance of the hand fails. The contrast helped establish that visual judgments and visually guided actions can be differentially vulnerable [@goodale1991; @pereninvighetto1988].
The task conditions determine how strongly the dissociation appears. D.F.’s action performance depends on immediate viewing, fixation, and available haptic feedback [@schenk2012]; optic ataxia varies with field, hand, delay, and lesion anatomy [@pereninvighetto1988; @pisella2000pilot]. The cases support differently weighted contributions within interacting networks. They do not require perception and action to operate as independent visual systems.
Optic ataxia is especially informative because it reveals the difference between locating an object in a visual scene and continually relating that object to a moving limb. Naming the cup, describing its orientation, and reaching accurately toward its handle are not interchangeable tests. Each places different demands on vision, attention, body-state estimation, and online correction.
32.10 A different failure: limb apraxia
A person with optic ataxia has particular difficulty guiding the limb with current visual information. A person with limb apraxia can have a different problem: organizing familiar, learned actions even though elementary strength, range of movement, sensation, and comprehension are not sufficient to explain the failure.
Clinical assessment compares performance across tasks to establish the disproportionate disturbance of skilled action. Weakness, sensory loss, aphasia, neglect, and other accompanying deficits must be considered because each can also disrupt the task.
Those tasks do not all measure the same thing. An examiner may ask a patient to:
- imitate an unfamiliar hand or finger posture;
- produce a familiar communicative gesture, such as waving goodbye;
- pantomime using a named tool when no tool is present;
- demonstrate how to use an actual object;
- select the appropriate tool for an object;
- organize several actions into a practical sequence.
A patient can fail one and succeed at another. Imitating a meaningless posture depends strongly on converting a seen body configuration into one’s own joint configuration. Pantomime requires generating an action without the object’s shape, weight, contact surfaces, or mechanical consequences. Actual tool use supplies those constraints, but it can still fail when knowledge of tool–object relations or the organization of the action is disturbed. Pantomime and actual use are therefore not simply voluntary and automatic versions of one stored motor program [@hoeren2014praxis; @goldenbergspatt2009tool].
The traditional labels ideomotor apraxia and ideational apraxia remain common. Ideomotor has generally referred to spatial and temporal errors in producing a familiar gesture, especially to command or imitation. Ideational has been used for impaired tool concepts, misuse of objects, or disorganization of multistep action. The categories are historically useful but do not divide patients into two natural, nonoverlapping diseases. Modern assessment increasingly describes the particular tasks, errors, and lesion connections rather than assuming that one broken box explains them all [@hoeren2014praxis; @buxbaum2014tool].
Limb apraxia has a strong left-hemisphere association, including left inferior parietal, lateral temporal, premotor, and frontal regions and the white-matter pathways connecting them. The anatomy is distributed, and different patterns of disconnection can yield different deficits. Lesion studies support separable profiles involving gesture production and tool-related action rather than one unitary store of skilled movements [@hoeren2014praxis; @buxbaum2014tool; @goldenbergspatt2009tool].
Context can help some patients because a real object supplies visual, tactile, and mechanical information that pantomime removes. Improvement with an object is therefore informative. It is not universal: patients can also misuse real tools or fail to organize a multistep task.
The syndrome is best understood as a disturbance at a different level from paralysis or optic ataxia. The problem is not merely getting force to the limb, and it is not limited to updating a target relative to the hand. It concerns learned relations among gestures, objects, body configurations, and larger actions.
| Clinical problem | Principal failure revealed by the task | What may remain possible | Important qualification |
|---|---|---|---|
| Weakness or paralysis | Generating force or transmitting effective descending and peripheral output | The target and appropriate action may still be understood | Profiles depend on lesion level, pathway, and time since injury |
| Optic ataxia | Using current visual information to guide the hand and update the reach | Object recognition, elementary movement, and some centrally viewed or nonvisual actions | Effects depend on visual field, hand, fixation, delay, and feedback |
| Limb apraxia | Organizing learned gestures or tool-related actions beyond elementary weakness | Some movements, gestures, or contextual actions | Imitation, pantomime, actual tool use, and sequence production can dissociate |
| Cerebellar ataxia | Calibrating timing, prediction, multijoint coordination, and correction | Strength and the intended outcome may be relatively preserved | The exact profile depends on cerebellar territory and task |
32.11 How the experiments identify a control variable
The strongest experiments in this chapter change the relationship between variables that normally travel together. An anti-reach separates a visible cue from an intended destination. Reverse pliers separate finger movement from object capture. A target jump changes the destination after movement has begun. Each comparison creates competing predictions about the activity or behavior to be measured.
Kinematic recording tracks the eyes, arm, wrist, and digits to establish what actually moved and when. Force sensors and electromyography add the mechanical load and muscle recruitment. These measurements are essential companions to neural recording: a response apparently related to a goal may also track a change in posture, grip, or force. Measuring those relationships allows the interpretation to become more specific.
Single-neuron recording follows activity on the timescale of cues, preparation, movement, and feedback. The delayed-reach and action-observation studies use that timing to ask what information is present before a later event occurs. Functional imaging extends the comparison to distributed human networks, including reach, grasp, tool use, and observation. Its spatial coverage comes at a different temporal and cellular scale, which is why overlap between two fMRI conditions and matching responses in one neuron are different findings.
Perturbation asks whether changing part of the system changes the action. Target jumps and mechanical disturbances perturb the task or limb. Reversible inactivation, electrical stimulation, and TMS perturb neural activity. Their interpretation depends on where, when, and how strongly the system is disturbed. A parietal TMS pulse during online correction addresses a different causal question from stimulation during initial preparation.
Clinical lesions reveal failures that an intact system normally prevents. Optic ataxia exposes the demands of relating a target to a moving limb; apraxia exposes the organization of skilled action. Lesions can affect several cortical fields and connecting pathways, so the most useful explanation relates the patient’s task profile to the affected network rather than treating the diagnosis as one missing module.
These approaches support one another. Neural selectivity identifies a candidate relation; careful behavioral measurement tests its specificity; perturbation and clinical evidence test its contribution to successful action. The BCI takes a further step by placing a neural readout inside a functioning control loop.
32.12 Brain–computer interfaces create a new route for action
A person can retain an intention to reach when injury has interrupted the normal route from cortex to the arm. A brain–computer interface, or BCI, establishes another route: neural activity is recorded, a decoder converts features of that activity into control signals, and a device acts on those signals. The controlled output may be a cursor or a robotic limb.
The examples in Chapter 31 introduced cortical control of robotic reaching and grasping in monkeys and people [@velliste2008bci; @hochberg2012bci]. The human study established that useful movement-related information can remain available when the usual output pathway is unavailable. The present chapter adds two questions. At what level of the action should the interface obtain its control signal? And how will the user learn what the device has actually done?
32.12.1 Decoding a destination and controlling its achievement
The reach studies provide several possible variables for a decoder: intended destination, direction, hand configuration, and the changing state of a movement. Choosing among them assigns different responsibilities to the user and the machine. A system that receives a target selection can use a robotic controller to generate much of the trajectory. A system driven by continuous movement signals gives the user more ongoing control but also requires a reliable stream of updates. Real interfaces can combine levels of control.
Aflalo and colleagues implanted recording arrays in the posterior parietal cortex of a person with tetraplegia. The participant imagined movements while neural activity was recorded. The investigators could decode intended goals, trajectories, and aspects of the imagined action, and use this activity to control external devices [@aflalo2015parietal]. This connects the primate reach studies to a human application: parietal activity related to an action’s intended destination can be available even when the person cannot produce the corresponding limb movement.
The decoder learns a relationship between measured activity and the control variable defined by the task. During calibration, instructions or observed device movements provide examples against which neural activity can be fitted. Once control begins, the user sees the consequences and can adjust the next attempt. A useful interface therefore includes the user’s learning as well as the algorithm’s mapping.
The reach goal is only one part of the achievement. Motors must generate force, the robotic joints must follow a feasible path, and the gripper must establish stable contact. Some of these operations may be handled by the device’s own controllers. The relevant scientific description should identify which choices came from decoded neural activity and which operations were supplied by the machine. This makes the accomplishment more intelligible: it is a partnership among a nervous system, an engineered controller, and a physical effector.
32.12.2 The artificial hand must return information
A hand provides a continuous report of its contact with the world. Without that report, successful control places a heavy burden on vision. The user must watch for the moment of contact, infer whether the grip is secure, and detect whether an object has begun to slip. The precision-grip experiments earlier in this chapter explain why a stream of outgoing commands is only half of an interface.
Flesher and colleagues added tactile feedback to intracortical control of a robotic arm in a participant with tetraplegia. Motor-cortical activity controlled the device, while contact-related information from the robotic hand drove electrical microstimulation in somatosensory cortex, producing sensations referred to the participant’s hand. The participant performed object-manipulation tasks with vision available in both conditions. Adding the stimulation-based feedback improved performance, reducing the median time to complete a successful trial on the tested clinical assessment from 20.9 to 10.2 seconds [@flesher2021tactile].
The return pathway improved the action. This is the same principle encountered in natural grasping: evidence about contact allows control to change when the object has actually been secured. The demonstration came from one participant and a particular interface. It establishes the functional value of the added feedback under those conditions, not the restoration of ordinary touch in all its richness.
An artificial sensory channel also raises a learning problem. Its pattern must become useful to the person who receives it. A sensation in the hand can signal contact with a robotic finger even though the biological finger has not moved. With practice, such consequences can become part of the relation between an intended action and its expected sensory outcome.
32.12.3 The interface reveals the organization of control
The tool experiment and the BCI experiment share a useful principle: the controlled consequence can lie beyond the biological hand. In the pliers task, the hand operates an implement; in an intracortical BCI, neural activity reaches a device through an electronic pathway. In both cases, the system succeeds when activity is brought into a reliable relationship with an outcome and its feedback.
This is why BCI belongs within the science of action rather than appearing only as an engineering application at its edge. Changing the output pathway exposes the division of labor among goal specification, movement generation, contact sensing, and correction. It also makes the costs of delay and uncertainty visible. A decoder may estimate what the user wants while the device still arrives late, encounters resistance, or grasps unsuccessfully. Continued control depends on discovering and correcting that discrepancy.
The research demonstrations described here used implanted electrodes in controlled settings. Their scientific achievements are distinct from the further work of making an interface dependable in everyday use: stable signals, manageable calibration, reliable feedback, and operation across changing tasks. Those are control problems as well as engineering ones. The next chapter’s account of prediction and adaptation will explain why they are so persistent.
32.13 From correction to prediction
Online feedback solves much of the problem, but it creates another. Visual, proprioceptive, and tactile signals arrive after delays. By the time visual cortex receives evidence that the hand has drifted, the hand has moved farther. By the time a corrective command reaches muscle and changes force, the body and target may have changed again.
Slow actions can tolerate more delayed correction. Fast actions cannot simply wait. The controller needs an estimate of the body’s current state, not only a record of its recent state. Motor-related signals can help anticipate the sensory consequences of an action; recent sensory evidence can correct the estimate; repeated errors can recalibrate the relation between command, body, and outcome [@flanaganwing1997internal].
Prediction does not replace feedback. A prediction is useful because later sensory evidence can test it. Feedback is useful because it updates a system that is already acting on an estimate. The distinction between feedforward and feedback control therefore describes different contributions within one recurrent process, not two independent modes that take turns.
This chapter has shown why such estimation is necessary. Orienting must remain coordinated with reaching. Reach and grasp develop together, shaped by the larger action they serve. A target can move after the hand begins; contact can reveal an unexpected load or an unstable grip. A tool or artificial limb introduces another relation between output and consequence. In each case, the controller must estimate the state on which its next adjustment will act.
The next chapter turns to the predicting machine that is especially important for calibrating these relations: the cerebellum. Its contribution will not be a finished movement plan added on top of the systems described here. It will be the continual improvement of prediction and correction within the same embodied control hierarchy.
Established findings. Visually guided action depends on recurrent interactions among sensory, parietofrontal, subcortical, spinal, and biomechanical systems. Primate tasks can separate neural activity related to a visual cue, a forthcoming reach, tool-mediated object capture, and a larger action sequence. Individual neurons can respond during both execution and observation. Intracortical interfaces can use movement-related activity to control external devices, and an artificial tactile return channel has improved manipulation in a human participant.
The organizing interpretation. These results fit a system in which several levels of control maintain relations among a body, an environment, and an intended consequence. Goals influence movement as it develops; sensory evidence continues to revise it. The clinical dissociations reveal different vulnerabilities within that interacting system.
Open questions. We still need to establish how the contributions of reference frames change during natural behavior, how action goals are learned and generalized, and when observation-related motor activity makes a necessary contribution to understanding another’s action. For BCI, a central challenge is maintaining useful control and sensory feedback across time, devices, and everyday tasks.