Hand-related rather than goal-related source of gaze-dependent errors in memory-guided reaching
+ Author Affiliations
Abstract
Mechanisms for visuospatial cognition are often inferred directly from errors in behavioral reports of remembered target direction. For example, gaze-centered target representations for reach were first inferred from reach overshoots of target location relative to gaze. Here, we report evidence for the hypothesis that these gaze-dependent reach errors stem predominantly from misestimates of hand rather than target position, as was assumed in all previous studies. Subjects showed typical gaze-dependent overshoots in complete darkness, but these errors were entirely suppressed by continuous visual feedback of the finger. This manipulation could not affect target representations, so the suppressed gaze-dependent errors must have come from misestimates of hand position, likely arising in a gaze-dependent transformation of hand position signals into visual coordinates. This finding has broad implications for any task involving localization of visual targets relative to unseen limbs, in both healthy individuals and patient populations, and shows that response-related transformations cannot be ignored when deducing the sources of gaze-related errors.
Introduction
When humans indicate remembered locations, their responses are often influenced by gaze direction (Fiehler, Rösler, & Henriques, 2010; Harrar & Harris, 2009; Lewald, 1998; Poljac, Neggers, & van den Berg, 2006). While such gaze-dependent errors are highly task-dependent (Dessing, Crawford, & Medendorp, 2011; Lewald & Ehrenstein, 2000; McGuire & Sabes, 2009), they are obviously linked to the eye, so it is generally assumed that they provide a direct window into early visuospatial representations of target location. A case in point is the supposition that human short-term visuospatial memory utilizes a gaze-centered representation of space that is updated with each eye movement. This theory is supported by several converging lines of evidence (Duhamel, Colby, & Goldberg, 1992; Batista, Buneo, Snyder, & Andersen, 1999; Medendorp, Goltz, Vilis, & Crawford, 2003; Merriam, Genovese, & Colby, 2003), but was originally inferred in humans by comparing reach errors measured before and after gaze shifts (e.g., Henriques, Klier, Smith, Lowy, & Crawford, 1998; see also Dessing et al., 2011; Khan et al., 2005; Vaziri, Diedrichsen, & Shadmehr, 2006).
Humans typically overshoot remembered visual targets relative to gaze when reaching in the dark (Bock, 1986; Enright, 1995; Henriques & Crawford, 2000; Henriques et al., 1998; Lewald & Ehrenstein, 2000; Vaziri et al., 2006). These overshoots are exaggerated in parietal-damaged patients with optic ataxia (Khan et al., 2005, 2007). To date, all studies of gaze-dependent reach errors assumed that these errors reflect misestimates of target position in the visuomotor transformation (see also Beurze, van Pelt, & Medendorp, 2006; Blohm & Crawford, 2007; Khan et al., 2007; McGuire & Sabes, 2009; Schlicht & Schrater, 2007) either because target direction is overestimated relative to gaze (Henriques et al. 1998) or gaze direction is underestimated relative to the target (McGuire & Sabes, 2009). In either case, such misestimates would result in the gaze-dependent reach overshoots illustrated in Figure 1A.
Reaches in the dark to a remembered visual target show a leftward bias in reaching when gaze is deviated to the right (left column) and a rightward bias when gaze is deviated to the left (right column) of the target. The traditional explanation for these gaze-dependent reach errors (A) is that they stem from misestimates of target position (grey dot: actual target position; orange dot: misestimated target position). The alternative possibility (B), tested here, would be that the reach errors stem from misestimates of final hand position (grey: actual arm position; orange: misestimated arm position). In both panels, the bias is indicated by red arrows.
However, these previous accounts ignore the possibility that gaze-dependent reach errors instead arise from misestimates of the manual response, i.e., if subjects underestimate the angle between gaze and final hand position (Figure 1B). Although some authors have considered the possibility that initial hand position modulates gaze-dependent reach errors (Beurze et al., 2006; Khan et al., 2007), no one has considered the possibility that a misestimate of hand position is actually the source of these errors. It has simply been assumed that anything gaze-related must originate from goal-related transformations.
Here, we examined the contribution of misestimates of hand position to gaze-dependent reach errors by manipulating hand visibility during the reach. Previous studies suggest that hand position is derived predominately from vision, as opposed to proprioception, when vision of the hand is available (McGuire & Sabes, 2009; Saunders & Knill, 2005; Sober & Sabes, 2003; Tagliabue & McIntyre, 2011). Thus, if misestimates of hand position underlie gaze-dependent reach errors (Figure 1B), online visual feedback of the hand should strongly suppress or eliminate those errors. Conversely, if these errors stem from misestimates of target position (Figure 1A), online visual feedback of the hand might improve reaches, but these reaches would still be aimed at the wrong position. To summarize the results of the actual experimental test, visual feedback completely suppressed gaze-dependent reach errors, supporting the new hypothesis that these errors stem from misestimates of hand position. This finding provides a striking demonstration that response-specific transformations must be accounted for in any attempt to infer cognitive processes from gaze-dependent behavioral errors.
Methods
Subjects
Six healthy, right-handed (Oldfield, 1971) subjects (four male, two female; mean age 28 years, range 22–34 years) with normal or corrected-to-normal vision participated in the experiment. They signed an informed consent prior to the experiment. All procedures of this experiment were in accordance with the Declaration of Helsinki and approved by the local ethics review board.
Set-up
Subjects sat in the complete dark behind a table in a chair of adjustable height with their heads immobilized using a personalized dental impression fixed to a rigid post and using a head-rest. They had to touch a remembered target position with their right index finger, while fixating to the left of, at, or to the right of the target throughout the entire trial. Red and green LEDs (Ø 3 mm) served as fixation points and reach targets, respectively. These were attached behind a plastic screen (with a dark coating blocking 95% of visible light) that was mounted in front of a 21 inch CRT screen (Dell, 1280 × 1024 pixels). The CRT screen was used to calibrate the EyeLink II system (SR Research, Mississauga, ON, Canada), which recorded right eye-in-head orientation. Finger movements were tracked using Optotrak (Northern Digital, Inc., Waterloo, ON, Canada; 3020 camera; Certus control unit; 100 Hz; positive x-axis pointing rightward, positive y-axis pointing down, positive z-axis pointing into the screen, all relative to the upper left corner of the screen). We attached a cluster of six markers to the finger. In addition, a red LED (Ø 5 mm) was taped to the right index fingernail.
Trial conditions and presentation order were controlled by a custom-written Matlab program (The Mathworks, Nattick, MA) using Psychophysics Toolbox extensions (Brainard, 1997). This program also sent synchronization pulses to Optotrak's digital acquisition unit, sampling at 1000 Hz, to define trial onset and offset in the Optotrak data files. This data acquisition unit also received the analog Eyelink signals (horizontal and vertical position of the right eye).
Procedure
After we explained the procedure, the subject signed the informed consent and completed the handedness questionnaire. Subjects were instructed to touch the screen at the remembered position of the red target LED as accurately as possible; they were explicitly told there was no time constraint. A bite bar was made out of dental impression compound (Kerr Corporation, Orange, CA). With the subject in the set-up, we calibrated the fingertip shape (as described below) and the EyeLink system before starting the actual recordings.
A trial started when the subject pressed a button (horizontally aligned with, 265 mm in front of, and 40mm below screen center) for 250 ms. Upon trial onset, the green fixation LED was illuminated for 1000 ms, followed by the red target LED for another 1000 ms. After an additional memory delay (1000 ms), an initiation cue (a 50 ms beep) was provided. In trials with online visual feedback, the red LED on the finger was illuminated immediately upon release of the button; this LED was the only light-source during the reach. None of the LEDs were on simultaneously, preventing the subjects from memorizing relative visual positions. After touching the screen, subjects returned their finger to and pressed the button, triggering two desk lights (40W incandescent bulbs) to turn on for 1000 ms (to prevent dark adaptation), after which the next trial started.
We used two target positions (−5° and 5°) and three fixation positions, relative to these targets (−10°, 0°, 10°). These six combinations were repeated 15 times within each session. The resulting 90 trials were presented in random order across three blocks (with two practice trials at the start of each block), which were presented after a practice block of 20 trials. Trials in which subjects released the button before the beep were recycled at a random position within the remainder of the block. Each session, including calibration of the Eyelink and Optotrak systems, lasted about 60-75 minutes. The two sessions, with and without the finger LED turning on during the reach, were run on separate days, with their order counter-balanced across subjects.
Data analyses
We ran our data analyses of the eye and finger movements in Matlab and statistical analyses in SPSS (IBM, Armonk, NY). We calibrated the shape of the finger tip relative to the markers on the finger by tracing the fingertip with an Optotrak pointer tip (∼10 times). The pointer tip positions were projected on the plane (determined by the first two principal components) and rotated within this plane by an angle that resulted in the best fit of a sixth order 2D polynomial through this data. The finger outline was defined as positions at 0.1 mm intervals along this polynomial. These could be reconstructed as virtual points in the rigid body coordinates of the finger markers.
Our main dependent variable was the horizontal reach error. We defined finger-screen contact to occur as the first sample at which the z-velocity of the fingertip was lower than 10 mm/s, provided the fingertip was within 15 mm of the screen. We took the most forward position (along the z-axis) on the fingertip at finger-screen contact as the contact position. After each session, subjects made 10 reaching movements to each target LED while looking at it in a dimly lit room (i.e., they could see their fingertip). We calculated the reach errors relative to the average fingertip positions recorded in the latter trials for each target (instead of the physical target positions), which means that our definition of the reach errors eliminates any individual systematic biases.
Eye position traces were analyzed using a custom Matlab program, which performed an automatic drift correction on the mean eye orientation attained in the last 800 ms the green fixation LED was on. The drift-corrected eye traces were used to exclude trials from the analysis; this was done when the horizontal eye orientation deviated more than 2° from the required fixation direction between target onset and finger-screen contact. On average, we excluded 11.4 (SD = 7.7) trials per session (which also included two trials across subjects in which finger-screen contact could not be determined).
We analyzed the horizontal reach errors in SPSS using a fully factorial Online visual feedback × Target position × Gaze direction linear mixed model in which each factor had a fixed (population) and random (individual) component; we allowed unique between-subject variances for all non-redundant levels of all factors by using SPSS' ‘Diagonal' covariance structure; degrees of freedom were obtained by SPSS using Satterthwaite's approximation. The effect of Gaze direction was expected to be linear for the used range of gaze directions and thus implemented as a covariate. We directly compared the effect of Gaze direction between reaches with and without online visual feedback using the Online visual feedback × Gaze direction interaction. To test whether these slopes differed significantly from zero, we conducted separate Target position × Gaze direction linear mixed models for trials with and without online visual feedback.
We also analyzed whether online visual feedback influenced the biases when looking at the target. Because the magnitude and sign of these ‘gaze-at-target' biases can differ across subjects, analysis of the intercepts in our linear mixed model is inadequate. The question of whether the gaze-at-target bias decreases with online visual feedback is in fact determined by the absolute value of the individual intercepts. We therefore estimated the intercepts for all conditions/subjects from separate linear regression models of reach errors as a function of Gaze direction. The average absolute intercepts for reaches with and without online visual feedback were compared using a sign test (because absolute values are not normally distributed). The alpha level in all analyses was set to 0.05. For the purposes of the present paper, we did not focus on the effects of Target position. We conducted the same analyses on the vertical reach errors, confirming that the reported effects were unique to the horizontal direction.
Results
In our control experiment subjects reached in total darkness towards remembered visual targets while fixating to the left, toward, or to the right of one of two possible target positions. In this situation, subjects showed the typical gaze-dependent target overshoots that have been reported in many previous studies. This pattern is illustrated in Figure 2A, using top views of the average movement paths. Figure 2C isolates the gaze-dependency by subtracting paths for central fixation from paths for left and right fixation. As expected, reaches without visual feedback of the finger overshot the target relative to gaze, resulting in rightward errors for fixation left of the target and leftward errors for fixation right of the target. These errors arise smoothly from reach onset until screen contact.
Top view of average movement paths and their 95% confidence intervals across subjects (color coding—including overlap—indicated between the panels). Panels A and B show the average movement paths for all three fixation directions without and with online visual feedback, respectively. The 95% confidence intervals overlap considerably because different subjects followed slightly different paths. To isolate the actual effect of gaze direction, we subtracted the paths (i.e., their lateral coordinates) for central fixation from the paths for leftward and rightward fixation. Panels C and D depict the averages of these two relative movement paths across subjects and their 95% confidence intervals, for trials without and with online visual feedback, respectively. The depth coordinates of these paths (i.e., vertical dimension in the figure) reflect the average across Gaze directions.
In our new experimental condition, subjects performed the same task but could see an LED on their fingertip from the time of reach initiation until screen contact. The results are illustrated in Figures 2B and D, using the same conventions described above. Online visual feedback of the finger completely suppressed the gaze-dependent effects, causing the finger to follow the same path whether fixation was left, right, or center (Figure 2B, D).
Task behavior was quantified using the horizontal reach errors at the point of finger contact with the stimulus screen. Figure 3 depicts these for individual subjects as a function of gaze angle relative to the target without visual feedback (A) and with visual feedback of the finger (B). As expected, there was variability in the patterns of individual subjects (Henriques et al., 1998; Henriques & Crawford, 2000), but influence of visual feedback was clear for all subjects (A vs. B). The average responses in Figure 3C illustrate that the typical overshoot pattern (black lines; leftward fixation: 3.0[1.3]°; central fixation: 1.0[1.7]°; rightward fixation −1.3[2.1]°; slope: 0.226[0.071]; linear mixed model: F[1, 5.12] = 55.20, p < 0.005) was completely suppressed by online visual feedback (Gaze direction × Online visual feedback interaction: F[1, 6.68] = 159.86, p < 0.0005), to the point that Gaze direction had no significant effect (Figure 3C; red lines; leftward fixation: 0.6[1.1]°; central fixation: 0.2[0.4]°; rightward fixation 0.0[1.1]°; slope: 0.0089[0.091]; F[1, 5.01] = 0.68, p > 0.44). We found that errors occurring when subjects looked directly at the remembered visual target were also smaller with online visual feedback (median 1.4°; range: 0.9°–4.1°) than without (median 5.3°; range: 3.3°–6.8°; sign test: p < 0.05; see Methods). These results confirm that misestimates of hand position underlie gaze-dependent errors as well as gaze-at-target errors in memory-guided reach (Figure 1B).
Gaze dependency of average reach errors in degrees of our subjects (n = 6), indicated by different symbols and colors (circles). A. Data for reaching without online visual feedback (OVF). B. Data for reaching with OVF. The individual data is aligned with the group mean for reaches when looking at the target, to emphasize the differences in slope. C. Average reach errors (±SEM), showing that OVF suppresses the effect of Gaze direction.
Discussion
This study examined the origin of gaze-dependent overshoots when reaching in the dark. These errors have formed the basis for numerous psychophysical studies of the sensorimotor control (e.g., Beurze et al., 2006; Bock, 1986; Henriques & Crawford, 2000; Henriques et al., 1998; Khan et al., 2007; Lewald & Ehrenstein, 2000; McGuire & Sabes, 2009; Schlicht & Schrater, 2007; Vaziri et al., 2006). Although these errors were variable and depended on task conditions (Henriques & Crawford, 2000; Henriques et al., 1998; McGuire & Sabes, 2009), the current study is the first to report a manipulation that causes complete abolishment of gaze-dependent errors in reaches to remembered visual targets (for a partial suppression with vision of the hand position before reach onset, see Beurze et al., 2006; McGuire & Sabes, 2009).
The central question posed in this study was, what is the source of gaze-dependent reach errors to remembered visual targets? Previous studies already rejected the notion that these gaze-dependent errors arise intrinsically within visual representations, because any position on a visual map may be linked to any motor response (Henriques & Crawford, 2000; Henriques et al., 1998; see also Dessing et al., 2011; McGuire & Sabes, 2009). Instead, it is generally agreed that gaze-dependent errors during reaching in the dark likely arise in the necessary reference frame transformations that require gaze direction signals; biases within these signals have been proposed to underlie gaze-dependent reach errors (McGuire & Sabes, 2009; Schlicht & Schrater, 2007). Gaze-dependent errors could potentially arise in either the target-to-body or hand-to-vision transformation. If hand and target position during reach planning are compared in body-centered coordinates (Khan et al., 2007; McGuire & Sabes, 2009; Schlicht & Schrater, 2007; Tagliabue & McIntyre, 2011), a vision-to-proprioception transformation of target position (using gaze direction signals) is needed. Alternatively, if hand and target position are compared in gaze-centered coordinates (Batista et al., 1999; Beurze et al., 2006; Blangero, Rossetti, Honoré, & Pisella, 2005; Buneo, Jarvis, Batista, & Andersen, 2002; Crawford, Henriques, & Medendorp, 2011; Dessing et al., 2011; Khan et al., 2007; McGuire & Sabes, 2009; Tagliabue & McIntyre, 2011), a gaze-dependent proprioception-to-vision transformation of hand position is needed (Figure 1B). Because proprioceptive signals are constantly available, errors within the latter transformation could arise throughout the entire reach. Hand-related errors are not just a late motor response bias; they likely arise from high-level interactions between effector, gaze, and target signals.
Even though a number of previous studies have suggested the existence of a gaze-dependent transformation of hand position signals into visual coordinates (e.g., Batista et al., 1999; Beurze et al., 2006; Blangero et al., 2005; Ren et al., 2007), surprisingly, none of these investigations suggested that this might be the source of gaze-dependent errors observed in memory-guided reach. Here, we considered this alternative for the first time (Figure 1B). When our subjects—who showed gaze-dependent errors when reaching in the dark—could see their finger during the reach, gaze-dependent reach errors disappeared and gaze-at-target biases were reduced. This was expected if these errors stem from misestimates of hand position within a gaze-centered reference frame, because with online visual feedback hand position in this frame would be derived primarily from accurate visual signals; a decreased importance of erroneously transformed proprioceptive hand position signals eliminates reach errors (McGuire & Sabes, 2009; Saunders & Knill, 2005; Sober & Sabes, 2003; Tagliabue & McIntyre, 2011). Reach overshoots relative to gaze suggests that the proprioception-to-vision transformation of hand position underestimates the angle between visual hand position and gaze due to errors in the hand and/or gaze signal (see Fiehler et al., 2010; Harrar & Harris, 2009, 2010).
To explain the observed suppression we posited that visual feedback would increase the reliance on visual signals relative to the erroneous transformed proprioceptive feedback signals. However, it has been argued that with visual feedback reaches are planned purely on the basis of visual signals (McIntyre, Stratta, & Lacquaniti, 1998; Tagliabue & McIntyre, 2011). This hypothesis leaves several alternative explanations of the observed suppression, which we will therefore discuss here. If, with visual feedback, biases arise within the vision-to-proprioception transformation of both target and hand position, the body centered movement plan would be error-free simply because both transformed signals contain a similar error. This explanation depends on the assumption that accurate proprioceptive signals concerning the body-centered hand position would be entirely ignored in the presence of visual feedback. With visual feedback, the reach vector could also be calculated purely in visual coordinates (Tagliabue & McIntyre, 2011). While this possibility is not incompatible with our explanation of the gaze-dependent reach errors, it leaves room for an alternative explanation that still involves erroneous visual target position transformations. If suppression of gaze-dependent reach errors in our study reflects a preference for ‘same-frame' calculations (Tagliabue & McIntyre, 2011), one should also expect gaze-dependent errors to disappear when the reach vector can be calculated from proprioceptive signals alone (i.e., in reaches to proprioceptive or combined visual and proprioceptive targets); this is not the result typically obtained (Jones & Henriques, 2010; McGuire & Sabes, 2009). We prefer our explanation, which requires only one source of error (in the proprioception-to-vision transformation of hand position), requires suppression of erroneous transformations rather than of correct sensory signals, and is compatible with experiments on proprioception-guided reaching. Further experiments (perhaps manipulating the timing of visual feedback) are nevertheless necessary to disambiguate these different explanations.
While our findings suggest that gaze-dependent reach errors stem from hand-related biases, they do not rule out a combination of hand- and target-related error sources. Variations in the relative role of these sources may for instance explain part of the between-subject variability in the error suppression (Figure 3A, B) and results of other paradigms (Henriques & Crawford, 2000; McGuire & Sabes, 2009). It is, however, noteworthy that gaze-dependent errors have been consistently observed in studies that employed reaching or pointing movements of the arm and hand, irrespective of how the reach goal was visually specified (Dessing et al., 2011; Poljac et al., 2006), but are less consistent in experiments that do not involve hand movement (Eggert, Ditterich, & Straube, 2001). This apparent discrepancy is now easily understood from our new hypothesis that these errors arise primarily from misestimates of hand position within a gaze-centered reference frame. Hence, these errors need to be considered in psychophysical studies that use reaching and pointing as an indicator of target position.
Our new hypothesis is still consistent with studies of spatial updating that compared reach errors before and after a saccade, because it predicts that reach errors depend on gaze direction during the reach (Henriques et al., 1998; Dessing et al., 2011; Vaziri et al., 2006). Our hypothesis does not contradict the main conclusion of these studies, because the new model also requires that visual targets are represented and updated relative to gaze, for the purpose of comparison with hand position in visual coordinates. Similarly, our findings suggest that the exaggerated gaze-dependent reach errors shown by optic ataxia patients (Khan et al., 2005, 2007) may in fact be hand-related (see Blangero et al., 2007). This suggestion would, for example, explain why gaze-dependent reach errors in optic ataxia patients with unilateral parietal damage are independent of the visual hemifield that was originally used to view the target (Khan et al., 2005). Our findings thus suggest that enhanced visual feedback of the hand may have practical applications for patients who suffer from optic ataxia.
More generally, our results provide a striking example of the danger of inferring features of ‘early' visuospatial representations from gaze-dependent errors, without considering effector-related transformations. Considering the gaze-dependent nature of the errors in the reach response, it was perhaps natural to assume that they related to vision, but gaze-dependent errors might equally arise from any source that is transformed into visual coordinates (in this case, the hand). Conversely, our data strongly support the presence of this proprioception-to-vision transformation for reach control in the human (Batista et al., 1999; Beurze et al., 2006; Buneo et al., 2002; Crawford et al., 2011; Khan et al., 2007; McGuire & Sabes, 2009). Our new finding has general implications for reach planning whenever visual feedback of the hand is compromised, and for any behavior that depends on localization of visual targets relative to unseen limbs (Fiehler et al., 2010; Harrar & Harris, 2009, 2010), and provides a cautionary warning for all studies that infer brain function from behavior: the motor response itself can interact with ‘early' cognitive representations in unexpected ways.
Acknowledgments
JCD's contribution was supported by NSERC CREATE and CIHR and JDC's contribution by the Canada Research Chair program.
Commercial relationships: none.
Corresponding author: Joost C. Dessing.
Email: joostdessing@gmail.com
Address: Centre for Vision Research, York University, Toronto, ON, Canada.
- Received June 4, 2012.
- Accepted September 10, 2012.
- © 2012 ARVO
