The visual representation of space in the primate brain

Saccadic suppression

These are only two of many examples of saccadic misperception. In addition to other modulatory effects (e.g., on temporal or numerosity perception, see below), vision as a whole is markedly influenced by saccades. Contrast sensitivity is reduced, with the decrease starting to evolve already prior to saccade onset and lasting until 100 ms thereafter. This effect is called saccadic suppression (Diamond et al., 2000). Figure 1 illustrates an experimental approach to investigate saccadic suppression and its time course. Remarkably, reduced visibility is predominantly found for information processed in the dorsal (i.e., motion sensitive) visual cortical pathway (Burr et al., 1994). Furthermore, the finding that saccadic suppression starts to evolve already before saccade onset suggests that it is foremost an active process likely playing a key role in visual perceptual stability.

Figure 1:

(A) Schematic of the time course of an experimental paradigm to test for saccadic suppression. Across trials, a brief luminant visual stimulus is shown at various times relative to a saccade. Colored lines indicate the (exemplary) time of stimulus presentation (red), example horizontal and vertical eye traces (blue), and the timing of the fixation and saccade target (purple and green, respectively). (B) Comparison of behavioral and physiological measures of saccadic suppression: for the behavioral data (black line. Data from: Diamond et al., 2000), the horizontal axis shows the time of stimulus presentation relative to saccade onset, and the right vertical axis indicates normalized contrast sensitivity. Neuronal data, as detailed in the next section, were shifted along the time axis to correct for response and processing latencies and represent neuronal excitability (left vertical axis) of populations of neurons from macaque areas MT and MST (cyan and blue curves, respectively) and area VIP (red curve). The time course of neuronal excitability in all three motion areas of the macaque revealed a good qualitative match with the time course of perceptual loss of sensitivity around saccades in human observers. Adapted and modified from Bremmer et al., J Neurosci., 2009. Copyright 2009 Society for Neuroscience.

Not only saccades, but also smooth pursuit modulates visual spatial perception. Perceived stimulus locations are shifted in the direction of pursuit, with the error being dependent on the stimulus location with respect to the fovea (van Beers et al., 2001; Dowiasch et al., 2020). Like for saccades (see below), a combined neurophysiological and modeling approach provided an idea about the underlying neural mechanisms of this perceptual illusion (Dowiasch et al., 2016).

Self-motion and navigation

Not only foveating, but also reflexive eye movements like the optokinetic nystagmus (OKN) induce systematic spatial mislocalization (Kaminiarz et al., 2007). OKN is induced either by frontoparallel motion or by rotational movements. During everyday life, however, we often navigate in full 3D space, dominated by translatory forward self-motion. For successful navigation, we need to monitor our self-motion direction (heading) and our travel distance (path integration). Without eye movements, heading perception is close to veridical. Typical errors are in the range of 2°. This performance can be well explained by the readout of populations of neurons being responsive to the direction of (visually simulated) self-motion (Lappe et al., 1996; Schmitt et al., 2020). Eye movements introduce distortions of the optic flow field (Matthis et al., 2022). However, humans can compensate for such spatio-temporal warping induced by smooth eye movements during heading judgements (Lappe et al., 1999) and a neural signature of this ability had been identified in monkey extrastriate and parietal cortex (Bremmer et al., 2010; Kaminiarz et al., 2014). It was only recently that also the effect of saccades on heading perception was investigated (Figure 2). In these experiments, brief (40 ms) self-motion stimuli, simulating self-motion across a ground plane in various directions, were presented perisaccadically (Bremmer et al., 2017). Indeed, saccades led to a perceptual compression of heading towards the line of sight with a time-course very similar to those of the above-described effects of perisaccadic modulations of vision. Based on neurophysiological recordings in monkeys and modeling, we now have a good understanding of the neural basis of this visual perceptual illusion.

Figure 2:

Panels A and B show the temporal sequence (A) and spatial layout (B) of the experiment. Stimuli were presented on a tangent screen covering the central 81° × 65° of the visual field. Across trials, simulated self-motion was pseudo-randomized in one of five directions: forward to the left (−30° and −15°), straight ahead (0°), or forward to the right (15° and 30°). Self-motion stimuli consisted of five consecutive frames of 100% coherent dot motion, i.e., lasting 40 ms. Each trial started with presentation of the fixation target and the stationary ground-plane stimulus. After a randomized time, the fixation target was switched off and the saccade target was switched on (until the end of the trial), inducing a visually guided upward saccade of 10°. Across trials, the onset of the self-motion stimulus ranged from about 200 ms before to 200 ms after saccade onset. At the end of each trial, the saccade target and the ground plane stimulus were switched off and a ruler with a random sequence of numbers was presented on the screen. Subjects had to indicate via keyboard input the number on the ruler that appeared closest to their perceived heading direction. Panels C and D indicate the time course of compression of perceived heading. Responses from one example subject are shown in C. Symbols represent data from single trials: upward pointing triangles for heading to the right (magenta: +15°, red: +30°), downward pointing triangles for heading to the left (dark cyan: −15°, green: −30°), and circles for heading straight ahead (blue, 0°). Solid lines represent running means of five consecutive samples each, assigned to the central sample value. Dashed lines show the performance for the same experiment during continuous fixation. Panel D indicates compression of perceived heading, defined as the normalized standard deviation of the five time-courses of perceived heading for all subjects (colored lines). The robust effect was observed in all participants. Maximum compression as indicated by the minimum value of the normalized standard deviation was observed just prior to saccade onset. Adapted and modified from (Bremmer et al., 2017).

Path integration is another key feature of self-motion processing and navigation. It is often tested in the context of homing, i.e., the ability to return to the starting point of a journey after an outbound route including translations and rotations. Homing appears to be a universal feature across the animal kingdom, documented not only in humans (Warren, 2019), but also in insects, rodents, birds, camels and elephants (for reviews, see e.g., Heinze et al., 2018; Poulter et al., 2018).

Saccadic suppression

As discussed above, vision of luminant stimuli is impaired by saccades. These stimulus features are processed predominantly along the dorsal visual pathway. In search for a neural correlate of the behavioral finding, various areas along this pathway in the macaque have been tested for their perisaccadic responsiveness (Bremmer et al., 2009. Figure 1B). As hypothesized, responsiveness of neurons predominantly in motion sensitive areas MT, MST, and VIP were affected perisaccadically. The time-course of this modulation was very similar to human behavioral data. Notably, functional equivalents of all three areas have been identified in humans (Areas MT and MST: Huk et al., 2002; Area VIP: Bremmer et al., 2001), suggesting a functional link between response properties of these areas and saccadic suppression.

Predictive remapping

While saccadic suppression is thought to facilitate visual perceptual stability, we are not blind across saccades (Nicolas et al., 2021). So, the question arises, how we link images of the world as seen before and after an eye movement and if and how this link could contribute to visual perceptual stability. In a seminal study, Duhamel, Colby and Goldberg provided first evidence for a potential neural basis of the behavioral effects (Duhamel et al., 1992). The authors had trained macaque monkeys on a visually guided saccade task and probed the responsiveness at the neuron’s current and future receptive field at various times relative to saccade onset. Remarkably, about 40% of neurons in area LIP started to respond to stimuli presented at the future RF even before the saccade, i.e., when the eyes still fixated the initial target, a phenomenon termed predictive remapping. Neurons of this type anticipate the sensory consequences of an upcoming saccade. Similar functional characteristics have also been observed in other brain structures involved in the control of saccadic eye movements, especially the frontal eye fields (FEF) in frontal cortex and the midbrain superior colliculus (SC). It thus seems that saccadic control areas are also involved in facilitating visual perceptual stability.

Gain fields

Predictive remapping provides a plausible neural basis for visual perceptual stability. Yet, it cannot account for perisaccadic distortions of space perception, i.e., shift and compression. An alternative hypothesis has been put forward by Morris, Krekelberg and Bremmer (Morris et al., 2012). The authors studied the dynamics of so-called gain fields, i.e., an influence of the position of the eyes in the orbits on neuronal discharges (Bremmer, 2000; Morris et al., 2013, 2016). Modeling studies showed that such response properties are suited to represent visual information in a head-centered frame of reference (Bremmer et al., 1998; Dowiasch et al., 2016). Different from previous studies, the approach of Morris and colleagues allowed to probe for the temporal dynamics of eye position effects across saccades. Results were surprising: average activity of a population of neurons from parietal cortex (areas MT, MST, LIP, and VIP) started to change prior to saccade onset. Yet, the dynamics of this change in activity were slower than the eye movement itself. This functional characteristic allows to explain the above described perisaccadic shift, thereby suggesting that gain fields play an important role in spatial perception.

Perisaccadic modulation of visual receptive fields

A shift of perceived spatial locations is one of two types of perisaccadic mislocalization. The other is a so-called perisaccadic compression. Krekelberg and colleagues (2003) could show that such a compression of visual perceptual space can be traced to a representation of retinal position in cortical areas MT and MST of the monkey. An alternative explanation was put forward by Goldberg and colleagues, demonstrating a perisaccadic expansion of the visual receptive fields of neurons in macaque area LIP (Wang et al., 2016). Like for area MT and MST, also for area LIP a functional equivalent has been identified in human parietal cortex (Konen et al., 2004), suggesting a leading role of these areas for the perisaccadic modulation of space perception.

Head-centered encoding of visual space

In all findings described above, there was an implicit assumption or even explicit statement: except for brief, perisaccadic modulations, the location of a visual receptive field with respect to the fovea does not change. This, however, is not the case. In another seminal work, Duhamel and colleagues could show that about one third of the neurons in macaque area VIP show head-centered visual receptive fields (Duhamel et al., 1997). As of today, it is unclear how neurons in area VIP achieve this remarkable response behavior. While speculating, this could mean that these neurons receive input from the whole retina (via projections through the LGN and visual cortex), but only part of this input is gated for a given eye position. Follow-up studies revealed that response latencies of VIP neurons encoding visual space in head-centered coordinates are on average longer than response latencies of neurons encoding visual space in an eye centered reference frame (Avillac et al., 2005; Schlack et al., 2005). This might suggest that it takes the brain a few 10 milliseconds to transform visual spatial information from the initial, i.e., eye centered encoding to a head-centered encoding. More experimental and computational work, however, is needed to better understand this remarkable finding.

Self-motion

Today, we have a rather good understanding of the basic principles of the neural processing of self-motion information, especially heading (Noel and Angelaki, 2022). Nevertheless, the control of self-motion can be a challenging task. Under certain circumstances, immediate adjustments might be required to keep on track. Hence, it would appear advantageous if the processing of self-motion information was predictive and quasi-reflexive, i.e., independent from attentional load. Predictive coding is suggested to facilitate sensory processing by attenuating responses to predictable sensory information and enhancing responses to unpredicted events like unexpected changes in heading (Friston, 2018). Along the same vein, a preattentive processing of self-motion direction could accelerate and thereby facilitate successful navigation irrespective of cognitive load. In this respect, heading would be different to path integration whose accuracy has been shown to be modulated by a secondary task (Glasauer et al., 2009).

A specific electroencephalography (EEG)-component, the so-called visual mismatch negativity (vMMN), has been suggested to be indicative of predictive and preattentive processing of sensory stimuli (Stefanics et al., 2018). The MMN is typically recorded in an oddball experiment employing standard and deviant stimuli, presented typically in 80 and 20% of the trials, respectively, and it is computed as the difference between two event-related potentials (ERPs). Deviant stimuli elicit a more negative N2 ERP-component than frequently presented standard stimuli, which leads to the MMN (deviant-standard). In the predictive coding framework, the MMN carries the prediction error signal elicited by the mismatch of a sensory event with the predictions formed by prior experience.

A recent study tested the hypothesis of a predictive encoding of visual self-motion information in humans and macaque monkeys with similar experimental protocols (Schmitt et al., 2021). Participants were presented visually simulated self-motion (Forward to the left and right) across a ground plane. Visual evoked potential (VEPs) for identical self-motion directions showed a visual mismatch negativity (vMMN) between standard and deviant trials in both humans and monkeys (Figure 3) thereby suggesting a predictive encoding of heading.

Figure 3:

(A) The experimental paradigm. Each trial started with a stationary ground plane stimulus, followed by a stimulus mimicking self-motion (blue arrow) into one of two directions. (B) Topographic maps show the mean difference of the ERPs for identical headings as recorded in deviant and standard trials. The analysis window ranged from 127 to 143 ms after self-motion onset. The MMN is indicated by the blue color over parietal and occipital electrodes. It was slightly lateralized with a stronger MMN for contraversive headings. (Adapted and modified from Schmitt et al., 2021).

Navigation

Numerous studies in rodents as well as primates – including humans – have provided clear evidence for a causal involvement of the hippocampus and the parahippocampal, entorhinal and retrosplenial cortices in spatial navigation, scene detection and spatial memory (e.g., Moser et al., 2017). Different functionally defined classes of neurons are considered to contribute to establishing of what has been termed a cognitive map, i.e., a representation of the spatial environment to support locating oneself and to the guidance of future navigational action (McNaughton et al., 2006). Especially the role of the hippocampus has been unveiled by these studies. This critical involvement in (real or simulated) spatial navigation also in humans has recently been further substantiated by intracranial recordings in presurgical epileptic patients (Kunz et al., 2019). While being invasive, only this approach allows for an understanding of visual spatial processing at the highest possible spatial and temporal resolution.

There appears to be a mutual exclusion of research referring to the (para-)hippocampal formation and related brain regions and their role for spatial navigation and work investigating self-motion responses at the visual cortical level. This is surprising given that it appears obvious if not imperative that both sub-systems are functionally linked. As of today, only very few studies have aimed to unveil links between both networks. Research on monkeys showed that a subset of MST neurons exhibits functional properties similar to hippocampal place-cells (Froehler and Duffy, 2002). Likewise, fMRI studies in humans aimed to connect work on self-motion processing and its related cortical activation with navigational tasks activating the hippocampal formation and related structures (Sulpizio et al., 2020).

The mental number line

The most frequently used example for the link between numbers and space is the SNARC effect (spatial numerical association of response codes). In a SNARC experiment, human participants show shorter reaction times to the left for small numbers and to the right for large numbers, when judging number-parity with button-presses using the left and right hand. In general, the SNARC effect is seen as an indication of the concept of a mental number line (MNL), i.e., a fixed link between space and numbers. More recent behavioral research even suggested an orientation of this mental number line in full 3D space, with smaller numbers being represented left, down, and near, and larger numbers being represented right, up, and far (Aleotti et al., 2020; Hesse and Bremmer, 2017).

Eye movements affect spatial, temporal, and numerical perception

We have detailed above that eye movements induce characteristic misperceptions of space. One of these illusions was a compression of perceived space. If space, time, and number were treated similarly at the neural level, saccades should also modulate temporal and numerical perception. This is, indeed, what behavioral experiments have shown. In a first set of studies, participants were asked to compare the time intervals between two pairs of extended horizontal bars while they made large horizontal saccades. The first pair was a test stimulus, with the interval between flashed bars being fixed at 100 ms, presented at unpredictably varying times relative to the saccade. The second pair was a probe stimulus of variable interval, presented 2 s after the test. Temporal perception during steady fixation was close to veridical. Briefly before a saccade, however, with the eye yet not moving, the 100 ms test interval appeared as being only 50 ms long, suggesting that subjective time had been compressed by a factor of two (Morrone et al., 2005). A similar result was found for an abstract concept of quantity. In these experiments, subjects consistently underestimated the results of rapidly computed mental additions and subtractions, when the operands were briefly displayed before a saccade (Binda et al., 2012). However, the recognition of the number symbols was unimpaired. These results are consistent with the hypothesis of a common, abstract metric encoding magnitude along multiple dimensions: space, time, and number.