Introduction

The visual system of primates (Figure 1) is one of the key models to study the neural signals and codes that underlie perception (Parker and Newsome, 1998). Over the past half century, in particular the visual system of the Rhesus macaque monkey has been mapped structurally and functionally in unrivalled detail. An important juncture was the decision of the National Eye Institute (NEI/NIH) in the early 70s to work on the visual system of the non-human primate as the closest model to humans (Kupfer et al., 2009). Since the macaque monkey can be trained to detect, discriminate and report the perceptual appearance of visual stimuli, we can relate signals of single neurons in real-time to percept formation. Using causal intervention methods, like electrical microstimulation, we can link brain structures and signals causally to specific percepts (Histed et al., 2013, Cicmil and Krug, 2015). The advent of non-invasive imaging methods, like fMRI, has allowed us to relate the study of neurons in non-human primates to the activation of brain areas in humans and postulate functional homologues (Lippert et al., 2010). The next major challenges lie in deciphering the spatially distributed neuronal signals and interactions that shape complex visual percepts and in exploiting this knowledge for advancement in the design of neuro-prosthetic devices.

Fig. 1:

Visual areas in the primate. Brains and eyes are reconstructed from magnetic resonance images; on the left: rhesus macaque, on the right: human. Generally, the back third of the primate brain processes visual information. In total, there are probably around 30 distinct visual areas in the cortex. Primary visual cortex (V1) receives most of the input from the eyes. Beyond V1, one speaks of extrastriate visual cortical areas. Together areas V1, V2 and V3 are usually considered ‘early’ visual areas. As signals move forward in the brain, what aspects of vision are processed becomes more and more complex. For example, visual motion is processed in areas V5/MT and MST, which in the macaque monkey are burrowed deep in a cortical fold. Aspects of shape are represented in area V4 and objects and faces in IT. For many visual areas, humans and monkeys have directly functional homologues.

Neural codes for visual perception: from single cells to networks

Vision depends fundamentally on the arrival of light at a detecting surface. This is equally true of the mammalian eye as it is for physical detectors of light. Ever since the beginnings of neurophysiology, scientific investigators have sought to probe the relationship between the physical event of light arriving at the detector with the psychological event of reporting a conscious percept (Hecht et al., 1941). At very low light levels, even the arrival of single quantum may be an event that is consciously perceived (Tinsley et al., 2016).

At higher light levels, the vision of humans and other mammals is not limited by the arrival of individual quanta. There is nonetheless a boundary between perceptible and imperceptible differences in sensory stimuli, which was classically referred to as a threshold by Fechner (Elemente der Psychophysik, 1860). A critical conceptual step for understanding the role of central nervous system (CNS) was the formulation of the single neuron doctrine by Horace Barlow (Barlow, 1972, Barlow, 1995). In essence, this proposed that the units of perception (psychological events) should be directly identified within the nervous system at the level of single nerve cells.

Single nerve cells signal to one another with a temporal sequence of action potentials. Formulation of the single neuron hypothesis was a great stimulus to neurophysiologists, who set out to measure the performance of single nerve cells on a number of perceptual tasks. They found that analysis of the stream of action potentials, initially by counting the number of action potentials within a window of fixed duration, revealed that these individual neurons could be as sensitive as the entire observer in detecting or discriminating small changes in a visual stimulus (see Figure 2A for a recent example).

Ultimately, this work progressed to the stage where single neurons could be recorded in awake, behaving animals as these animals simultaneously performed a psychophysical task for which they had been previously trained (Britten et al., 1992, Prince et al., 2000). Recording of single neurons simultaneously with the performance of a psychophysical task revealed another perceptually related component of neuronal activity. Some neurons increase or decrease their firing rate with the perceptual choice that the animal makes. This change is in addition to the change in firing that is induced by the external visual stimulus. Figure 2B shows an example of a visual neuron’s response, correlating presentation-by-presentation with the perceptual choice of a monkey, even though each time an ambiguous visual stimulus was shown. In the initial research on this phenomenon in the visual cortex, the animal was performing a discrimination task to distinguish the direction in which a set of moving dots travelled. The neurons typically had a strong activation for dots moving in one direction and a much weaker activation to dots moving in the opposite direction. Added to this visually driven activation, when the animal judged the set of dots to be moving in the preferred direction for the neuron, there was an enhancement in the activation of the neuron, regardless of actual direction in which the dots were moving (Celebrini and Newsome, 1994, Britten et al., 1996).

Fig. 2:

Example of linking single neuron activity to the reported perceptual appearance of a rotating visual object. A. A rhesus monkey makes decisions about the perceptual appearance of a structure-from-motion cylinder with two transparent surfaces with dots placed on them. Depending on the specific combination of motion and depth signals, the cylinder is perceived as either rotating clockwise (CW) or counter-clockwise (CCW) (shown on the left). Monkeys can discriminate between the different directions of rotations well (right top) and so can single brain cells in visual motion area V5/MT through their pattern of spiking activity (right middle). This neuron responds more strongly to counter-clockwise rotation (CCW). B. The spiking activity of individual V5/MT neurons to the presentation of the same stimulus varies from trial-to-trial. This variability in firing rate can predict the monkeys’ perceptual choice in a statistically reliable way, based on the preference of the neuron for different stimuli. In this case, the stimulus is a perceptually ambiguous structure-from-motion cylinder that has the same depth signals on both surfaces. The firing rates are color-coded according to the perceptual choice made by the animal. The neuron responds more strongly when the animal chooses the CCW direction, even though the stimulus does not change. The plot at the extreme right shows the frequency histograms and smoothed probability density estimates for the individual neuron’s responses shown to the left. The peak of the probability density estimate for CCW judgments lies above the peak for CW judgments. (Figure based on Dodd et al., 2001, Krug et al., 2004).

These choice-related activations have been seen in a variety of different areas of the visual cortex and for a variety of visual stimuli and tasks (Krug, 2004). One interpretation of the choice-related activity is that it might signify that this particular neuron is directly involved in forming the perceptual decision, just like the simplest version of the single neuron doctrine. What is striking about the phenomenon is the frequency with which it is observed in experimental studies. This relative ease of finding neurons with choice-related signals is made less puzzling by the finding that, within a restricted portion of sensory cortex, the activities of neurons are correlated with one another (Bair et al., 2001, Cohen and Newsome, 2009, Zohary et al., 1994). The choice-related signals also appear to be shared through the local network, for a recent review see (Parker, 2013). Here we use the term network to mean the set of functionally connected neurons in a spatially limited domain of sensory cortex, on the scale of a few millimeters in the macaque visual cortex. So, on this definition, this would comprise the set of neurons with similar functional specificity that are activated at the same time by external stimuli.

Experimental measurements show that the stronger the choice signal, the more sensitive the neuron tends to be in the task (Parker et al., 2002, Britten et al., 1996, Krug et al., 2016). At first sight, this seems to offer a simple explanation: the animal exploits the signal from the more sensitive neurons during the performance of the task, therefore these neurons show stronger choice-related activity. Recent theoretical modelling and experimental findings reject this simple explanation. Consideration of the correlations across the network is key to these recent advances (Haefner et al., 2013, Moreno-Bote et al., 2014).

Notably, it is not just the correlations in firing across the network that are important. What matters is how the distribution of correlated activity will influence the readout mechanism, which is generally conceived as the weighted sum of neural activities across the population. When that readout mechanism is a linearly weighted sum, there are direct predictions about the link between size of choice-related signal and the sensitivity of the neuron in the task, relative to the psychophysical performance of the animal. These predictions have been tested in two recent experimental studies. The outcomes are rather different, since in one case (Pitkow et al., 2015) the predicted relationship was observed, whereas in the other (Clery et al., 2017) the prediction based on linear weighting was conclusively rejected.

The general conclusion from this recent theoretical analysis and renewed wave of experimental studies is that the choice-related signalling arises through interactions in a network of task-sensitive visual neurons. However, not all members of the network contribute equally. Some primary members of the network must receive directly the choice-related neural signal, whereas others acquire this signal by connectivity with the primary members. At the current stage of development, there is no reliable means for distinguishing primary from secondary members of the network simply by recording their activity. Intervention within the network by stimulation of specific members of the network may offer a route into this question and we discuss this further below.

It is also the case that we have to assume that the functional connectivity within network of activated visual neurons is not completely static. The underlying connectivity will be continuously updated by visual experience both within the experimental setting and outside of it. The experimenter has limited control over the latter, but that experience may be critical in adjusting the functional connectivity within the network of visual neurons that are probed during the experimental measurements (Parker, 2013). Changes in connectivity driven by experience-dependent mechanisms that are active outside the experimental setting will lead to measurable correlations between neuronal activations. Many of these connections will have been established before the animal is introduced to the lab and trained for the experimental paradigm. So, there may well be correlations that can be recorded within the experimental setting but are difficult to interpret solely within the framework of the experimental paradigm.

There is a potentially interesting convergence between different streams of thinking about the neural signals that specifically govern perceptual events. In one line of work, it has been suggested that co-ordinated signalling in the form of oscillatory activity is associated with perceptual and cognitive events, such as the allocation of attentional resources to visual stimuli (Fries et al., 2001, Fries et al., 2008). In another, it is proposed that temporal synchrony between the firing of individual neurons (Singer and Gray, 1995, Kreiter and Singer, 1996, Womelsdorf et al., 2007) is a mechanism that could lead to perceptual binding, that is to say the recognition that two or more visible contours belong to one another in the Gestalt principle of common fate.

It is perfectly clear that the timing of spiking activity must affect the transmission of correlated activity from one neuron to another. This is because neurons are devices with specific temporal integration windows. As a result, spiking activity from other neurons that arrives within a tight temporal window has a much greater chance of exciting the target neuron than activity that arrives with more widely spread timing. Therefore, to explain better the choice-related signalling, greater attention needs to be given not just to the distribution of correlated activity across the network but also to the detailed temporal structure of those correlations.

This consideration is important when we turn to methods of intervention within the nervous system. We can present the issue in the form of a specific experimental test. Which mode of stimulation is required for effective modulation of perceptual decisions? Is it sufficient to arrange for the activation of certain sets of neurons within the network? Or must the applied stimulation contain a specific spatio-temporal pattern of activation events in order to be fully effective? In the next section, we evaluate the prospects for making such tests, given the present development of stimulation studies in the mammalian brain.

Electrical micro-stimulation inserts a naturalistic signal into the brain circuitry to change visual perception

A range of methods that have been employed to alter brain activity in order to change perception and behavior, starting with Fritsch and Hitzig’s experiments in 1870 identifying the motor representations of fore- and hind-limbs in the dog (Fritsch and Hitzig, 1870). These methods include electrical, pharmacological, magnetic and most recently ultrasound and opto-, chemo- and thermogenetic interventions. Looking at the range of experimental evidence, it is clear that focal electrical microstimulation, directly inserted into the brain, is still the most powerful and reliable method to change visual perception in a constructive and predictable way (Cicmil and Krug, 2015).

Focal electrical stimulation in early areas of visual cortex in awake humans induces a phosphene, a flash of light in a specific part of the visual field dependent on the location of stimulation on the cortical surface (Brindley and Lewin, 1968, Penfield, 1958). Experiments in monkeys confirm this (Bartlett and Doty, 1980), although it has been suggested that the ‘phosphene’ can be darker than the background and coloured (Schiller et al., 2011). Two striking aspects of the human microstimulation maps are, on the one hand, the apparent absence of evoked visual percepts through microstimulation in higher extrastriate visual areas (Murphey et al., 2009) and on the other hand, that focal electrical microstimulation did not reliably evoke more complex percepts than phosphenes in any parts of visual cortex. Experiments where monkeys were trained to detect electrical micro-stimulation with very small currents across a range of extrastriate visual areas demonstrate that this is not simply due to a lack of an evoked neuronal response (Murphey and Maunsell, 2007).

A paradigm shift was achieved through the combination of electrical microstimulation with contemporaneous visual stimulation in macaque monkeys. The seminal experiments by Salzman, Newsome and colleagues showed that a small artificial electrical signal directly inserted, focally into extrastriate visual area V5/MT could change the perceived motion direction in a set of random dots presented on a visual display (Salzman et al., 1990). For this intervention to work, the visual stimulus was closely matched to the receptive field properties, including size and motion preference, of the microstimulated neurons. This experiment exploited the columnar structure in area V5/MT, where neurons that are selective for the same motion direction are grouped together (DeAngelis and Newsome, 1999).

Since these experiments, similar microstimulation paradigms have been carried out in a number of visual cortical areas. Perceptual appearance of 3D depth was changed through stimulation in V5/MT, V4 and IT, face vs object discriminations were affected in area IT, and in area MST heading direction (see Cicmil and Krug, 2015 for review) (see Figure 1). More recently, it has been shown that focal electrical microstimulation can alter the perceptual appearance for more complex visual objects that depend on the specific conjunction of more than one visual parameter, in this case motion direction and depth. Electrical microstimulation in area V5/MT alters the perceived direction of rotation of a structure-from-motion cylinder robustly and in a direction predicted again from the multi-unit visual field properties at the microstimulation site (Krug et al., 2013) (Figure 3). In these experiments, the stronger the multi-unit tuning to relevant visual parameters (like binocular depth) at the stimulation site and the more closely visual stimulus and task were matched to the preferences at the stimulation site, the stronger was the micro-stimulation effect (Figure 3C).

Fig. 3:

Electrical micro-stimulation changes the appearance of a visual object dependent on the conjoint coding of 3D depth and motion (with permission from Krug et al., 2013; Cicmil et al., 2015). A. In these experiments, visual presentation of a rotating structure-from-motion stimulus was combined with focal electrical stimulation in visual area V5/MT. B. Example of the effect of electrical microstimulation at a single brain site. Electrical microstimulation at this brain site increases the likelihood of the monkey to choose counterclockwise rotation (y-axis). Across different strengths of visual signals (x-axis), the artificial electrical signal is fully integrated and treated as if the visual input has changed. C. If a stimulation site is more selective to the stimulus at the centre of the perceptual task (higher Disparity Tuning Index, DTI, x-axis), the larger is the effect of electrical microstimulation. Also, the more precisely the selectivity of electrically stimulated neurons and the visual task are defined and matched (for example in both direction of motion and depth signals), the bigger is potentially the effect of the artificial signal on perception. Here, this effect is described as the lateral shift of the red function relative to the black function fitted in B. D. Summary data of the effect of electrical microstimulation for one monkey. Artificial electrical signals, visually evoked signals and signals related to expected reward were integrated in visual cortex and affect behaviour. Here, we set these signals in opposition to each other to probe their effects, but by providing appropriate incentives, we should be able to boost the read-out of artificial signal by subjects.

This indicates on the one hand that the neuronal signals and selectivity we have characterized in these brain areas (see Figure 2) are directly relevant to perception (by themselves or through their downstream targets). On the other hand, the artificial electrical signals, which we introduce, are integrated with the visually-evoked neuronal signals as if we had added simply more visual information. The changes in perceptual report are specific to the properties of the electrically stimulated neurons. If microstimulation had activated neurons with less specificity, the visual stimulus might appear more noisy and the animal would have been less able to do the visual task under microstimulation. The naturalistic treatment of artificial electrical signals directly inserted into the neuronal circuitry is also supported by the interactions of sensory signals, electrical stimulation and signals related to expected reward we can see in extrastriate visual cortex (Cicmil et al., 2015) (Figure 3D). This suggests that we can, in principle, alter or replace visual perception of complex objects and scenes with artificial electrical stimulation – as long as we can unravel the underlying cortical circuits and signals.

Neuro-prosthetics: what does it take to replace vision?

Of course, the paradigms described in the previous section still required the combination of visual and artificial electrical input in order to shape more complex visual percepts. If we want to move towards a neuroprosthetic device that directly reads visual information into visual cortex, we need to be able to replace also the visual stimulus with an artificial signal (Figure 4). As laid out before, visual processing and perception likely requires the activation of many neurons at disparate sites and with a specific timing. This might seem initially a daunting task, but focal microstimulation experiments in primates have already shown that very specific percept modification can be achieved by stimulating just one cortical site – as long as the stimulation site is well characterized and directly task relevant. In the earlier cases (section 2), it is not sufficient simply to activate a portion of cortex. The experiments reveal a direct and specific relationship between the properties of the neurons in the stimulated location and the behavioural effect on the monkey’s choices.

Fig. 4:

Illustration of different intervention strategies to alter visual perception. A. Current paradigms combine electrical microstimulation and visual stimulation and thus effect changes in complex percepts. B. One option for vision replacement would be to increase the spatio-temporal resolution of electrical stimulation and, in this way, replace the need for concomitant visual stimulation. C. Another vision replacement strategy could be to utilize optogenetics to activate, for example, specific neuronal cell types.

Therefore, one strategy could be to analyse the neuronal spiking activity over a wider area of cortical tissue in space and time as a monkey views and makes a decision about the appearance of visual stimuli. Thus, we drive forward our spatio-temporal understanding of neuronal coding for visual processing and perception. Then, using multiple stimulation and recording sites over a number of electrodes, attempts could be made to replicate (‘read in’) the recorded electrical pattern through specifically timed and placed electrical currents (Figure 4B). The electrical microstimulation pattern would be adjusted until a similar pattern of neuronal activation is achieved as with visual stimulation. One important methodological question to explore are the interactions between currents delivered across multiple sites; effects can for example differ depending on spatial separation as well as location (Ghose and Maunsell, 2012). Another is the timing of stimulation currents, which can alter the efficacy of eliciting spiking activity in neurons (Doron and Brecht, 2015). As the different stimulation patterns are applied, a monkey would be instructed to carry out a discrimination task on the resultant percepts.

Until recently, progress in the use of prosthetics for sensory substitution has been driven by the goal of replicating the sensory surface at its input (retina or cochlea) (Jeschke and Moser, 2015, Stingl et al., 2013). Use of prosthetics in visual cortical areas has followed this lead, thinking in terms of replicating the visual array of retinal inputs with a spatially organized array of stimulating electrodes on the cortex. But, given our knowledge about the organization of cortex into functionally specialized compartments, such as columns and other elements of functional architecture, it is relevant to think about the use of specific stimulation at cortical sites that encode specialized aspects of visual processing, such as motion and depth in our studies. This is a difficult target to aim for, but recent experimental studies indicate it should become part of our thinking about the strategic use of electrical stimulation devices as prosthetics. For example, if such a device were to be targeted to stimulate many sites across area V5/MT, this might potentially have the facility to encode motion and depth information about the visual world to aid the bearer of such a device to navigate and move around. But in order to generate different static visual objects through such a neuroprosthetic device, we would require the capacity to activate different groups of neurons at different times, perhaps in another visual area, like V4. Another possibility might be to activate the same group of neurons with different activation patterns.

For neuroprosthetics, it is also likely to be critical to exploit the known capacity of primates to learn to detect artificial electrical microstimulation (Ni and Maunsell, 2010) and to exploit interactions between reward and artificially inserted electrical signals (Cicmil et al., 2015) for perceptual learning of the stimulation patterns inserted into cortex. Primates’ ability to direct visual attention and thus alter visual cortical processing (Treue and Maunsell, 1996, Xue et al., 2017) could also aid read-out and learning of electrical signals generated by a prosthetic. The initial signals must be ‘good enough’ to be related to aspects of the outside world. As with ‘natural’ vision, it would be the continuous exposure to and utilisation of these incoming patterns of activity for instructing behaviour that shape and train brain circuits and with it perceptual processing. This raises an important contrast with attempts to use electrical stimulation to replicate the spatial pattern of inputs at the primary sensory surface. If the individual receiving a neuroprosthetic device is expected to learn how to interpret the signals coming from the device by exploiting capacity for learning and plasticity still present in the adult brain, then this suggests that such devices should be implanted in later stages of the visual hierarchy, where there appears to be greater plasticity throughout adult life. Research on human echolocation in the blind suggests that relevant visual information, for example about distance and space, can in principle be processed even when it is very different in nature and arrives in the brain by a different route (Thaler et al., 2011).

At this stage, other methods, like opto- or chemogenetic approaches might contribute by enabling us to activate spatially-distributed neurons of specific cellular types or with specific connections (Figure 4C). However, they do not currently offer the spatial and temporal specificity of electrical stimulation in visual cortex of primates and therefore do not seem suitable for sensory substitution here (but see also work on developing new cochlear implants (Jeschke and Moser, 2015)). The direction for these technologies is more likely to be directed towards cases of neurodegenerative loss or incapacity of specific classes of neurons. One potentially important line to pursue is that optogenetic activations could be employed to mimic the effect of reward signals arriving within a specific volume of neuronal tissue (Stauffer et al., 2016). This could be used to open up learning mechanisms during a phase of learning about how to make effective use of a newly-implanted prosthetic device.

Conclusion

We can identify neural events that drive or change specific visual percepts in primates at the level of single neurons and circuits. Artificial electrical signals directly inserted into the primate brain can exploit this in order to alter the appearance of visual objects. One avenue for vision replacement at the level of the cortex is to scale up these methods to provide spatio-temporally defined, multi-site artificial electrical signals that alter network activity in a coordinated way.