Computational challenges in multisensory perception

In everyday life our senses are constantly bombarded with many different signals: the motor noise of the trucks, a sparkling motor-bike passing by at high speed, the smell of smoke and fumes and the sight of other pedestrians. How does the human brain transform this sensory cacophony into a veridical percept of the world? To misperceive the looming truck as talking and shiny, and your companion as roaring and smelly could be disastrous! This illustrates that multisensory integration and segregation is critical for our daily interactions. Information integration increases the salience of sensory signals thereby allowing us to detect and respond faster and more accurately to important events, such as an approaching truck (Diederich & Colonius, 2004; Frassinetti et al., 2002; Gillmeister & Eimer, 2007; Noesselt et al., 2008). Further, combining complementary (for example object shape by viewing it from the front and touching it from the back) or redundant (for instance object location by vision and audition) information across the senses enables a more robust and reliable percept (Ernst & Bülthoff, 2004).

The Bayesian framework in neuroscience posits that the brain forms a probabilistic generative model of the sensory inputs that is inverted during perceptual inference (Kersten et al., 2004; Kersten & Yuille, 2003; Knill & Pouget, 2004). Bayesian probability theory offers a precise formulation of how observers should combine uncertain information to form a representation of the world. Critically, multisensory perception relies on solving two fundamental computational challenges. First, the brain needs solve the so-called ‘causal inference problem’ and infer whether or not signals come from a common source and should be integrated (Shams & Beierholm, 2010). Second, if signals come from a common source, the brain should integrate them into the most reliable or precise (i. e. least variable or noisy) percept of the environment by weighting them according to their relative reliabilities (Alais & Burr, 2004; Ernst & Banks, 2002). Bayesian Causal Inference models account for these two challenges by explicitly modelling the causal structure of the world (Körding et al., 2007; see also Deroy et al., 2016; Rohe & Noppeney, 2015a, 2015b; Shams & Beierholm, 2010; Wozny et al., 2010).

Let us focus on one simple example: Imagine you are an enthusiastic ornithologist prowling through the forest at dawn in order to gather the best photos and sound recordings of birds. Suddenly, you spot a little robin sitting on a branch and you hear a little robin singing in the bush. How should you direct your camera and your microphone? Should you integrate information from vision and audition in order to obtain a more reliable estimate of the bird’s location? Or should you use the information only from vision for directing your camera and only from audition when directing your microphone? The answer to this question depends on the hidden causal structure of the world. There are two hypotheses or models that the brain should entertain.

First, there may be one bird sitting on the branch that is the same bird that you hear singing in the bush. In this ‘common source’ case, you should indeed integrate signals from vision and audition weighted by their sensory reliabilities. This is the classical ‘forced or mandatory fusion’ model that has dominated the field of multisensory integration and cue combination over the past two decades (Alais & Burr, 2004; Ernst & Banks, 2002; Hillis et al., 2004). As described by maximum likelihood estimation (MLE) an observer obtains the most precise estimate in this common source case if s/he integrates signals weighted by their relative reliabilities, which is the inverse of variance or noise in the signal. For instance, you would assign a smaller weight to a weak unreliable visual signal at dawn than to a strong and clear visual signal during daylight. Critically, multisensory integration according to MLE principles should lead to a variance reduction of the multisensory relative to the least variable unisensory percept, which is greatest (i. e. by a factor of 2) when the variances of the two unisensory signals are equal. Indeed, several psychophysics studies have shown that human observers integrate signals that are likely to come from a common source near-optimally, close to the predictions of maximum likelihood estimation (Alais & Burr, 2004; Bresciani et al., 2006; Ernst & Banks, 2002; Hillis et al., 2004; Jacobs, 1999; Knill & Saunders, 2003). Yet, evidence is not unequivocal. Accumulating research has also highlighted situations where human observers overweight the sensory modality (Battaglia et al., 2003; Burr et al., 2009; Butler et al., 2010; Rosas et al., 2005) that is usually more reliable in everyday life for a particular task and property (Battaglia et al., 2003) or show a smaller multisensory variance reduction than predicted by MLE (Battaglia et al., 2011; Bentvelzen et al., 2009).

Yet there is a second hypothesis about the signal’s causal structure: there may be two birds, one that you can see sitting on the branch and one that you can hear singing in the bush. In this ‘independent source’ or ‘full segregation’ case, information integration would be detrimental. Instead, you should use only the auditory information for directing your microphone and the visual information for directing your camera.

Critically, the individual sensory signals do not directly inform the brain whether they arise from common or independent events. Instead, we must actively infer the ‘hidden’ causal structure from a range of multisensory correspondences such as signals happening at the same time (‘temporal coincidence or correlations’: Lee & Noppeney, 2011a; Lewis & Noppeney, 2010; Magnotti et al., 2013; Maier et al., 2011; Munhall et al., 1996; Noesselt et al., 2007; Parise & Ernst, 2016; Parise et al., 2012; van Wassenhove et al., 2007), same space (‘spatial colocation’: Lewald & Guski, 2003; Slutsky & Recanzone, 2001; Spence, 2013), semantic (Adam & Noppeney, 2010; Bishop & Miller, 2011; Kanaya & Yokosawa, 2011; Lee & Noppeney, 2011b; Noppeney et al., 2010), metaphoric (Sadaghiani et al., 2009; Parise & Spence, 2009) and other higher-order statistical or learnt congruency cues. Yet some uncertainty about the world’s causal structure will remain. To account for this causal uncertainty, the brain computes a final spatial estimate by combining the estimates from the two causal structures using one of various decision functions (for details see: Wozny et al., 2010). For instance, using the computational strategy called model averaging, it should estimate the location for directing the microphone by combining spatial estimates that are computed by the ‘forced fusion’ and the ‘full segregation’ models, weighted by the posterior probabilities that audio-visual signals were more likely caused by one single or two different birds (Körding et al., 2007).

Accumulating evidence suggests that human observers arbitrate between sensory integration and segregation qualitatively in line with the principles of Bayesian causal inference (Beierholm et al., 2009; Bertelson & Radeau, 1981; Landy et al., 1995; Roach et al., 2006; Shams & Beierholm, 2010; Wallace et al., 2004). In the laboratory this has been shown in particular for spatial localization (Körding et al., 2007; Rohe & Noppeney, 2015a, 2015b; Wozny et al., 2010) and speech recognition tasks (Magnotti & Beauchamp, 2017; Magnotti et al., 2013). In spatial localization experiments, observers are presented concurrently with auditory signals (for example, bursts of white noise) and visual signals (for instance flashes) at the same or different locations with variable audio-visual spatial disparities. On each trial observers report the location of the flash and/or the location of the noise burst. The results show that an observer’s perceived sound location is shifted towards a spatially displaced but synchronous visual flash and vice versa depending on the relative auditory and visual reliabilities. In line with Bayesian Causal Inference these audio-visual spatial biases are attenuated or even abolished for large audio-visual spatial disparities when it is unlikely that auditory and visual signals come from a common source. In other words, audio-visual spatial disparity is a critical cue that observers use to determine whether or not to integrate sensory signals (Körding et al., 2007; Rohe & Noppeney, 2015a, 2015b; Wozny et al., 2010).

Figure 1:

A. Bayesian Causal Inference model: The generative model of Bayesian Causal Inference for spatial localization determines whether the ‘sight of the bird’ and the ‘singing’ are generated by common (C=1) or independent (C=2) sources (Körding et al., 2007). For common source, the ‘true’ audio-visual location (SAV) is drawn from one prior spatial distribution. For independent sources, the ‘true’ auditory (SA) and ‘true’ visual (SV) locations are drawn independently from this prior spatial distribution. We then introduce independent sensory noise to generate auditory (XA) and visual (XV) inputs.

B. Visual bias on perceived sound location as a function of audio-visual spatial disparity. As predicted by Bayesian Causal Inference the audio-visual spatial bias depends non-linearly on spatial location. For small spatial disparities, the observer integrates auditory and visual spatial estimates weighted approximately in proportion to their relative reliabilities. For large spatial disparities audio-visual interactions and biases are reduced (Rohe & Noppeney, 2015b).

C. Bayesian Causal Inference within the cortical hierarchy: Primary sensory areas represent predominantly the location of their preferred sensory signals (for example sound location in auditory regions). Posterior intraparietal cortex integrates sensory signals weighted by their reliabilities approximately according to forced fusion principles. Anterior intraparietal sulcus computes the final Bayesian Causal Inference estimate that takes into account the observer’s uncertainty about the causal structures that could have generated the sensory signals (Rohe & Noppeney, 2015a).

The audio-visual spatial bias that emerges for small spatial disparities is in fact the so-called ventriloquist effect (Bertelson & Radeau, 1981; Bonath et al., 2007; Driver, 1996), a perceptual illusion that was used for religious purposes already in ancient times and later for entertainment at travelling fun fairs (Vox, 1981). To create the ventriloquist illusion the puppeteer speaks without making any articulatory movements. Further, he holds the puppet close to his own face and moves the lips of the puppet in synchrony with his own speech. As a result of the temporal correlations of the auditory (i. e. puppeteer’s speech) and visual (i. e. puppet’s facial movements) signals the observer infers that auditory and visual signals are generated by a common source and integrates them into a coherent percept weighted by the relative auditory and visual reliabilities. As the spatial reliability of sound perception is usually inferior to the precise visual spatial estimates, the observer most commonly misallocates the speech of the puppeteer to the puppet (Alais & Burr, 2004). In short, the ventriloquist illusion tricks the brain based on the computational principles of Bayesian Causal Inference: it artificially brings auditory and visual signals into spatial conflict while maintaining temporal synchrony to enable integration (see excursion box 1 for perceptual illusions in multisensory perception).

Multisensory interactions are ubiquitous in neocortex

Traditionally, it was thought that multisensory integration is deferred until later processing stages in higher order association areas such as parietal or prefrontal cortices (Avillac et al., 2007; Barraclough et al., 2005; Beauchamp et al., 2004; Calvert et al., 2000; Driver & Noesselt, 2008; Ghazanfar et al., 2008; Macaluso et al., 2003; Miller & D’Esposito, 2005; Sadaghiani et al., 2009; Schroeder & Foxe, 2002; Stevenson & James, 2009). However, over the past two decades neuroimaging in humans (Foxe et al., 2002; Lee & Noppeney, 2011a, 2014; Lehmann et al., 2006; Martuzzi et al., 2007; Molholm et al., 2002; Noesselt et al., 2007; Werner & Noppeney, 2010a), neurophysiology in non-human primates or rodents (Atilgan et al., 2018; Bieler et al., 2017; Bizley et al., 2006; Bizley & King, 2009; Foxe & Schroeder, 2005; Ghazanfar et al., 2005; Ibrahim et al., 2016; Iurilli et al., 2012; Lakatos et al., 2007; Schroeder & Foxe, 2002; Kayser & Logothetis, 2007); and neuroanatomical (Falchier et al., 2002; Rockland & Ojima, 2003; Schroeder et al., 2003) research have accumulated evidence suggesting that multisensory integration emerges already in early sensory and even primary sensory areas and then progressively increases across the cortical hierarchy. Provocatively, it was even proposed that ‘the entire neocortex is multisensory’ (Ghazanfar & Schroeder, 2006).

In support of low-level integration, numerous fMRI and EEG studies in humans have shown that multisensory interactions can be observed in primary sensory areas and at early processing stages even before 100 ms post-stimulus (Besle et al., 2008; Foxe et al., 2000; Molholm et al., 2002; Molholm et al., 2004). Likewise, neurophysiological recordings in non-human primates (Kayser et al., 2008, 2010; Lakatos et al., 2009; Schroeder & Foxe, 2005) or rodents (Atilgan et al., 2018; Bieler et al., 2017; Bizley & King, 2009; Bizley et al., 2007) revealed that the response to the preferred stimulus in sensory areas, for instance auditory belt and parabelt areas, can be enhanced or suppressed or gain in information content by a concurrent stimulus in a non-preferred sensory modality. While multisensory interactions in low-level sensory areas may be due to top-down influences from superior temporal or parietal cortices (Seltzer & Pandya, 1994), they may also be mediated via thalamo-cortical mechanisms (for example pulvinar) or direct connectivity between sensory areas (Musacchia et al., 2014; Schroeder et al., 2003). Indeed, neuroanatomical tracer studies have shown sparse direct connectivity from early or even primary auditory to visual cortices and vice versa in primates (Falchier et al., 2002; Rockland & Ojima, 2003) and rodents (Bizley et al., 2007; Budinger et al., 2006; Campi et al., 2009; Ibrahim et al., 2016).

This ubiquity of multisensory interplay at all stages of cortical processing challenges traditional hierarchical models of late integration. It suggests that multisensory interactions emerge at multiple cortical levels and within several circuitries including thalamo-cortical, cortico-cortical and higher order association cortices (Musacchia & Schroeder, 2009; Schroeder et al., 2003). Hence, we need to move beyond identifying multisensory regions towards characterizing their functional properties and behavioural relevance.

In primary and low-level sensory cortices, previous research has described driving and modulatory multisensory influences (Atilgan et al., 2018; Bieler et al., 2017; Bizley & King, 2009; BIzley et al., 2007; Kayser et al., 2008; Lakatos et al., 2009; Meijer et al., 2017; Meredith & Allman, 2015). First, both neuroimaging studies in humans (Leitão et al., 2012; Werner & Noppeney, 2011) and electrophysiology in rodents (Ibrahim et al., 2016; Iurilli et al., 2012) have suggested that unisensory stimuli induce deactivations or synaptic inhibition in non-corresponding sensory cortices. For instance, visual stimuli have been shown to elicit a negative BOLD-response in auditory cortices, while auditory stimuli induce synaptic inhibition and fMRI deactivations in visual cortices (Ibrahim et al., 2016; Iurilli et al., 2012; Leitão et al., 2012). Second, a stimulus of a non-preferred sensory modality may not necessarily elicit a reliable response in itself, but instead modulate the response to a stimulus of the preferred sensory modality. For instance, auditory core and belt areas are predominantly responsive to auditory rather than visual signals, yet their auditory response and information can be modulated by a concurrent visual input (Kayser et al., 2010). Lakatos and colleagues suggested that these modulatory interactions may rely on mechanisms of phase resetting of theta oscillations (Lakatos et al., 2009; see also Sieben et al., 2012 for related research in rodents). Because in our natural environment the visual signal often precedes the auditory signal (for instance facial articulatory movements often precede speech output), it can modulate the sound-induced activity by resetting the phase of ongoing oscillations (Lakatos et al., 2009; Schroeder et al., 2008). This may be an important mechanism whereby multisensory integration can increase the salience of multisensory events and facilitate their detection. In support of a temporally-sensitive mechanism, recent neurophysiological (Kayser et al., 2010) and fMRI studies (Lewis & Noppeney, 2010; Werner & Noppeney, 2011) have also shown that audio-visual interactions in Heschl’s gyrus and planum temporale were sensitive to temporal coincidence or correlations over time. Critically, multisensory response enhancements at the primary cortical level were then gated into higher order association cortices (for instance ventral object vs. dorsal motion recognition system) depending on task context, suggesting that low-level integration effects can propagate to influence higher-order processing to guide behavioural responses (Lewis & Noppeney, 2010).

Figure 2:

Late and multistage integration models: Traditionally it was thought that multisensory integration emerges at later processing stages in association cortices. We propose that different types of multisensory interactions occur at multiple stage of the cortical processing hierarchy.

Multisensory interactions in higher-order association areas such as superior temporal or parietal sulcus are usually less sensitive to the exact timing of the sensory inputs and are characterized by larger temporal binding windows (Werner & Noppeney, 2011). Rather than salience detection, these areas may thus be involved in integrating signals into task-relevant representations (for example spatial, object, speech etc.). In support of this conjecture, the profile of multisensory interactions in STS and IPS directly predicted whether human observers benefitted from multisensory integration: the greater their multisensory enhancement in superior temporal and parietal cortices, the greater was observers’ audio-visual benefit for object categorization (Werner & Noppeney, 2010a, 2010b).

Finally, even if sensory signals cannot be integrated into a unified percept because they are incongruent, they can still interact at a decisional level and influence response selection. Using selective intersensory attention tasks a myriad of studies have demonstrated that a task-irrelevant yet incongruent visual stimulus can interfere with observers’ decisions on a task-relevant auditory stimulus (Noppeney et al., 2008) and vice versa (Krugliak & Noppeney, 2015; Marks, 1987). Combining a Compatibility Bias model and fMRI we have previously suggested that the prefrontal cortex accumulates sensory evidence from multiple senses until a decisional threshold is reached and a response elicited (Noppeney et al., 2010). Further, in situations of congruent sensory signals the prefrontal cortex shows suppressed responses to audio-visual relative to unisensory signals in line with response facilitation at the decisional level (Sugihara et al., 2006; Werner & Noppeney, 2010a). Interestingly, in line with research in rodents showing multisensory interactions predominantly in transition zones between sensory cortices (Wallace et al., 2004), the suppressive interactions were predominantly found in border zone between auditory- and visual dominant regions (Werner & Noppeney, 2010a).

In summary, accumulating evidence suggests that multisensory integration is a multifaceted process emerging at multiple stages across the cortical hierarchy. While some sensory interactions take place early and even in primary sensory areas, it is likely that other information may also propagate to higher cortical levels prior to being integrated. Potentially, multisensory interactions in low-level sensory areas serve to amplify the signal strength and salience of multisensory events, which in turn impacts representational integration processes and decision-making in higher levels of the cortical hierarchy (Werner & Noppeney, 2010a).

How the brain performs causal inference and reliability weighted integration

At the beginning we discussed that the brain faces two critical challenges in a multisensory world. First, it needs to identify and bind signals that come from a common source based on a range of correspondence cues such as temporal coincidence or spatial colocation. Second, if signals come from a common source they should be integrated, weighted in proportion to their relative reliabilities. While section 1 summarized the behavioural evidence, in the following we will review neurophysiology and neuroimaging research that provides insight into the underlying neural basis.

Since the seminal work by Stein and colleagues on multisensory integration in the superior colliculus (Meredith & Stein, 1983; Wallace et al., 1996; Stein & Meredith, 1993) a vast number of neurophysiological and neuroimaging studies have shown that the multisensory interactions depend on spatial colocation, temporal synchrony and correlations, as expected for causal inference (Stein & Stanford, 2008). More specifically, Stein and others demonstrated that audio-visual interactions were superadditive (i. e. neural response for the audio-visual stimulus was greater than the sum of the unisensory responses) for spatially collocated audio-visual signals, but turned additive, subadditive or even suppressive when auditory and visual signals were presented at different locations and one signal fell outside the receptive field for the other signal (Stanford, 2005; Stanford & Stein, 2007; Wallace et al., 1996). Thus, organization into receptive fields may enable causal inference based on spatial correspondence cues.

Similar to the role of spatial concordance, audio-visual interactions of transient signals in superior colliculus were limited to a temporal window of approximately 500 ms. Recent modelling approaches suggest that temporal binding windows for naturalistic continuous signals such as speech may rely on detecting multisensory correlations based on the Hassenstein-Reichardt-detector as a relatively simple, yet physiologically plausible model component (Parise & Ernst, 2016). One interesting question that has recently been asked is whether temporal binding may be related to the brain’s internal rhythms, i. e. neural oscillations. First, cross-modal phase resetting was put forward as a temporal mechanism that would allow a signal from one sensory modality to modulate the processing of another sensory signal as a function of oscillatory cycle length and audio-visual asynchrony (Lakatos et al., 2009). Second, more recent studies even suggested that the time-varying cycle length of the alpha oscillations in individual observers may determine their audio-visual binding window. Human observers with faster oscillations were associated with a smaller temporal binding window (Cecere et al., 2015; Samaha & Postle, 2015). While the idea that oscillation cycles may serve a similar function for temporal binding windows as receptive fields do for spatial binding windows is intriguing, future studies and more detailed and specific theoretical models are needed.

Recent neurophysiological studies in non-human primates focused on how single neurons and neuronal populations integrate signals weighted by their reliabilities. In a visuo-vestibular heading discrimination task Fetsch and colleagues demonstrated that macaques showed similar near-optimal performance to human observers (Fetsch et al., 2012). Concurrent recording from neurons in dorsal motion area MSTd showed that congruent neurons combined the visual and vestibular inputs subadditively (Gu et al., 2008) and weighted by their relative reliabilities (Fetsch et al., 2012), giving a higher weight to the more reliable sensory signal (Fetsch et al., 2012) on a trial-by-trial basis. As predicted by maximum likelihood estimation under forced fusion assumptions, neurons were more sensitive to heading direction under visuo-vestibular than unisensory stimulation (Gu et al., 2008) in line with behavioural performance (see also Nikbakht et al., 2018 for a related study in rodents). Further, neural population decoding revealed neural sensory weights that corresponded closely to the sensory weights computed from the monkeys’ behavioural performance (Fetsch et al., 2012). Additional electrical microstimulation and chemical inactivation of MSTd provided a causal link between the neural computations in MSTd and behavioural performance in a heading discrimination task (Gu et al., 2012). Collectively, this elegant and extensive body of work suggests that neuronal populations and single neurons in MSTd integrate visual and vestibular signals weighted by their relative reliabilities in representations of heading direction that guide behavioural decisions. While these computations are in line with maximum likelihood estimation, they can be obtained through mechanisms of divisive normalization (Ohshiro et al., 2011, 2017), a canonical neural computation that has previously been proposed for visual processing and attentional modulation (Carandini & Heeger, 2012). Moreover, divisive normalization can also explain a response enhancement that is maximal when the strength of individual signals is weak – a principle referred to as inverse effectiveness since the seminal studies by Stein and colleagues (Stein & Meredith, 1993). Consistently across species and methodologies research has shown that the operational (i. e. super- vs. subadditive) modes of multisensory integration depend on the signal strength as well as a neuron’s or voxel’s response to unisensory stimuli (Kayser et al., 2008; Siemann et al., 2015; Stanford et al., 2005; Stein & Meredith, 1993; Stein et al., 2014; Werner & Noppeney, 2010b).

At the neural systems level functional imaging studies in humans have shown that higher order association cortices such as parietal or superior temporal sulci integrate sensory signals weighted by their reliabilities in speech recognition (Nath & Beauchamp, 2011), spatial localization (Rohe & Noppeney, 2018) and shape discrimination tasks (Beauchamp et al., 2010; Helbig et al., 2012). Two recent studies moved beyond reliability-weighted integration and investigated how the human brain performs Bayesian Causal Inference in a spatial ventriloquist paradigm at the neural systems level using functional imaging (Rohe & Noppeney, 2015a, 2016). Inside the scanner, observers were presented with audio-visual signals that varied in the spatial disparity and visual reliability. On each trial, they either located the sound or the visual stimulus. Combining psychophysics, fMRI, Bayesian modelling and multivariate decoding the study showed that the brain accomplishes Bayesian Causal Inference by encoding multiple spatial estimates across the cortical hierarchy. At the bottom of the hierarchy, auditory areas encoded predominantly the location of the sound and visual areas the location of the visual stimulus (= segregation). In posterior intra-parietal sulcus, location is estimated under the assumption that the two signals are from a common source (= forced fusion). Only at the top of the hierarchy, in anterior intraparietal sulcus, is the uncertainty about the world’s causal structure taken into account. As predicted by Bayesian Causal Inference, location is estimated by combining the segregation and the forced fusion estimates weighted by the posterior probabilities of common and independent sources. Thus, anterior IPS forms a spatial estimate that gracefully transitions from integration to segregation as a function of spatial disparity (Rohe & Noppeney, 2015a, 2016).

Conclusions

In our natural environment our senses are constantly bombarded with many different signals. Ideally, the brain should integrate signals weighted by their reliabilities when they come from a common source, but process them independently when they come from different sources. Human observers have been shown to arbitrate between integration and segregation in line with Bayesian Causal Inference. At the neural level neurophysiological studies in non-human primates and other species have unravelled how the brain integrates signals from common events weighted by their relative reliabilities into a unified representation. Initial neuroimaging studies in human observers suggest that the brain integrates sensory signals in line with Bayesian Causal Inference by encoding multiple perceptual estimates along the cortical hierarchy. Future research combining psychophysics, computational modelling, neurophysiology and neuroimaging across different species is needed to bridge the gap between neural mechanisms, computational operations and behaviour and explore the functional consequences of multisensory integration.

Excursion: Multisensory binding as a mechanism for perceptual illusions

The computations of our perceptual system are optimised for effective interactions with our natural environment. In the laboratory, we can play tricks on observers’ perception by placing them in situations that violate the natural statistics for which their perceptual system has been optimized. Particularly, in multisensory integration we can bring sensory signals artificially into conflict along one particular dimension (for example space, time, number, phoneme), while providing sufficient multisensory correspondence cues along another dimension. Thereby, we can persuade the brain to integrate conflicting signals into one unified illusory percept. Multisensory integration has been used to create a myriad of perceptual illusions. In the following we will highlight the most prominent examples.

In the double-flash illusion (Shams et al., 2000) observers are presented with a single flash of light temporally sandwiched between two beeps. In such cases observers will usually report seeing two flashes, indicating that their final percept placed more weight on the temporally-precise sound signal than on the relatively unreliable flash. Thus, while the ventriloquist illusion exploits the spatial uncertainty of hearing (we can in most circumstances locate an object better by vision than audition), the double-flash illusion exploits the temporal uncertainty of vision (the ears are more reliable than the eyes at determining when something happened). In that respect the double-flash illusion may be considered a temporal equivalent of the ventriloquist effect. But, of course, temporal and spatial dimensions are not quite comparable. While spatial ventriloquism illustrates how the brain estimates the spatial location of an event (i. e. estimation task), the double-flash illusion reveals how it determines the number of events (i. e. detection task).

Multisensory speech signals can also be manipulated to produce illusory percepts. In the so-called McGurk-McDonald illusion (Mcgurk & Macdonald, 1976) the observer is presented in synchrony with a video clip of a speaker articulating /ga/ and a sound recording of the phoneme /ba/. Because of the synchrony cues the observer integrates these conflicting audiovisual phonemes into an illusory /da/ percept. This illusory percept can again be explained by reliability-weighted integration. Using a speech synthesizer, one can generate an artificial ‘phoneme’ dimension and progressively morph from a /ba/ to a /ga/ phoneme. The perception of ‘ba’ – ‘da’ – ‘ga’ phoneme categories emerges as a result of human categorical perception (Liberman et al., 1957).

The rubber-hand illusion (Botvinick & Cohen, 1998) is an example of our own bodily perception being tricked. In order to track our body’s position in space, humans rely on proprioception – the sense that allows us to, for instance, clap with our eyes closed. The rubber-hand illusion overrides this sense by utilising visual and proprioceptive cues that conflict with our understanding of our body’s position. The participant is seated with one hand placed on a table. This hand is then concealed from them using a divider, and a replacement rubber hand placed in clear view. The person performing the illusion then proceeds to simultaneously stroke both real and rubber hands with paint brushes, taking care to match the strokes as closely as possible. As the participant continues to see the strokes on the rubber hand but feel them on their real hand, the brain is presented with increasing evidence that these signals are perfectly temporally matched and should be integrated, despite conflicting information from the proprioceptive system that their hand is actually far to the right. The result, in the majority of participants, is a growing belief that the rubber hand has replaced their real one. Such demonstrations often conclude with the experimenter unexpectedly hitting the rubber hand with a hammer.

Figure 3:

The rubber hand illusion. The participant is seated at a table, with their right hand obscured by a divider, and looks at a rubber hand. The experimenter strokes the participant’s hand and the rubber hand simultaneously with paintbrushes, using varied but matching strokes that suggest these haptic and visual signals have the same source and should be integrated. The resulting illusory percept usually manifests as a sense that the rubber hand is a part of one’s own body.

Finally, even our sense of taste is not exempt from multisensory illusions. Professor Charles Spence specialises in the sensory perception of food, and his lab has demonstrated a variety of ways in which other senses can influence what we taste. The weight and material of cutlery (Harrar & Spence, 2013), the colour and shape of the plate (Piqueras-Fiszman et al., 2012), and the texture of packaging (Piqueras-Fiszman et al., 2012) have all been shown to influence our experience of food. In 2008, Professor Spence (alongside his colleague Massimiliano Zampini) was presented with the Ig Nobel prize for the novel demonstration that digital sound manipulations can make potato crisps seem crunchier (Zampini & Spence, 2004).

Acknowledgment: This research was funded by ERC-2012-StG_20111109 multsens.