Volume 8, Issue 2 -

In this paper, we present an in depth study of the interplay between various semiotic modes in the specific instructional setting of a scientific café. We analyzed the multimodal performance of five female instructors delivering a monologue explanation during instruction on the following dimensions: speech, gesture (hand gestures, head orientation and gaze) and use of written didactical material. Results first point out the crucial role played by referential hand gesture together with gaze-body behavior both in representing new concepts ( conceptual mediation ) and in building bridges between information displayed in several modes ( semiotic mediation ). They also show cross-individual differences in instructors’ multimodal performance, that we propose to interpret as three diverse modes of mediating knowledge, guiding being the only one providing both conceptual and semiotic mediation.

In recent years, increasing attention has been devoted to investigating what aspects of grammar in spoken language are coordinated with gesture. However, previous studies have focused on gesture use in relation to other grammatical categories rather than the basic grammatical categories of nouns and verbs. Against this background, this study, taking the tool noun and tool verb alternation as a starting point, considers to what extent gesture relates to the tool frame and to what extent it relates to the choice of nouns or verbs and accordingly the means of construal afforded by nouns and verbs. It is found that the number of gestures produced and the dominant gesture mode are largely dependent on the tool frame, but some other aspects in gesture do vary depending on the choice of nouns or verbs in speech, and these differences in gesture seem to correlate with the difference between how speakers conceptualize nouns and verbs, as hypothesized in the theory of cognitive grammar: nouns profile THINGS while verbs profile PROCESSES. This finding adds to previous studies about which aspects of grammar relate to gesture and how they do so, thereby furthering our understanding of the integrated nature of speech and gesture.

This EEG study explores similarities and differences between verbal and visual metaphor comprehension processes. The results of the study show that interpreting either a verbal stimulus or a visual stimulus metaphorically activates similar brain areas, but there are significant differences between the two in terms of neural connectivity and resource allocation patterns. A greater intra-hemispheric connectivity was observed during verbal metaphor comprehension as compared to greater inter-hemispheric connectivity during visual metaphor comprehension. Results of our novel method, called µ index, suggested a distinct information processing mechanism for metaphor in verbal and visual modalities. We found that metaphorical interpretations, especially highly creative interpretations, of visual stimuli required allocation of greater resources when compared to highly creative interpretations of verbal stimuli. Overall, these results provide a deeper insight into the multimodal nature of metaphor processing.