Volume 28, Issue 3 - Special Issue: Viewpoint Phenomena in Multimodal Communication, Guest Editors: Barbara Dancygier and Lieven Vandelanotte

In this introduction to the special issue on viewpoint phenomena in multimodal communication, we highlight central questions concerning the nature of multimodality and of conceptual viewpoint, which the issue as a whole expands and clarifies. We argue that multimodality needs to be rethought as a varied but cohesive phenomenon, and we briefly illustrate both embodied multimodal interaction (in an example from stand-up comedy) and the meaning emergence in artifacts relying on both text and image (in an example of a poster with an environmental message). Correspondingly, the category of multimodal constructions already recognized for embodied interactions should be expanded to cover conventionalized image/text combinations. Finally, we stress that viewpoint is the pivotal concept that elucidates how communicators use the various modalities for cohesive communicative purposes across the wide range of artifacts and multimodal forms discussed in the special issue, including gesture in political speeches, viewpoint in comics, grammatical forms in Internet memes, ASL, stance expressions, eye gaze, and embodied responses to objects and architectural artifacts.

Viewpoint has been shown to be a powerful construal mechanism in multimodal spoken and signed discourse, as well as in various other modalities and genres. This paper investigates embodied viewpoint strategies that have been observed when speakers combine speech, gestures, postures, gaze, and simulated action to describe their interaction with spatial artifacts such as gallery buildings, virtual architectural models, and paintings. Simulated artifact immersion is introduced as a multimodal viewpoint strategy whereby speakers submerge into their mental representation of an artifact by perceiving and experiencing it from an internal vantage point. It is argued that this viewpoint strategy tends to be employed when there is no narrative structure for the speakers to fall back on. The paper’s aim is twofold: (a) to show that when speakers talk about their own experiences with spatial artifacts, distinguishing between immersed and non-immersed experiential viewpoint strategies may be more fitting than distinguishing between character and observer viewpoint; and (b) to discuss how considering the interaction of iconic, indexical, and metonymic principles in gesture may elucidate viewpoint phenomena in general.

Political speeches are a prime example of how discourse often requires speakers to convey multiple competing viewpoints, both their own and others’. Cognitive linguists have shown how, in speech, speakers express viewpoint through individual choices at the lexical and grammatical level. Recently, cognitive linguists have also shown that speakers express viewpoint using speech-accompanying gestures. To date, the study of viewpoint expression has focused on cases where speakers deliver the same viewpoint across modalities. By examining the persuasive uses of gesture in Obama’s A More Perfect Union speech, I show how speakers can communicate multiple different viewpoints across gesture and speech, simultaneously. There are moments when Obama expresses his opponents’ viewpoint in speech, while framing them in terms of his own viewpoint in gesture, and vice versa. I discuss how the deviation of viewpoints across modalities provides key insights into multimodal cognition, with respect to working memory, metaphor, and persuasion. Specifically, I argue that, as an implicit medium, gesture allows speakers to inject viewpoint into the uptake of speech, below the conscious radar of recipients, and I discuss how this rhetorical capacity is evolving as a result of communication technologies.

In this paper, we present an embodiment perspective on viewpoint by exploring the role of eye gaze in face-to-face conversation, in relation to and interaction with other expressive modalities. More specifically, we look into gaze patterns, as well as gaze synchronization with speech, as instruments in the negotiation of participant roles in interaction. In order to obtain fine-grained information on the different modalities under scrutiny, we used the InSight Interaction Corpus (Brône, Geert & Bert Oben. 2015. Insight Interaction: A multimodal and multifocal dialogue corpus. Language Resources and Evaluation 49, 195–214.). This multimodal video corpus consists of two- and three-party interactions (in Dutch), with head-mounted scene cameras and eye-trackers tracking all participants’ visual behavior, providing a unique ‘speaker-internal’ perspective on the conversation. The analysis of interactional sequences from the corpus (dyads and triads) reveals specific patterns of gaze distribution related to the temporal organization of viewpoint in dialogue. Different dialogue acts typically display specific gaze events at crucial points in time, as, e.g., in the case of brief gaze aversion associated with turn-holding, and shared gaze between interlocutors at the critical point of turn-taking. In addition, the data show a strong correlation and temporal synchronization between eye gaze and speech in the realization of specific dialogue acts, as shown by means of a series of cross-recurrence analyses for specific turn-holding mechanisms (e.g., verbal fillers co-occurring with brief moments of gaze aversion).

This contribution focuses on verbal amplifiers and comical hypotheticals in a corpus of face-to-face interactions. Both phenomena qualify as markers of a mental viewpoint expressing an (inter)subjective construal of a certain experience. Whereas amplifiers offer a straightforward view onto a speaker’s evaluative stance, comical hypotheticals provide an intersubjective account of a viewpoint construal. As part of their meaning, their use reveals a speaker’s assumption about the interlocutor willing to allow or participate in a particular type of interactional humor. Our research interest for these phenomena concerns their occurrence as well as their interactional alignment in terms of mimicry behavior. In order to capture the impact of both linguistic and psychological variables in the use of these items, we adopt a differentiated methodological approach, which allows to correlate findings from our corpus linguistic analysis with the values obtained for interpersonal difference variables. As our data consists of male dyads of which the participants never met before the beginning of their conversation, we expected to witness an increase, along with the growing familiarity among the interlocutors, in both the use and alignment of these viewpoint phenomena. Indeed, results show a clear increase in the use of both verbal amplifiers and comical hypotheticals over the course of the interaction and independently from the also observed overall increase of communicativeness. However, with respect to the alignment of both viewpoint phenomena, our study reveals a differentiated result. Participants aligned their use of verbal amplifiers with that of their partners over the course of the interaction, but they did not do so for comical hypotheticals. Yet, within the broader discussion of the experiment’s design, this unexpected result may still seem plausible with respect to our general hypothesis. Beyond the limits of this study, the set-up and results of our study nicely connect to recent research on empathy-related behavior in social neuroscience.

Composite utterances are utterances that are built from multiple signs of multiple types, meaning that in any conversational “move” speech, gestures, eye-gaze, intonation patterns, physical stance, etc. all participate in the utterance, and the meaning derived from it is constructed by the composite of these participant types. likewise considers utterances as multimodal ensembles. The present study investigates how the notion of composite utterance plays out in a signed language such as ASL. Articulated in the same modality as are gestures, the distinction between language and gesture has seemed less clear, leading some to ask whether signers even gesture at all and some to suggest that gestures and formal signed language are substantively different systems. On the other hand, others have posited a continuity approach to gesture and signed language especially in light of grammaticalization studies. Here I examine topic-comment constructions and perspectivized clauses in ASL through the lens of Enfield’s composite utterances proposal, looking at component parts and how they function to ground elements in the discourse and guide the interlocutor through the textual structure. I use Enfield’s conventional versus non-conventional type categories in examining lexical and prosodic elements in topic and perspective-taking constructions.

I analyze multimodal viewpoint construction in comics to engage with how modalities function within the medium as a specific discourse context with distinct conventions and material qualities. I show how comics employ established storytelling practices with character, narrator, and narrative viewpoint levels, while building up and interweaving these through strategic uses of the modalities of the medium. I mobilize the cognitive theories of embodiment, domains, mental simulation, and mental space blending as an analytical framework. I examine the asynchronicity of viewpoint elements between modalities and their synthesis into composite character viewpoints in several examples. I show how modalities can be prioritized and their different qualities and functions strategically manipulated for viewpoint construal. These brief examples show the complexity inherent in multimodal communication and interpretation and the usefulness of encouraging the medium-specific and interdisciplinary analyses of cultural works from a cognitive linguistic perspective.

This paper considers a range of so-called image macro Internet memes and describes them as emerging multimodal constructions relying as much on image as on text, and apportioning roles to images much like constructional slots, for instance to fill in a subject role in a subjectless clause, or even to provide the main clause content to a textually given when -clause. In addition to existing or partially altered linguistic constructions, many examples also rely on specific top text/bottom text division of labor, and crucially depend on frame metonymy, with limited formal means quickly cueing richly detailed frames (for instance by using iconic images). The popularity of memes, forming series and cycles of iterations and remixes, and their role in establishing and maintaining discourse communities seems to be driven by a need to express and reconstrue viewpoints, often starting from ideas, affects or stereotypes assumed to be intersubjectively shared with viewers, whose responses they solicit. This paper argues that a proper description of Internet memes of the type considered requires a construction grammar approach, complemented by an understanding of viewpoint dynamics in terms of a Discourse Viewpoint Space regulating the network of spaces and viewpoints.