Volume 10, Issue 1

In this article we introduce this special issue of Multimodal Communication. We briefly describe the founding of the Multimodal Research Centre and the journal Multimodal Communication before introducing each of the articles featured in this issue.

Digitally mediated service encounters such as digital payments are becoming more common and super-sticky social media such as WeChat, applications which mediate an indispensible part of our day-to-day practices, are increasingly used as the main tool for these services. However, how digital technology affects the way semiotic modes realize lower level actions in service encounters has not widely been studied. This paper carried out a multimodal analysis of videoed service encounters combined with interviews and ethnographic field notes to analyze how the use of WeChat Pay affects how modes such as proxemics, gaze and gesture realize lower level actions such as handing, smartphone handling, and ratifying payments. Second, analyses of the wider semiotic trajectories involved in the use of super-sticky social media in digitally mediated service encounters show how a mundane digital payment has important implications for how we record and disclose our personal behavior in big data and for the distribution of knowledge and skills deploying semiotic resources and digital literacies in society.

Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse . Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development . Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics . Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework . Routledge, London. https://doi.org/10.4324/9780203379493; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis . de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis . Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality . Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.

In this paper, I apply a metapragmatics-based approach to visual communication, combined with adapted concepts of Social Semiotics (“visual modality”) and CDA-oriented visual analysis (“canons of use”), to reconstruct two visual registers of authenticity which are prevailing within a social media photo sample of recent Austrian chancellor Sebastian Kurz (Facebook, Instagram; total of 84 photos), posted during the parliamentary elections in 2017. Triangulated with the discourse analysis of the marketing manager’s metapragmatic reflections on this social media campaign, the study shows how the partly intertwined registers of (1) “professional sensorism” (as a blended register comprising emblems of sensory modality and balanced composition, thereby drawing on the conceptualization of authenticity as sensory and affective experience of “now”) and (2) “voyeuristic fictionalization” (comprising indexicals associated with fiction genre, and based on the notion of authenticity as arising via “unnoticed observing”) are conceptualized and implemented as a—superior—visual stylization, acting as a social positioning, in mediatized political communication.

The paper proposes the integration of Audience Research in Multimodal Critical Discourse Studies to investigate media discourse. The novel approach is exemplified through a case study: the media interaction between one viewer and the text of his choice, BBC’s travel documentary Burma with Simon Reeve . The qualitative research design incorporates a pre- and after-viewing questionnaire, an interview and the multimodal textual analysis. The analytical focus is on the representation of social actors and it is argued that, through this integrated approach, it is possible to explore media effects on an audience by using the cognitive concepts of contextual effects , adapted from Relevance Theory, and ideological effects , a concept derived from the former and modelled around van Dijk’s definition of ideology.

Non-fluent aphasia is characterized by frequent word search and a much slower speech rate than non-aphasic speech. For patients with this type of aphasia, communication with those around them is therefore made difficult and is often severely impaired. One of the therapeutic proposals to improve the quality of life of these patients is to re-educate them with more multimodal alternatives. This of course assumes that gestures represent possible alternative means of communication for patients, and that their gestures are not affected in the same way as their speech. This article therefore proposes to study the gestures of 4 aphasic people and to compare them to the gestures performed by non-aphasic people, but also to establish correspondences between those gestures, intonation contours and the way people with aphasia develop their discourse. Results show that although gesture rate is not different in the two groups of participants, the gesture-to-speech ratio is higher for people with aphasia (PWA) than for non-aphasic people (NAP). Considering the fact that PWA also gesture more than NAP during silent pauses, which are longer but not more frequent than in NAP’s speech, and the fact that their gestures coincide less often with a lexical word, we believe that PWA use their gestures as compensation strategies for deficient speech. Yet, their speech impairment is also reflected in their gesturing: more gestures are prepared but abandoned before the stroke in this group and pre-stroke holds are longer, which means that PWA hold their gestures in the hope that they will better coincide with the word they are supposed to accompany and which takes more time to be uttered than in non-pathological speech. Their gestures are also less linked to each other than in the NAP group which goes hand in hand with the fact that they tend to utter independent syntactic phrases with no cohesive marker between them. This is also reflected in their less frequent use of flat and rising tones in intonation, which generally indicate that two sentence parts are dependent one upon the other, as well as their less frequent use of gestures showing discourse organization. In terms of gesture types, the PWA in this study perform many rhythmic beats and rely much on conventional gestures to compensate for their speech impairment rather than on their own creativity. Globally, this means that if multimodal therapies may benefit PWA to improve their communication with other people, speech therapists nevertheless need to be aware that life-long habits of gesture-speech alignment and synchronization may not be so easy to overcome for patients.