Towards developing multimodal literacies in the ESP classroom: methodological insights and practical applications

In educational settings, the importance of developing multimodal literacies to help learners construct meanings from texts that integrate multiple semiotic resources is now widely recognized (Jewitt and Kress 2003; Royce 2007). This trend is clearly linked to rapid and ongoing advances in digital technologies that have had a profound impact on how we interact with others not only in daily life, but also in educational institutions where it has “revolutionized traditional academic practices” (O’Halloran et al. 2016, p. 256). Indeed, the use of widely available new media resources on online platforms (e.g., websites, blogs/forums, TED Talks, digitally distributed filmic genres) encourages learners to make meaning from a range of semiotic modes (e.g., verbal, visual, aural, spatial, and gestural) that are highly characteristic of such resources. Moreover, the effective harnessing of multimodal and multimedia resources in the classroom is now seen as a crucial strategy to successfully engage with increasingly digitally savvy learners (Street et al. 2011).

In language teaching, a multimodal approach enhances awareness of semiotic modes beyond verbal language, which can then be leveraged to both comprehend and produce discourse in the target language more effectively on a variety of levels: linguistic, discursive, pragmatic, and cultural. With particular reference to English for Specific Purposes (ESP), learners face additional challenges driven by language and communicative practices associated with specific disciplinary and professional domains. Among these are not only specialized lexico-grammatical features, registers, and genres, but also domain-specific situated and embodied practices that entail multisemiotic processes or “chains of acts, representations and people across spaces and over time” (Prior 2013, p. 520). An area of critical interest is the role of non-verbal features (e.g., facial expressions, gesturing, body orientation) in facilitating the comprehension of digital forms of communication associated with specialized discourse communities by means of replication, reinforcement, and integration of a verbal message (Dahl and Ludvigsen 2014; Querol-Julián and Arteaga-Martínez 2019; Sueyoshi and Hardison 2005). Indeed, to help learners cope with such input, practitioners need to foster “multimodal listening skills” (Campoy-Cubillo and Querol-Julián 2015, p. 199), which entail not only listening but also watching in order to glean information from semiotic modes other than verbal language. Equally important is the development of multimodal competence in producing specialized communications, for example, by acquiring skills of multimodal composition (Selfe 2007), in which learners produce written texts that integrate words, images, and sounds that draw on the affordances of digital technology. Learners also need to be made aware of the contribution of non-verbal cues during their own speech production, reflecting the well-established practice in professional contexts of training people to use such resources effectively (Goman 2008). In response to these challenges, linguists and practitioners working in ESP settings are called upon to devise methods and materials that take into account how multiple semiotic resources contribute to meanings, and then implement them to enhance language learning and foster the multimodal competence associated with the discourse processes of a given specialized domain. The selection, preparation, and application of methods and materials to be used to achieve these objectives can thus benefit from underpinning research that highlights their multimodal/multimedia dimension from various theoretical and analytical perspectives, including multimodal social semiotics (Kress and van Leeuwen 2001); multimodal discourse analysis (Baldry and Thibault 2006), and multimodal interaction analysis (Norris 2004).

This Special Issue originated from an international conference held in Pisa in November 2019 entitled “Knowledge Dissemination and Multimodal Literacy: Research Perspectives on ESP in a Digital Age”. The conference was organized under the auspices of the CLAVIER (Corpus and Language Variation in English Research) Interuniversity Research Center and was hosted by the Pisa research unit whose activities partially coincided with a nationally funded research project^[1] focusing on the interface of knowledge dissemination, multimodal literacy, and ESP (Crawford Camiciottoli 2019). The conference brought together researchers with a shared interest in investigating innovative approaches for analyzing and applying multimodal texts in the context of ESP in higher education. Thus, the aims of the Special Issue are to (1) advance the current state of knowledge about how multimodal and multimedia resources can be leveraged to improve learning in ESP settings, (2) provide a platform for research that highlights innovative approaches to ESP practice, with a strong focus on application and/or experimentation in the classroom, and (3) offer insights for bridging the traditional gap between smaller-scale and larger-scale multimodal research methods (Bateman 2014) without, however, losing sight of the value of each perspective. Towards these aims, the five contributions selected for this Special Issue reflect a wide range of methodological approaches to multimodal research and practice—from case studies to corpus-based applications—across specialized discourse domains. The research and practical applications discussed in the Special Issue also showcase different types of multimodal input, including professional written reports; intercultural communication in the workplace; job interviews; tourism documentaries, guided tours and docu-tours; government debates; and classroom interactions.

The first two contributions of the Special Issue highlight workplace communications. In the first study, Hartle, Facchinetti and Franceschi, and propose a practical way to teach multimodal communication strategies in a specialized English for Professional Purposes setting. Based on Littlewood’s (2014) learning continuum, which ranges from analytical study to experiential practice, they demonstrate the effectiveness of a teaching framework that enhances students’ awareness in the use of multimodal resources and intercultural skills. The authors present two case studies in the course English for the World of Work at the University of Verona. In the first one, their aim is to develop multimodal competence in report writing by means of the analysis and deconstruction of genre specific writing, as well as the visual elements accompanying it. In this activity, learners first reconstruct the writing jointly and then do so independently in a task-based approach. In the second case study, students are asked to build upon competences already acquired in previous modules, namely, how the different elements come into play and interact in intercultural communication. After focusing on individual aspects in two semi-guided activities, they are encouraged to employ all the different aspects and modes of intercultural communication in the final activity. This activity consists of an intercultural communication simulation in which non-verbal cues signaling social and power dynamics have a prominent role, and students need to accommodate to a different way of communicating in order to sign a business deal that is satisfactory to both parties.

In the second contribution, Fortanet-Gómez and Beltrán-Palanques focus on the multimodal features of the job interview as a highly relevant genre for Business English learners attending bachelor’s degree courses in business and economics. They present a research-informed pedagogical proposal based on a video recording of a simulated but realistic job interview, with the aim of developing learners’ multimodal communicative competence to help them better navigate this key professional genre. The underlying research supporting the pedagogical proposal draws on multimodal interactional analysis (Norris 2004) to determine how different semiotic resources (i.e., verbal language, intonation, pitch, pauses, hand gestures, gaze, and facial expressions) mediate the interaction during the interview. To formulate the pedagogical proposal, the authors implement Lim’s (2018) framework for developing multiliteracies in the classroom (situated practice, overt instruction, critical framing, and transformed practice) and also describe instruments and procedures to evaluate its effectiveness. In this way, Fortanet-Gómez and Beltrán-Palanques offer a useful strategy for enhancing learners’ awareness of how multimodal affordances can be leveraged to more effectively both conduct and participate in a job interview.

The third contribution by Bonsignori and Cappelli addresses the specialized discourse domain of tourism, representing a popular area of study within ESP instruction. Building on previous research on a multimodal corpus consisting of video recordings of authentic guided tours, documentaries and docu-tours that revealed the prominent role of non-verbal strategies (particularly images and gesturing), the authors apply these findings to task-based project work for learners in an English for tourism course. The project involved learners creating two sets of video clips before and after instruction on effective communicative strategies for docu-tours, which enabled the compilation of a multimodal learner corpus. This corpus was then analyzed with the support of multimodal annotation software, and comparisons were made between the pre and post instruction clips, and between the learner-produced clips and the previously analyzed authentic materials. The results indicated an overall improvement in mastering the features of docu-tours, including the use of multimodal resources, and contributed to the professional development of the learners by providing them with key skills for their future career choices.

The fourth contribution by Incelli is also a teaching application based on materials from a multimodal corpus that were analyzed with multimodal annotation software within the context of an ESP course for students in an undergraduate economics degree program at an Italian university. The course has an interdisciplinary slant that encompasses political economics and thus also addresses issues of economic policy in government institutions. Taking as a point of departure the multimodal analysis of two recordings of parliament debates, one in the United Kingdom and the other in the United States, the students are engaged in activities that enhance their multimodal literacy competence as well as their critical thinking skills by means of the interpretation of the verbal and nonverbal features of the parliamentary debates. The objective is to make students aware of how the use of several semiotic modes constructs meaning, while reinforcing identity and ideological stance, representing a key aspect of the type of specialized discourse to which these learners need to be exposed.

In the final contribution to the Special Issue, Bateman reflects on multimodal studies and advocates for larger-scale empirical research in order to bridge the gap between multimodal and corpus studies. He revisits the definition of ‘semiotic mode’ and its relationship with ‘materiality’ and ‘discourse’ to propose new ways to apply the teaching and learning of multimodal discourse in the classroom, by means of larger corpus research and through the use of technology. A methodology is proposed in which transcriptions and annotations are carefully described. According to the author, the analysis of videos featuring face-to-face interactions must be considered as illustrations of those interactions, rather than models to follow, and the circumstances of the recordings must always accompany the analyses. He further suggests that single event analysis may not reflect reality and should therefore be characterized as subjective. However, thanks to ongoing advances in technology, analyses of larger multimodal corpora will be increasingly feasible and this will allow researchers to better generalize findings related to the use of multiple semiotic modes in discourse. The deep reflection in Bateman’s article can pave the way for future research on how to use the results of multimodal analysis in the language classroom.

The innovative collection of papers presented in this Special Issue serves to shed new light on how ESP practice can benefit from a multimodal approach, but also to underscore the important role of multimodal research in the field of ESP which, as noted by Belcher (2006) and Prior (2013), has thus far been largely missing since ESP research has traditionally focused primarily on written texts.