Multimodal (inter)action analysis in sociolinguistics: an essay analysing a digital video conversation illustrating emotion and heritage language maintenance

3.1 Multimodal discourse analysis

Multimodal Discourse Analysis is a method that extends traditional discourse analysis (Paltridge 2021) by incorporating various semiotic resources beyond language, such as images, gestures and actions, into the analysis of meaning-making processes (Qin and Wang 2021). This approach has gained prominence in different fields, including education, linguistics, and psychology, with researchers highlighting its advantages and emphasising its ability to offer unique insights and analytical parameters compared to other analytical methods (Qi and Hu 2022).

3.2 Multimodal (inter)action analysis

Norris’s (2004) framework provides a theoretical background and a set of analytical tools for MDA. This framework and set of tools, called Multimodal (Inter)action Analysis (MIA), highlight the various semiotic resources that mediate communication and interaction. The parentheses emphasize that the analysis not only examines interaction (the dynamic, real-time exchanges between people) but also considers action (the broader, contextually situated activities that people engage in) (Norris 2019). This nuanced naming underscores the importance of both immediate interactions and the larger actions within which these interactions are embedded. These interactions may involve semiotic resources such as speech, gestures, and tone to understand the social dimensions of language use. According to Norris (2016), “a main theoretical notion in this framework is the concept of mediation” (147). Norris adopted the concept of mediation into a specific unit of analysis, – “mediated action” (Norris 2016: 147), which refers to how human participants use various communication modes and cultural tools (media) to act and interact in a specific setting. Norris (2013) defines a mode as a “system of mediated action with regularities” (156) and depicts how (inter)actions can be produced through multiple modes being employed simultaneously, with some modes becoming more relevant than others in an (inter)action (Norris 2020). Recently, researchers across a wide variety of disciplines, including education, health, psychology, and literacy, have started applying Norris’s (2004) MIA. Multimodal (inter)action analysis is appropriate for breaking down elements like gaze, proximity, and arm movements. It has been used to study interactions, such as those between teachers and students (Wigham and Satar 2024) and in live classroom settings (Norris 2014). In a video ethnographic study of a Skype interaction between family members in New Zealand, Norris (2016) shows, using various analytical tools such as modal density, that as conversations proceed, the focus changes from the medium of interaction (Skype) to the conversation itself with the conversation becoming more natural.

In the MIA framework, all actions are mediated actions, and the three key methodological units of analysis are Lower-Level Mediated Action (LLMA), Higher Level Mediated Action (HLMA), and Frozen Mediated Action (Norris 2016). Norris (2020) illustrates every social (inter)action as potentially multimodal. These and the three other units described below allow us to dissect these intricate interactions across different levels of detail. The units of analysis or analytical tools are as follows:

1. Lower-Level Mediated Action (LLMA): The most basic unit of meaning within a specific communication mode, which is treated as a “social actor acting with or through mediational means or cultural tools” (Norris 2019: 237) represents this framework’s most granular building block. For instance, a single emoji or a brief text message would be categorised as LLMA.

2. Higher-Level Mediated Action (HLMA): Represents a broader concept constructed from a series of lower-level actions and can be labelled as “the multiple chains of lower-level mediated action that come together to produce the higher-level mediated action at the same time as they are produced by the higher-level mediated action” (Norris 2019: 238–239). For instance, an entire video conversation between a child and grandparent would be considered an HLMA, encompassing numerous LLMAs like gestures, spoken words, and changes in facial expressions.

3. Frozen-Mediated Action: Focuses on how past actions become embedded within the digital insider space, where communication is taking place, and can be featured as “lower or higher-level mediated actions, performed at an earlier time, may become frozen in objects or the environment” (Norris 2019: 241). This could involve a child customising his/her virtual background during a video call, where the background becomes a frozen representation of a prior action.

4. Modal Density: Analyses the intensity and complexity of lower-level actions co-occurring within a higher-level action (Norris 2019). For example, a child’s actions while excitedly talking to their grandmother on a video call might include handling the phone, smiling, and speaking – all happening concurrently. This scenario would exhibit a higher modal density than a simple text message exchange.

5. Modal Density Foreground-Background Continuum of Attention/Awareness: Examines how participants distribute their attention across different modes during a higher-level mediated action (Norris 2019). An example might involve a child simultaneously switching her attention to a different mode while communicating with her grandmother through a video call or a higher-level mediated action, such as changing the background of her computer screen. These analytical tools are summarised in Table 1.

Norris’s (2004) MIA is also flexible in that it allows the integration of multiple forms of data, including images, interviews, text, and video. This integration emphasises the interconnectedness of different semiotic resources, allowing researchers to analyse how these modalities interact to create meaning. Thus, for instance, integrating visual elements with spoken language, gestures and images can complement and enhance verbal communication (Norris 2002, 2011). The MIA framework is robust enough to encompass and analyse diverse data types, providing a comprehensive understanding of multimodal interactions.

It is important to realise that in addition to MIA, other approaches to multimodal analysis exist (LeVine and Scollon 2004; Van Leeuwen 2005). This essay, however, utilises Norris’s (2004) MIA framework to capture the complexities of digital communication. Unlike social semiotics, which prioritises social and cultural meaning (Karamullaoglu and Sandıkçı 2019), Norris’s (2004) framework offers an approach for exploring interactions across various modes on digital platforms like Facebook Messenger and WhatsApp. This scope is essential for understanding how digital communication modes and emotions arising during online interactions contribute to meaning-making (Norris 2016), which is a principal goal of this study. While social semiotics and Norris’s MIA have different focal points, they can be complementary. Social semiotics provides insights into the cultural and social contexts of communication, whereas Norris’s MIA offers a detailed analysis of the interactional dynamics across multiple modes, enriching our understanding of digital communication (Poria et al. 2017). This applies to our study, which aims to demonstrate the use of MIA in sociolinguistics to analyse video communication data among immigrants. The specifics of our study are described next.

3.3 Participant selection

The family investigated in this example belonged to a larger heritage language maintenance project examining digital communication practices among transnational Bengali-speaking immigrants in Australia. Following purposive sampling (Cohen et al. 2002), the family was recruited through a Facebook advertisement posted by the researcher (Inclusion criteria specified Bengali immigrant families residing in Australia with a focal child between the ages of five and eighteen who regularly communicated at least once a week) with transnational family members, preferably grandparents, using digital platforms like WhatsApp or Facebook Messenger.

The family profile (with pseudonyms) is as follows: The Ali family has resided in Australia for over ten years. Isha (aged eight at the time of the video conversation) engages in weekly video calls with her grandmother, Nasima (a 66-year-old housewife residing in Bangladesh). Isha’s mother, Fana, lives with her daughter and husband in Australia. Bengali served as the family’s primary or heritage language.

3.4 Collecting data

In this example, data were collected from two sources. The first was a participant-led video recording provided by the mother, Fana, of the focal child or the child in the family who is being studied interacting with her transnational grandmother, captured via the Facebook Messenger App. The specific video recording was chosen because of the richness of the multimodal data in the clip, its relevance to our research objectives, and its encapsulation of critical moments in the conversation, including the crying and kissing sequence described in the analysis section.

The second source consisted of researcher-led semi-structured interviews with the immigrant parents and grandparents. The mother and grandmother were interviewed separately over the phone at times convenient to them for around an hour. The interview data included questions about the participants’ beliefs or ideologies, practices, and heritage language maintenance strategies, which helped unpack their sociocultural, emotional, and linguistic practices. Information was obtained on the participants’ multimodal communication practices, which helped influence their heritage language maintenance. The interview data provided additional layers of insight that video data may not capture on their own, and thus enabled researchers to see things from the participants’ perspective, understand the bigger picture, allow for triangulation, and ensure the accuracy and depth of their analysis (Norris 2011; Norris and Maier 2014). Interviews were translated and transcribed by the first author.

Data were collected between March and August 2022, when the worst of the COVID-19 pandemic was over though some families were not travelling internationally because of various restrictions and fear of infection.

Regarding positionality, the first author is from a similar cultural and ethnic background to the participants, and provides an emic perspective (Cohen et al. 2002), enabling a deep understanding of the data and playing a supportive role by providing context and cultural understanding that can lead to a more accurate and meaningful translation of the data. Also, the relevant University ethics board approved this study (Approval Number: HC220021).

3.5 Coding data

Video conversation recordings were analysed using the European distributed corpora project Linguistic ANnotator (ELAN) software (Max Planck Institute for Psycholinguistics 2023). The first author became aware of the ELAN multimodal software from Norris’s (2019) book, researched the software, and then translated the video recordings into English and reviewed and transcribed video recordings using the ELAN software. This software allowed for annotating and timestamping interactions and embodied actions (e.g., smiles) (Sugahara et al. 2022). A detailed step-by-step guide to how the ELAN software was used in this study is provided in supplementary material A.1. Overall, ELAN provides a comprehensive suite of tools for managing and analysing multimodal data. Figure 1 below shows a screenshot of an ELAN annotation, illustrating how communication and embodied actions are timestamped and annotated.

Figure 1:

ELAN annotation screenshot.

Having discussed how the analysis can be realised, in the next section, we move on to an example of using MIA to explore the close bond between Isha (child) and Nasima (grandmother). All analyses are presented with screenshot data of the interactions. Before proceeding to the application of MIA, we will discuss what additional knowledge is gleaned from the interview data, including the context in which the video calls unfold.

4.1 Interview data: the importance of video calls in maintaining heritage language

The interview data provides context to the video call between Isha and Nasima. Video calls were typically made twice a week from Isha’s home in Australia and her grandmother Nasima’s home in Bangladesh, though occasionally Nasima might call from other locations in Bangladesh. The twice-a- week video calls lasted, on average, for 50 min, but if Isha had homework or other work, the conversation would be shorter. In the interview, Isha’s mother (Fana) thought that video calls through the Facebook Messenger app were essential for both Isha and Nasima (her grandmother) for emotional bonding and heritage language maintenance: “I don’t want my daughter to be very, very fluent in Bengali. I just want her to talk in Bengali to make her grandparents comfortable.”, – the statement underscoring the importance of bonding over heritage language practice in such conversations. Both Bengali and English are used during these conversations. This paper includes both the original heritage language (Bengali) and the English translations of the Bengali conversations. According to her mother (Fana), Isha is fluent in English and has basic proficiency in Bengali. In turn, Nasima is a native speaker of Bengali with some basic English proficiency.

During the interview, Nasima expressed her positive attitude towards helping her grandchild practice Bengali through the Facebook messenger app (See Excerpt 1 below):

The interview data thus tell us that there is a strong emotional need as well as the need for heritage language to communicate regularly over video chats. In the next section, we observe how Isha consoles her distraught grandmother during a video conversation over Facebook Messenger and analyse it using Norris’s MIA.

4.2 Video data: emotions and heritage language practice using multimodal (inter)action analysis

The entire example, which is a video conversation between a child and her grandmother, is an instance of what Norris (Norris 2004, 2019) calls an HLMA since, as we shall see, this HLMA comprises multiple chains of LLMA coming together.

The beginning of the video conversation shows Nasima crying and appraising her emotion, saying, “Āmāra cōkhē pāni āsachē [Tears are coming to my eyes] (Figure 2: Panel 1). The image annotated with an arrow pointing to Nasima wiping tears from one eye with her left hand shows her face and mouth are contorted from crying. When Isha asks her grandma the reason (Figure 2: Panel 2, also with enlargement) with a smile, Nasima says, Tōmarā Bangladesha āschōnā. Tomar bābā sick, Māmā sick, Ēbaṁ ṭumi’ō COVID-19 ēra kāraṇē sick, ēbaṁ āmi tōmākē dēkhatēō yatna nitē pārachi nā [You are not coming to Bangladesh. Your daddy is sick, mamma is sick, and you are also sick due to COVID-19, and I can’t see and take care of you guys]. Empathising with her grandmother’s sorrow, Isha tries to console her: “Didā, please cry karōnā. Āmarā asustha halē ki habē? Āmi jāni, ānēka sad āsē. We have to deal with it. Tabē āmarā sabasamaẏa bhiḍi’ō kala karatē pāri”. [Grandma, please don’t cry. What will we do if we are sick? I know, this is very sad. We have to deal with it. But we can always do video calls].

In this case, the HLMA aims to provide emotional support and maintain the connection between a grandchild and her distant grandmother. Nasima’s initial expression of sadness (tears, emotional statement) initiates the HLMA, prompting Isha’s empathic response (words, smile) within the video call environment. This example demonstrates how HLMA facilitates emotional exchange and coping strategies within digital grandparent-grandchild relationships, highlighting the unique affordances of video calls for practising heritage language and managing strong emotions despite physical distance. The LLMAs that comprise this HLMA are discussed next.

Isha then continues by enacting a fun activity over the video, “Didā, topsy-turvy karabō, yay!” [Grandma, I will do topsy-turvy, yay!] (Figure 2: Panel 3–6, – dotted line showing phone rotation), where Isha slides the mobile phone from a vertical to a horizontal position and then again back to an upright position. These activities seem to bear fruit, as grandma Nasima is visibly happy and smiling (Figure 2: Panel 7). The grandmother then reciprocates her happiness by kissing Isha over the phone (Figure 2: Panel 8–10). The circles and arrows (Panels 8–10) point to the kissing process. In Panel 8, Nasima’s mouth is open in an “O” in preparation for kissing; in Panel 9, her mouth is pressed flat against the screen as she starts to kiss, and in Panel 10, only her nose can be seen as she presses her mouth even more into the screen to complete the kiss. Nasima’s kissing her granddaughter over the phone and Isha’s reciprocity through her smile further strengthen their love and affection for each other. Although Nasima was busy interacting with Isha, which was the main focus of the video conversation, the task consisted of multiple LLMA of high modal density, with intense emotions, switching of attention between grandmother and the phone with coordination of actions (Isha rotating the mobile phone) (Norris 2019). This interaction showcases LLMA (Norris 2019), encompassing individual actions within the “topsy-turvy” activity. Isha’s LLMA includes announcing the activity, rotating the phone, and smiling. Nasima’s LLMA includes facial expressions, responding verbally, and performing the kissing gesture. Each LLMA contributes to the goal of shared joy.

Through multimodal cues, such as smiling, sliding the mobile phone from vertical to horizontal and back again, and verbal comfort, such as announcing the phone rotation activity, Isha attempts to achieve the shared goal of easing Nasima’s sorrow and reinforcing the importance of video communication. Furthermore, high modal density is evident within each LLMA and throughout the activity. Isha’s announcement combines speech and hand gestures. Nasima’s kiss involves facial expressions, body language, and interaction with the phone screen. This dense combination of modes or high modal density enhances emotional expression and understanding. Analysing LLMA and modal density reveals the intricate communication dance between Isha and Nasima, highlighting how they use diverse modes to achieve their shared goal of emotional connection and strengthen their bond despite physical distance. In line with Norris’s (2019) foreground background continuum, the “topsy-turvy” activity (Isha’s speech, phone manipulation) becomes the foreground as a playful attempt to cheer Nasima up. Here, Nasima’s facial expressions (previously background) might become foreground again as she reacts to Isha’s actions. The act of kissing the phone (Nasima’s facial expressions, body language) becomes foreground as a display of affection. Isha’s smile (previously background) becomes foreground again in response to the kiss. The kiss also serves as an instance of frozen mediated action since it is not real-time and represents a fleeting snapshot of a social act-an act of emotional expression. The use of Bengali as a heritage language by Isha to support her actions is backgrounded.

In the above activities, the grandmother expresses sadness, and the grandchild consoles her; these actions are placed in the foreground. The child’s playful activities, like rotating the phone and making the grandmother smile, are also foregrounded. All of these foregrounded activities emphasise emotional bonding. Heritage language practice, while present, is not the main focus, it is part of the background and supports the primary goal of bonding. The use of Bengali language occurs naturally within the context of their emotional exchanges, and it is secondary to the expressions of empathy and affection. The above interaction is characterised by mutual emotional support, shared activities, and the use of heritage language. Isha and Nasima thus co-construct their identities horizontally through their peer-like relationship, emphasising emotional connection and shared experiences over hierarchical family roles.

While there is no strict definition of empathy in the literature, there is general agreement that empathy comprises taking another person’s perspective, sharing emotions, relating-connection-bonding, and taking action to help others (Hall et al. 2021), all of which are identifiable in the conversation between Isha and Nasima. While in Bengali culture, the open expression of emotions is not particularly common and may involve various gendered cultural norms (Gross and John 2003), the special bond between grandparents and grandchildren is cherished and considered of great importance, being observed previously in the literature, for instance, among Bengali grandparents in East London, primarily through the use of “touch, gesture and gaze” (Jessel et al. 2011: 42). There are additional reasons for the display of emotions. First, the family’s transnational nature and the stress of the COVID pandemic and forced isolation caused the visible emotional outburst. The mother vouched for the bond between grandma and grandchild, mentioning during the interview that on the video, Isha used to say regularly in Bengali, “Didā āmi tōmāẏa anēka ‘love’ kari” [Grandma, I love you very much].

The affordances of multimodal digital media-based emotional streaming (King-O’Rian 2015) allow a high degree of empathy between the child and the grandmother during heritage language practice, which would have been difficult if the communication were to be realised over a modally poor medium. This conversation, especially with the emotional context, can contribute to Isha’s overall language development (Curdt-Christiansen and Huang 2020) (Figure 2).

Figure 2:

Emotions and heritage language practice. The dotted arrow in panels three to six shows the phone rotation.

6.1 Reflections on the use of multimodal (inter)action analysis

The first reflection from this study is that including interview data from the child’s mother and grandmother provided valuable context and helped understand the participants’ motivations and perspectives regarding the child’s heritage language practice during digital communication, highlighting the importance of different data sources in such analyses (Norris 2011; Norris and Maier 2014). Future MDA studies can thus utilise interviews of children to further support and contextualise their findings.

One critique of this study could be that Norris’s (2004, 2019) MIA framework was used to analyse specific aspects of digital interaction. Relying solely on one particular framework and analytical tools such as MIA may have its limitations. For instance, the MIA framework relies on dissecting multimodal elements of the data. However, broader social and contextual elements such as cultural norms or family dynamics do not guide the analyses. Alternatively, the framework does not account for the power dynamics between the people engaged in digital discourse, which may require different considerations (Van Leeuwen 2012). Also, deciding how to segment interactions into HLMA and LLMA may be subjective and require considerable effort, time, and clarity of judgment on the part of the analyst. Some analysts may find setting up tiers in the ELAN software time-consuming, though this comes with the benefit of providing analytical depth. Therefore, alternative MDA frameworks and other considerations may be contemplated depending on the research question, and these might also expose different insights.

Various aspects of our study provide directions to strengthen future research. Thus, pilot testing the chosen approach on a small sample of data before applying it to the entire dataset may reveal the nuances of the specific communication context. Also, as discussed above, multimodal analyses combined with analyses of data such as interviews, surveys, and observations are essential to develop a more comprehensive understanding of the communication process. This broadened scope could help solidify the findings and provide a more generalisable understanding. As alluded to earlier in the text, MIA can be applied and is not limited to the sociolinguistics discipline. In addition to the examples from the education discipline discussed earlier (Wigham and Satar 2024), the approach may offer researchers unique opportunities in media and advertising, where multiple media are often utilised to convey various messages (Zhao et al. 2020). For example, the richness of advertised messages could be analysed using the concept of modal density, and specific elements of advertisements, such as slogans and facial expressions, could be dissected and understood using the LLMA tool. Once again, aspects of modal density and their psychological effects or the effect of specific LLMA elements could be analysed. In addition, future research could explore how these findings apply to different family structures, languages, and technological environments. Additionally, investigating the long-term impact of these digital interactions on heritage language maintenance and emotional well-being would be a valuable area for future study.

This essay showcases the emotional aspects of family bonding and heritage language practice within a transnational immigrant family in Australia using digital communication through Norris’s (2019) MIA framework and tools, and creates new opportunities, as seen here for researchers in socio-linguistics and applied linguistics. By employing MIA to examine the complex interplay between emotions and heritage language practice, we demonstrate that family bonding, rather than heritage language maintenance, is the key motivator of these digital interactions. This highlights the significance of MDA in uncovering the emotional landscape when practising heritage language within seemingly simple family video calls. Our study offers practical insights for families and educators, such as the need for regular, emotionally engaging interactions between children and their transnational family members. For example, grandparents and grandchildren could play online games together, share stories, or engage in creative projects. These activities can foster emotional connections and make heritage language practice more enjoyable. The findings have broader applicability beyond the immediate context of Bengali immigrant families in Australia, in other disciplines and contexts to understand digital communication and emotional bonding, providing a valuable framework for future research.