Reliability in the identification of metaphors in (filmic) multimodal communication

1 Introduction

One of the major challenges in the analysis of multimodal communication is the development of procedures and protocols that can be applied by different scholars in the same way. This is needed to ensure that the differences observed across data sets can be interpreted as genuine differences, rather than as noise due to the fact that different analysts applied the same procedure in different ways.

Generally speaking, when human judgment is involved in the analysis and classification of any type of data, unwanted variability is usually very high (Kahneman et al. 2021). Such variability can be attributed to different sources of error. In front of the same judging task, different analysts can behave in different ways on different occasions, for distinct reasons: some judges are inherently more severe than others, some judges may be influenced by external factors that are irrelevant to other judges, and so forth. The analysis of multimodal communication is particularly difficult, because of the variety of information encoded, inferred, and entangled within multimodal messages. To limit such unwanted variability, it is important that the tools provided to the analysts to perform their judgment tasks are well designed. In this way, it is more likely that independent analysts may approach the task in comparable ways. When this is the case, the method used for the analysis is considered to be reliable and it stands a higher chance to deliver the same results when the study is replicated by different analysts. Reliability, as defined by Krippendorff (2011: 94), “is the extent to which different methods, research results, or people arrive at the same interpretations or facts”.

This paper investigates the reliability of a method developed and used to analyze multimodal data, and in particular to identify whether a message expressed by filmic means contains metaphorical elements. This is a non-trivial task because it adds to the complexity given by multimodality, the identification of metaphorical elements, where metaphors themselves are difficult to define and identify. The Filmic Metaphor Identification Procedure (FILMIP; Bort-Mir 2019) is a procedure developed for the identification of filmic elements that are used metaphorically, in multimodal (filmic) communication. We hereby show how this procedure delivers reliable and thus arguably replicable analyses of multimodal (filmic) communications. We present two empirical studies in which FILMIP was applied by independent analysts for the identification of metaphors in filmic advertisements. We also discuss critical aspects and potential limitations of the method under scrutiny, as well as the broader implications for multimodal communication.

2 Theoretical background

The scientific literature on content analysis of multimodal communication shows great variability in the approaches adopted by different analysts, in different disciplines. The tasks are varied and not easily comparable. Content analyses of multimodal communication may have different goals and address different research questions. They may involve the classification of users’ configurations of emoji into communicative strategies used to express different types of concepts (Danesi 2016; Das et al. 2019; Wicke and Bolognesi 2019). They may involve classifications of sign language descriptions of spatial relations between objects shown in pictures (Ortega et al. 2017), or they may involve classifications of gestures to investigate their different possible communicative functions (Cienki 2016; Kopp et al. 2008; Mittelberg 2019).

Even though there are some attempts to automate the process of metaphor detection and interpretation in written language (Fass 1991; Mason 2004; Su et al. 2017; Wilks 1978, among many other studies), the identification of metaphors in multimodal communication is considered complex phenomena that cannot, for the moment, be identified automatically. Thus, content analyses performed by human judges are commonly used in fields such as cognitive linguistics, to identify and manually classify metaphorical constructions in texts, visuals, or audiovisuals. The goals of these studies may be diverse, ranging from the classification of different types of pictures into metaphorical and non-metaphorical ones (e.g., Bolognesi et al. 2017; Stampoulidis and Bolognesi 2019), or the classification of different types of metaphors and other figurative constructions in multimodal advertisements (Kjeldsen and Hess 2021; Pérez-Sobrino 2016). Conceptual Metaphor Theory (Lakoff and Johnson 1980) spread the research on metaphor in areas other than just linguistics, as is the case of cognitive neuroscience (Johnson 2010; Van Dijk et al. 2008) or psycholinguistics (Gibbs and Colston 2012), among many others. The main idea of this theory on the study of this particular trope is that metaphor is not just a linguistic device employed to embellish poetry or prose, but something that people use in their everyday life to make sense of abstract concepts that are difficult to explain or to talk about. The so famous ARGUMENT IS WAR conceptual metaphor is used through linguistic expressions such as “Your claims are indefensible”, “I demolished his arguments” or “He shot down all of my arguments” (Lakoff and Johnson 1980:4). Thus, the way we structure the action that we perform when arguing is by means of one literal concept (war, the source domain, as it is the concept from which we use attributes that we map onto the other concept) being somehow compared to a more abstract concept (argument, the target domain, as it is the concept that we aim to talk about).

For the identification of metaphors in moving images, FILMIP has been recently proposed and it will be hereby evaluated. The procedure takes inspiration from related procedures, developed in the past decades, for the reliable identification of metaphors expressed in different semiotic systems. The pioneering procedure for metaphor identification, known as MIP (Metaphor Identification Procedure) was developed by the Pragglejaz Group (2007) and it focuses on linguistic metaphoric expressions used in written texts. MIP is intended to lead analysts to the decision of whether a lexical unit is used metaphorically in a given context. It takes the researcher into the identification of the basic and also the contextual meaning of a lexical unit, both established under the definitions found in, at least, two dictionaries. The basic and contextual meanings must then be contrasted and compared, thus leading the analyst to the final decision of whether that lexical unit is metaphorically used. For instance, given a sentence like Do you see what I mean?, the word see would be considered an MRW (metaphor related word), as the contextual meaning from the sentence is found in the fourth entry in the dictionary as “to understand something” (“See”, Cambridge English Dictionary, n.d.), and out of the Essential Meaning section in the Merriam-Webster as “to perceive the meaning or importance of” (“See”, Merriam-Webster, n.d.). The basic meaning of the word is shown in the former dictionary as “to notice people and things with your eyes”, and in the latter as “to notice or become aware of (someone or something) by using your eyes”. Thus, as both basic and contextual meanings differ from each other, the MRW see could be marked for metaphoricity.

Relating certain linguistic forms to their underlying conceptual structures and determining which set of correspondences are involved in the correlation between two different cognitive domains (or cross-domain mappings) is one of the main issues tackled by Steen (2001, 2002, 2009 resulting in a new, refined version of MIP, called MIPVU (MIP plus the initials of the Vrije Universiteit, Steen et al. 2010). The difference between the two procedures is that whereas the MIP aims at identifying metaphorically used words in spoken and written discourse, MIPVU focuses on the identification of ‘metaphor-related words’, including here “all lexical units in the discourse that can be related to cross-domain mappings in conceptual structure” (Dorst 2011: 102). Thus, the MIPVU bases its metaphor identification on the well-established idea that if metaphor is a matter of thought, then the cross-domain mapping between domains can be understood as all phenomena that imply a connection of similarity between two distinct domains (Grady et al. 1999; Steen 2008; Zbikowski 1997).

Other procedures have been developed more recently for the identification of deliberate metaphors in language (DMIP, Reijnierse et al. 2018) and for the identification of other types of figurative language, notably hyperbole (HIP, Burgers et al. 2016) and verbal irony (VIP, Burgers et al. 2011). As for semiotic systems other than verbal language, a procedure for the identification of metaphor in still images has been developed and applied to various visual genres (VISMIP, Šorm and Steen 2013). This procedure focuses on the identification of visual units with a metaphorical use in different persuasive materials, such as political cartoons or advertisements. VISMIP differs from MIPVU in language in that the latter looks for differences between the basic and contextual meanings of lexical units within dictionaries; as there are no such dictionaries for images, VISMIP uses Wordnet Online, a big database of English words created by Princeton University where researchers can find nouns, adjectives, adverbs, and verbs classified and grouped according to their meanings. For instance, assuming that there is an ice-cream cone in a picture, but instead of the ice-cream scoop it is the Earth that is melting, we would search for Earth or planet in Wordnet and would compare the results with the search for the words ice-cream scoop. As the results would be different, it would be assumed that there is a comparison between both cognitive domains (the “planet Earth” domain versus the “melting ice-cream” domain), and the picture would be marked for metaphoricity.

FILMIP evolved from some basic theoretical assumptions that characterize the identification procedures described above. In particular, the seven steps of the procedure are analogous to MIPVU and VISMIP’s steps: the three entail the identification of the basic and contextual meaning of the units under analysis, and the comparison between cognitive domains is based upon incongruity, for instance. However, due to the complexity of filmic pieces in comparison to language and still images, FILMIP diverges in several stages of the procedure from the other methods in order to adapt it to the intricacy of multimodal materials, as is the case of its first step, with both macro and micro analysis of the materials, where the videos should be decomposed to the smallest level of granularity in order to envisage some kind of abstract meaning and the message that each video is intended to express. For this to happen, the method deepens into what multimodal theory adds to metaphor research.

There is no consensus among the research community about the basic tenet for the construction of an appropriate theory on multimodal metaphor, even though there is remarkable research done in the field (Foreceville and Uriós-Aparisi 2009; Jewitt 2011; Kappelhoff and Müller 2011; Müller and Cienki 2009; Rossolatos 2013, 2014). Some authors (Cienki and Müller 2008; Müller and Cienki 2009) claim that multimodal metaphors are constructed in various articulatory modalities, which entail, in the context of film theory, all the tools that cinema has to express meaning, that is, visuals, sounds, editing techniques, and the like. Forceville and Urios-Aparisi (2009) supports this claim and adheres to it in what he defines as a multimodal metaphor. According to the author, a multimodal metaphor is a metaphor in which the target and the source domains are represented in different communicative acts, semiotic resources (Pun 2008), perception channels (Norris 2013), or “modes” (Forceville and Urio-Apaisi 2009). These modes are mainly spoken and written language, visuals, sound, gestures, smell, taste, and touch. As an example, the linguistic metaphor PLANET WARMING IS ICE-CREAM MELTING could be depicted in a picture with a slogan or motto and planet earth melting like ice-cream. In this case, we have the visuals plus the written language, both domains of the metaphor use different communicative modes to be expressed (see Figure 1).

Figure 1:

Global warming poster designed by the Byrd Polar Research Center, Ohio State University.

Regarding multimodal metaphors in films, FILMIP endeavors the complex task of identifying these tropes in audiovisual materials (see Figure 2 for the visualization of the procedure as a whole). The method entails a set of seven steps within two different phases. In Phase 1, researchers go into an extensive analysis of the multimodal content, describing the general meaning of the materials by the identification and description of all their communicative modes, for later identification of a plausible abstract meaning and message. In Phase 2, the analyst submerges into the concrete identification of metaphorical components in a similar fashion as in VISMIP.

Figure 2:

FILMIP’s seven steps.

3 Method

The two studies are briefly introduced as follows:

1. Study 1: content analysis intended to test the reliability of the application of the seven steps. It entails a content analysis based on the Kappa and Alpha reliability coefficients to check agreement among the coding scheme developed in order to carry out study 2 (look into metaphor identification). Two different TV commercials were projected to 21 participants who had to analyze those commercials with the use of FILMIP. They had to fill out two questionnaires containing questions related to the procedure’s seven steps. The content analysis was carried out in order to assess and classify the content of what the participants wrote about the filmic ads. The results obtained from Study 1 respond to the question of whether the annotators (three coders) classified the content in similar ways and to what extent these independent annotators gave similar classifications about the meaning encoded in the textual data from the questionnaires.

2. Study 2: a qualitative analysis based on a percentage agreement index to check agreement among analysts about the metaphorically marked filmic components identified on the basis of FILMIP’s seven steps by responding to the following question: does FILMIP lead to identifying the same metaphorical elements by all analysts? Figure 3 summarizes the design of the empirical studies reported and discussed in the current paper.

Figure 3:

Design of the empirical studies.

3.3 Procedure

Two different procedures were adopted depending on the study to be performed:

1. Procedure for data collection:

Data was collected in December 2017. Participants took a 2-h training session on FILMIP in class during their English Semantics course, then they individually analyzed the two TV commercials and filled in the forms in the other two consecutive 2-h sessions. Participants received some guidelines as to how to proceed with their analyses. The first TV commercial was projected five times in class, and then the students were asked to apply FILMIP individually (they could use their notebooks for their analyses). The participants were requested to fill in the first online form as they finished their analyses of the ad. Twenty-one participants completed this first task.

The second TV commercial was projected five times again in another 2-h session in class, and the students followed the same instructions (individual application of FILMIP and filling in the online form). Eighteen participants completed the second questionnaire.

As students did their analyses individually in class, a few doubts were solved about the procedural steps. However, they were not allowed to ask any questions or make any comments while performing their analyses about their results so as not to bias any of the responses from the other students or even from the instructors.

The participants’ responses were collected online through Google Forms. All these individual responses (the data collected) from all the students were uploaded in two different spreadsheets (one spreadsheet per ad) to Google Docs. This data was segmented into clauses, as the clause is considered

the basic semantic unit of discourse […] (and) the building blocks of text which are easiest to classify. When a group of words contained a subject and predicate of its own and formed part of a sentence, this was a clause. As a rule, the clauses corresponded to the basic units of discourse. (Šorm and Steen 2013: 16–17)

All the clauses or segments extracted from the participants’ responses were separated into different sheets according to the five questions of the questionnaires (labeled under Question 1, Question 2, Question 3, Question 4, and Question 5. See Figure 4 below). The segments were in English, as the study was located within the English Studies Degree (at UJI).

1. Procedure for Study 1 (content analysis):

Figure 4:

Segments extracted from questionnaire 1.

A coding book was elaborated according to the segments (the clauses produced by participants in the two written questionnaires), and a corresponding coding scheme was designed for each question. As some of the concepts within each segment were abstract, several categories were designed under the premise of inferring what the participants meant when they produced a particular thought (Barsalou 1992). The coding scheme is shown in the table below (Table 1):

Table 1:

Coding scheme for the commercials designed according to the participants’ responses.

Question	Agua Fresca de Rosas	Black Opium
Q.1: Which is the message of the commercial?	Buy perfume to get a man Woman is purity Men are roses Feel sensual Selling perfume	You need the perfume as you need drugs Perfume makes you feel good as drugs do Perfume is tool for women to be themselves Selling perfume
Q.2: Which incongruous units have you found?	Nothing Action – Object (FISH_FLOWERS) Object – Location (ROSES_SEA) Object (NET) Object _ Action (SEA_SMELL) Object (BOAT)	Nothing Everything Object_State (PERFUME_LIKE DRUG) Music Camera movements Object _State (PUPIL-DILATED)
Q.3: What are they comparing?	FLOWERS(ROSES)_MEN (LOVE) ROSES _ BEAUTY ROSES_FISH ROSES_SENSUALITY WOMAN_GHOST PERFUME_ROSE NATURE_PERFUME FISHING_SMELLING ROSES_PURIFICATION Irrelevant responses	PERFUME_DRUGS SMELL OF PERFUME_ADDICTION BIG PUPIL_SOMETHING SHE WANTS
Q.4: Do you think there is a metaphor in this commercial?	I don’t know Yes No	Yesa
Q.5: Which is the metaphor that you see, if there is any?	Irrelevant responses ROSES ARE MEN MEN ARE FISH FLOWERS ARE EROTISM PERFUME IS SENSUALITY PERFUME IS TOOL TO FIND LOVE ROSE IS PERFUME CATCHING ROSES IS FISHING WOMAN IS ROSE	PERFUME BLACK OPIUM IS DRUG PERFUME BLACK OPIUM IS ADDICTION

1. ^aThe categories were created according to the participants’ responses, and none of the participants responded with anything but a “yes” in this question from the Black Opium commercial.

The segments from Questionnaire 1 and Questionnaire 2 (about the two TV commercials) were coded with their corresponding category. Only one category was allowed for each segment. The categories were previously created and described according to the data obtained (the participants’ responses).

Two codebooks were created in Google Spreadsheets with the following information:

1. all the segments resulting from the data collected in the online questionnaires, separated into five sheets (one per each of the five questions of each questionnaire)

2. annotation of segments according to the corresponding category by three independent coders.

A total of 114 segments were annotated for the analysis of the first questionnaire. A total of 100 segments were annotated for the analysis of the second questionnaire.

4 Analysis and results

5 Discussion and conclusion

Two studies have been performed to test whether FILMIP can be considered a reliable tool for filmic metaphor identification. Study 1, the content analysis, led to the annotation of all the responses written by the participants in two questionnaires after applying FILMIP, according to a coding scheme that was tested for agreement by means of an analysis of the Kappa and Alpha indexes. The high indices obtained indicate that significant results^[9] are achieved, with a mean value of 0.87 among two trained coders, and 0.89 among three annotators (trained plus novice).

The commercial Agua Fresca de Rosas (Adolfo Domínguez 2015), offers unanimous agreement about which incongruities are detected by the analysts and what is being compared in the ad. The final decision on whether there is a metaphor and what conceptual metaphor may be construed in the clip offers almost complete agreement. However, the result of the analysis of the first question (which is the message of the commercial?) is slightly lower (but still positive), with a 0.73 index. This may be due to the fact that question one is the vaguest of all, where participants had an ample range of thoughts for their responses. What this difference, though small, may be showing is that the more specific and concrete information we ask from the participants, the more unanimity can be reached with reliability tests.

However, as this difference does not exist in the analysis of the second commercial, we could state that a higher number of materials would lead to more refined conclusions about reliability metrics for multimodal analysis. More (and longer) materials with a higher number of elements to be analyzed would lead to more significant comparative results. The authors are also aware that genre may constitute an important factor in multimodal discourse analysis. In the future, the two studies conducted in this research will be applied to other cinematic genres such as documentaries, cartoon films, feature films, and even music videos. Such analyses with more materials from different genres will then lead to a deeper understanding of the construct of multimodal communication, letting analysts know to what extent empirical research in this field can be attained for distinct purposes, implying that multimodal materials are good candidates for reliability tests.

Regarding the qualitative analysis (Study 2), it reports very high indexes as well. However, still, some differences among the responses of the participants (different percentage indexes) can be found, which may be due to several factors:

1. Multimodal metaphors in the filmic medium are complex tropes as they are construed cross-modally through a complicated composition of mappings between both domains of the metaphor (target and source), all depicted by the interaction and precise construction of the filmic tools that the filmmaker makes use of (Müller and Kappelhoff 2018). This entails that clustering a filmic multimodal metaphor in a simple A IS B formula is a difficult task (Forceville 2006, 2007) that depends on many variables (the context, the time, the analyst, the genre, etc.).

2. Inter-rater reliability based on percent agreement does not take agreement by chance into consideration. Further alternative methods to compute this agreement will be considered for future research.

3. Individual differences play a key role in the interpretation of metaphors, and cinema is not an exception. The cultural or social background of the analysts and even their level of expertise in metaphor research usually influence their perception and understanding of a given metaphor.

However, as FILMIP is not considered a tool for the analysis and interpretation of metaphors in filmic multimodal materials, but a tool to identify metaphorical elements in these materials to later perform a different kind of analysis, the authors consider that, as the studies offer quite significant percentages in the majority of the questions, we could state that the application of the Filmic Metaphor Identification Procedure allows analysts to perform replicable studies of multimodal structured-semiotic analysis of (filmic) multimodal materials by revealing all the existing levels of signification and their aesthetic configuration. The procedure, thus, not only entails a mere description of the filmic narrative but also the reconstruction of its meaning, aiming, during its first phase (the content analysis phase, including micro and macro analysis of the multimodal materials, see Figure 1 in Section 2), at decomposing and composing the communicative modes, arriving at the finest granularity possible of the multimodal artifacts under analysis.

Nonetheless, and in a second evaluative phase, we consider that it would be greatly insightful for the method’s reliability to test the results obtained also by participants without the use of the procedure for comparability results.

To conclude, in a time where communication genres, media, and channels keep on expanding and growing, multimodal theory seems to be very well positioned for researchers in the field. Empirical methods that allow the scientific community to perform replicable multimodal analysis will raise the possibilities of multimodal communication to attain valuable global, interdisciplinary research.