Bold and impactful: a reappraisal of Gunther Kress’s (social) semiotic legacy in the light of current multimodality research

3.1 Signs and sign-making – Saussure

Perhaps the most fundamental and bold move for Kress’s semiotics was to reject the strict conventionality governing the coupling of signifier and signified (cf. Figure 1).

Figure 1:

Saussure’s bilateral sign-model.

In a 2016 interview, Kress characterizes Saussure’s notion of symbol as “a term that I do not usually use” (cited in Björkvall 2016: 26). To him any sign would always be motivated, and he was thus strictly opposed to arbitrariness as a semiotic principle (cf., Kress 1993).

“In Social Semiotics arbitrariness is replaced by motivation, in all instances of sign-making, for any kind of sign.” […] “The notion of arbitrariness goes directly against the notion of the sign-maker’s interest in the making of signs and meaning. […] Convention points to social agreement and power in sign-use. Motivation points to the need for transparency as a means towards shared recognition in the relation of form and meaning in communication.” (Kress 2010: 67, 64; original emphasis)

I can see why Kress wanted motivation to be elevated to a universal principle: His argument essentially is that the choice of a sign is motivated by the (ideologically conditioned) interest of the user. However, I would want to disagree, as others have done before (cf., McDonald 2012: 208; Forceville 2011). Forceville (2011: 3624) has called this full-blown motivation of all signs “an untenable claim” and argues that “Kress here appears to consider Saussurian arbitrariness as symptomatic of an ideology rather than as a plain fact”. To me, it seems Kress conflated two different levels of semiotics: sign structure or the formation of a sign and sign use. Whereas in terms of sign constitution, arbitrariness or the symbolic seem to be an undeniable reality for a large class of signs, in terms of sign use, one could talk about the metaphorical or metonymic re-making of signs in every instance. Kress’s well-known visual and verbal examples (Kress 2010: 55) of developing literacy skills of children (e.g., wheels for car – in drawing; heavy hill for steep hill – in speech) testify to this potential for the constant re-fashioning of forms for meaning and its cognitive basis.

But consider the following unequivocal interpretation of simple linguistic data as evidence for the original, systemic arbitrariness of linguistic signs and the easy move to structural, i.e., grammatical, motivation. In linguistic morphology, dis-incentiv-ize would be analysed as being derived from the Latinate root incentive, which clearly displays an arbitrary relation between form and meaning. As we move to the compositional level, elements of the complex structure become transparent: the suffix {-ize} regularly forms verbs from noun bases, and the prefix {dis-} reverses the base meaning.

What, then, are the consequences of denying signs conventional status as symbol and claiming motivation for all signs? First, it would seem wise to be able to gauge different degrees and types of motivation (by metonymy, metaphor, analogy etc.). If all signs are motivated, such distinctions are no longer possible as an option. Secondly, if the difference between arbitrary/symbolic and motivated (iconic/indexical) sign-object relations is ignored important distinctions are levelled, like the essential one between verbal and pictorial signs. In principle, the more semiotic distinctions one can make, the better – in this fashion, one can create sign typologies, a vital undertaking in multimodality studies (Stöckl 2016: 9–19).

Pictorial signs like the ones in the social advertisement in Figure 2 – i.e., a tree being felled and torn into pieces, two people holding hands and being torn apart – are by definition motivated through similarity with real world objects, which we can retrieve from our encyclopaedic knowledge once we recognize the likeness. The claim It’s not just the tree that gets torn apart relies on an easy recoverability of the visual analogy; we effortlessly grasp its sense because pictorial signs are motivated and do not require a recoding from abstract phonetic/graphic form into concrete referential meaning.

Figure 2:

Big Trees, TBWA Bangkok & Illusion Studios Bangkok, Lürzer’s Archives 4/2019: 97.

I do agree with Kress that motivation as a way of reinstating transparency in conversational/textual use is important in many ways. But this is something I would prefer to call re-motivation in two senses: one can trace a sign back to its origin, i.e., explicate its original motivation, and one can attach a new meaning to an existing sign and newly motivate it in context (cf. example of ping below). However, these are marked cases in communication. As a rule, any sign use inevitably leads to demotivation over time, which is necessary for us not to get lost in what some have called ‘wild semiosis’ (Gross 1994). So, while to ping (i.e., make a ping) may still evoke the original onomatopoetically motivated association with a sharp sound, the derived, metonymic meaning (i.e., send someone a notification) must lose it in the long run for an efficient functioning of language.

3.2 Sign use and meaning-making – Peirce

While Kress (2010: 65) explicitly rejected Peirce’s tripartite division of sign types, he generally approved of Peirce’s sign model (p. 62) (cf. Figure 3).

Figure 3:

Peirce’s trilateral sign-model.

This is understandable against his notions of motivation and his emphasis on the user’s flexibility in sign-making. In his definition of semiotic mode, Kress (2014: 61) stresses the representamen, when he stipulates that: “in a social semiotic approach to mode, equal emphasis is placed on the material ‘stuff’ of mode and on the work of culture over often long periods with that material” (my emphasis). In the same vein, his rejection of arbitrariness and his stress on the sign user’s interest make him embrace Peirce’s twofold notion of interpretant: as (1) the sign-processing and interpreting individual and (2) as the actual instantiated meaning of a sign. Kress (2010: 62) does not only highlight Peirce’s notion of a chain of semiosis as an imminent consequence of this dual notion of interpretant. He also takes Peirce to mean that: “‘Readers’ are agentive and transformative in their semiotic engagement with signs” (p. 62). This again is a nod to his own emphasis on the constant re-making of signs.

Two cautious comments seem in order here: First, Peirce (1931) himself introduced three ways in which the material of a sign can unfold meaning (cf. Nöth 1985: 39 and 40–41 and 42–43; cf. Table 1). While legisigns as the classic Peircean symbols (i.e., spoken/written words) largely abstract away from material qualities, the so called quali- and sinsigns utilize and thrive on material aspects of the signifier. Qualisigns can make full use of their material qualities: for example, all phonetic-acoustic qualities of speech sounds become meaningful (as in Dadaist poetry, soundscapes but also all visual qualities of images in naturalistic coding orientation). Sinsigns make only partial use of their materiality; singular qualities are picked out to represent meaning (such as colour in logos, volume in cries or length/breadth in diagrams).

There is a danger in Kress’s emphasis on signing material of assuming every change of material quality to effect a change in meaning. This would draw away attention from the fact that it is often only certain qualities of the material that become meaningful. Abstraction, therefore, remains a necessary ingredient in any sign-making; it is a notion that fundamentally facilitates and leads into grammar. In his brief discussion of the motivation behind the Red Cross logo, for instance, Kress (2010: 65) appears to suggest that the materiality in the various regional/national versions are differently motivated, putting what I believe too much of and too undifferentiated a focus on the signing materials. After all, despite the variety in form, the Red Cross logo is a legisign, which in its object relation clearly functions as a symbol with perhaps the addition of the metonymic index of the red colour (pointing to blood). The option to remotivate the logo’s main sign through the religious cross and Swiss neutrality (cf. Kress 2010: 65) does not alter the symbol-cum-index status of the logo.

Second, Peirce’s (1931) sign model also allows us to distinguish between sense and reference. While I agree that sign-users are quite flexible in the ways they make and use signs in order to refer to objects, phenomena or states/processes in the world, the same flexibility cannot be claimed for the sense of signs. This is because a sign’s meaning is not an island; it is much rather a relational affair: “the basic reality of language consists not of elements but of relationships” (McDonald 2012: 209). Saussure (2002) emphasized this when he preferred to talk about the value of a sign, which is ultimately only determined by the relations a sign keeps in paradigms of similar senses and in syntagmatic units of neighbouring senses. Sense is much more conventionalized and stable than reference is, or in other words, sense is less motivated than reference. For example, the English deer (Old English deor) only became free to acquire a new sense (special type of animal, i.e., cervid) once its original, much more general sense (beast) had been taken over by the Norman French loanword animal in the 14th century. It is such often intricate networks of semantic relations that determine the value of a sign in Saussure’s sense. On the other hand, it is true that speakers’ changes in the reference of a given word are motivated and conscious decisions (see the example of ping above), which was given a new, more specific reference. It seems true to argue then that it is novel references established in use that are the catalysts which lexicalize new senses.

3.3 Sign function and aptness in a rhetorical situation – Bühler

It is not surprising that Bühler’s (2011 [1934]) model (cf. Figure 4) fits Kress’s semiotic bill best, as systemic functional linguistics engaged with Bühler’s theory building early on (cf., Halliday 2007: 173–174).

Figure 4:

Bühler’s (2011 [1934]) ‘organon’ model.

The ‘organon’-model is both a model of the sign and a simple model of semiosis. It triangulates the sign by relating it to three well-known entities – i.e., sender, receiver, and objects/states of affairs – and in doing so derives sign-functions and processes. By regarding signs as emerging from individual expression, enabling representation and as instruments of relational work, Bühler (2011 [1934]) seems to have anticipated functional linguistic approaches (cf., three meta-functions).

The model is also reminiscent of Aristotle’s well-known triad of persuasive appeals: ethos, logos and pathos (cf. Figure 5).

Figure 5:

Aristotle’s three persuasive appeals.

Bühler’s (2011 [1934]) model acknowledges “the essential rhetorical fact that any sign use must in effect express the ethos of the rhetor, represent their rational take on the world (logos) and appeal to the emotional mindset of an envisaged audience (pathos)” (Pflaeging and Stöckl 2021: 2).

This is very relevant as Kress (2010: 121) was a vocal proponent of the rhetorical and argued that any sign maker is inevitably a rhetor involved in what he called a rhetorical process. Not only did he cast semiosis as an inherently task-based or task-driven activity, he also emphasized that the choice of any sign (and even more so any combination of signs) is guided by situational and functional aptness. “Aptness means that the form has the requisite features to be the carrier of the meaning” (Kress 2010: 55; original emphasis) and “aptness focuses on ‘fitness for purpose’: ‘this is the best fit (the most apt) for this purpose here’” (Kress 2010: 156; original emphasis).

Such considerations of aptness or appropriateness – to borrow a term from stylistics – directly lead into notions of genre or genre expectations: What multimodal design best solves a communicative task and which forms have become conventional in order to solve it? After all, one of the crucial ideas behind genre is the correlation between context (of culture and situation) and multimodal form.

3.4 Semiotic mode (code) and medium in semiosis – Jakobson

Jakobson’s (1960) model (cf. Figure 6) clearly is one of communication or semiosis.

Figure 6:

Jakobson’s (1960) model of semiotic functions and communication.

The model also has a fundamental functional orientation as each of its elements is tied to a basic communicative function (Jakobson 1960). For example, the medial contact between the communicators establishes the phatic function, i.e., a technological and socio-psychological communion between them. The code, for instance, sets up the meta-lingual/-communicative function, that is, our ability to reflect on the structure of our sign systems and their use.

Kress did not downright reject Jakobson’s model but he perceived it as too static, especially finding fault with “the shared code, which we absolutely cannot assume anymore” (cited in Björkvall 2016: 27). Instead, he simply stated that “communication happens when there is interpretation” (cited in Björkvall 2016: 26). Kress’s rejection of ‘code’ points again to his much more dynamic view of meaning-making as a process driven by the sign-using individuals and their situational and ideological motivations. But his position also affirms two different types of communication: simple unilateral communication, i.e., a sign-receiving individual taking certain perceptual information as meaningful (cf. Nöth 1985: 124), and multimodal interaction in all its processual complexity.

Most elements in Jakobson’s (1960) model and its concomitant functions are now seen as largely uncontroversial, apart from contact (medium) and code (mode). It is exactly the relation between media and modes that Kress was foundational in determining and this has influenced much thinking in multimodality research. This relation is anything but trivial, which has to do with the difficulties of defining the two notions in the first place. Kress and Van Leeuwen (2001: 21–22) made it very clear that one cannot be conceptualized without the other: media as material and tools afford and give rise to modes. Modes, in turn, are shaped by their affordances and come to depend on them.

This reciprocal relation is the sine qua non of modern semiotics but it is contested in essentially two ways. First, the very notion of medium has been interpreted in many ways – from narrow interpretations (just material) to broader ones (including technological tools, sites, procedures, communicative conditions etc.). While some have been adamant in keeping media analytically separate from modes (cf. Luginbühl and Schneider 2020: 65), others have sought to integrate material in a notion of mode (cf. Bateman et al. 2017: 113–117). It is clear that broadening out one’s notion of medium makes it harder to show how mode properties are shaped and constrained by the medium. Second, opinion differs as to “the boundedness and distinctiveness of modes” (Stöckl 2019: 48). Kress (2014: 65) can be seen to have taken a heuristic, socio-historical position here, arguing that “a mode is what a community takes to be a mode and demonstrates that in its practices”. Others, including myself (Stöckl 2016: 9–19), have advocated the possibility of delineating and characterizing distinct modes. In this latter view, code properties, i.e., regularities of grammar, are given priority over materiality in being a formative influence on mode.

Speech and writing are a case in point: one may treat them as medial variants of one mode (cf. Stöckl 2014: 276), highlighting their shared grammar, or as two different semiotic modes, highlighting their different materiality and situational uses. Kress (2010), emphasizing the shaping effect of medial material on the code, argued for them to be separate semiotic modes. Others have criticized this view, claiming that “what is shared by both written and spoken forms of a language is precisely the same langue, […] ‘an equal number not of concepts, but of values understood to be mutually differentiated’ (Saussure 2002: 60)” (McDonald 2012: 210–211). Kress (2010: 104) was well aware of these arguments and wisely conceded that: “at a lower level of generality they do differ and at a ‘higher’ level they don’t”.

Whatever the case may be, an adequate and astute characterization of the inter-relations between medium (in its many-faceted views) and mode remains the central question in multimodality studies. After all, if one wants to do multi- or inter-modal research, they need to clear up their concept of mode first and relate it to concomitant notions.

4.1 Defining semiotic mode

It is a truism that if one wants to do multimodality, being able to define semiotic mode would seem to be the single most important task to solve. Therefore, it may be worrying that opinion about this is quite divided still – so much so that some despair:

There is a serious risk that as a result of a precise definition of what counts as a mode, and of the cherished openness of the concept, mode-status can be accorded to any meaning-generating principle or dimension. […] the whole concept of mode becomes so hazy as to become completely vacuous. (Forceville 2020: 66–67; original emphasis).

While this may be a perhaps exaggerated critique, I think one must concede the persistent difficulty in ascribing mode status and delineating various modes. It is important to understand that the difficulties are inherent and stem from the interwovenness of the things we seek to separate out in the very processes of meaning-making: media and their material/technological procedures, sign systems or codes to acquire and master, types of communicative practices (genres) and the concomitant social and rhetorical competences and techniques (cf. Luginbühl and Schneider 2020: 65).

Kress (2010: 84–92) was deeply invested in defining mode and in discussing whether semiotic systems qualify as modes. He would agree with Jewitt et al. (2016: 9), who say a mode is “a set of socially and culturally shaped resources for making meaning [with] distinct affordances”. Importantly, Kress (2010: 87) recognized two sides to any mode: a social one and a formal one. He understood the social as the shifting representational needs of a culture and the formal as the functions that modes must invariably fulfil. According to Halliday (2007: 183–184), these are the ideational, the interpersonal and the textual. So, one way of ascertaining mode status would be to study how the resources of any potential mode work towards realizing these social and functional dimensions (cf. Stöckl 2014: 280–281).

Others have preferred a stratified modelling of mode, integrating various interconnected dimensions. Elsewhere I argue that any mode must be processed through a sensory channel, build on a mediating physical/technological material, and possess a however rudimentary grammar in the sense of lexis and syntax (Stöckl 2016). Bateman et al. (2017) suggest a stratification into materiality, grammatical regularities, and discourse semantics. In this model, the ingenious or notorious – depending on one’s views – notion of discourse semantics is incorporated as an inherent property of mode, which means discourse does not only emerge on the multimodal level but is scripted into each mode. Sceptics such as Forceville (2020: 67) have called this idea “daunting”, but do not offer alternatives.

Whichever way one decides to define mode, we must by all means acknowledge the fact that Kress’s insistence on the shaping of a material and a regularized (grammaticalized) set of resources has had a supremely formative effect on any approach to the concept. It is my conviction that a stronger emphasis on the grammar of modes and their reaches could facilitate the delineation of individual modes. Fuzzy boundaries between modes and their seamless integration in multimodal ensembles must not blind us to the importance of being able to clearly describe and claim mode status for semiotic phenomena.

4.2 Using language as a yardstick for measuring grammaticality

Doubtlessly, Kress’s insistence that all communication is invariably multimodal has shaped his semiotic thinking more than anything else (cf. Wei 2020: 3). It also formed the starting point for multimodality research proper. When some now claim that this is an independent academic discipline with a promising trans-/inter-disciplinary future (cf. Wildfeuer et al. 2019: 21–33), we must pay tribute to Kress’s early systematic moves. However, his focus on functional mode equality has also led to an interesting contradiction, which it is vital to acknowledge: Multimodality rejects the centrality and dominance of language, yet it has variously attempted to use the notion of linguistic grammars to model non-verbal semiotic modes. In line with the logocentric critique, Kress understood grammar metaphorically as some kind of mode-internal ‘regularity’ of structure, yet also took it perhaps too literally when transferring notions of verbal grammar onto the “grammar of visual design” (Kress and Van Leeuwen 1996). Some now look at these moves quite sceptically, for instance McDonald (2012: 214), when he asks “in what sense […] may we meaningfully regard language as being ‘paradigm for the semiotic’.” He goes on to claim that we must “put all semiotic behaviour […] on an equal footing” (McDonald 2012: 213). This is a plausible demand that calls for a refined body of semiotic concepts and necessitates more foundational work, comparing different modal grammars.

In my view, McDonald’s and similar critique of social semiotics for using language grammar in the description of images, sound or colour (Kress and Van Leeuwen 1996, 2002; Van Leeuwen 1999) is exaggerated and not entirely fair. This is because the modal grammars proposed are not strictly linguistic but ultimately functional grammars, i.e., they employ meta-functional categories. So, the accounts of pictorial and chromatic grammar (Kress and Van Leeuwen 1996, 2002) relate visual structures to meaning potentials in various functional dimensions, whereas sonic grammar (Van Leeuwen 1999) relates sound qualities to semantic values. The quest guiding these grammars then is to identify modal resources that can be more or less regularly tied to meanings and communicative functions. This is noticeably different from transferring the structures of the master code language onto non-verbal modes.

Generally, if we are to avoid the pitfalls of a linguistic (grammar) bias, two issues merit special attention: (1) By definition, pictorial, musical, verbal etc. signs are fundamentally different semiotically. Their materials allow for different regularities of structure, different meanings, different ways of perception and cognition as well as different ways of combining with other modes in inter-modal coherence. We would do well to acknowledge these distinctions more and accept the consequence, which – as McDonald (2012: 214) puts it – is “the challenge(s) of theorising all semiotic systems on their own terms”. This also entails accepting different degrees of grammaticality (strong vs. weak) and even differing conditions for the discreteness of signs in the first place (cf. Kress’s [2014: 63–64] ‘reach of mode’). (2) While it is important to compare semiotic modes and tease out their individual grammars, the main focus in multimodality studies must be not on a single mode but on the interaction between and the combination of two or more modes in multimodal genres. This shifts the emphasis to the notion of multimodal cohesion and coherence, i.e., intermodal ties and meaning multiplication or message complementarity (cf., Stöckl 2019: 53–58). In turn, an emphasis on mode-integration raises the issue of genre in two ways (cf., Stöckl 2019: 65): first, individual mode use instantiates a genre (e.g., advertising image vs. press photography or jazz vs. symphonic music), and second, multimodal texts or events only come in genres (e.g., news story vs. commercial vs. tutorial).

5.1 Multimodal genre

Kress (2010: 116) understood the significance of genre full well, when he stipulated that “genre mediates between the social and the semiotic”. In the light of Jakobson’s (1960) model, saying that genre straddles context and semiotic message structure would be an appropriate interpretation. Forceville (2020: 121) calls it an “ever-present ‘interface’ between discourse and context”. In the vein of Kress’s insistence on the material of signs, Bateman (2008: 10) has argued that “the very materiality of multimodal artefacts […] constitute[s] a crucial component of any complete account of multimodal genre”. As a “pattern of organization of semiotic material along a number of dimensions […] and in relation to particular types of situational constraints” (Frow 2015: 80), the significance of genre cannot be overstated. This centrality is echoed by Bateman (2016: 60), when he claims that “without genre allocation, it is often not possible to provide a sensible description of a text, multimodal or not, at all”.

What semiotic work do genres perform? In the light of Kress’s rhetorical process their function is dual: for the producer/designer, genres are “bundles of strategies for achieving particular communicative aims in particular ways” (Bateman et al. 2017: 131). For the recipient, genres are primarily a safeguard to ensure adequate understanding and situational response. If we look at the receptive side more closely, we find that genres provide discourse conventions which “encourage certain kinds of interpretive behaviour or response” (Forceville 2020: 121). Such conventions come in the form of generic cues (cf. Forceville 2020: 121–122), both internal (as structures and styles of semiotic form and content) and external (as the spatio-temporal circumstances surrounding a multimodal artefact or communicative event). Even though these interpretive conventions are not immutable but may be negotiated by the communicators within institutional frameworks, it is their generic cues that afford genre recognition or attribution.

For example, Pflaeging and Stöckl (2021) demonstrate how one can measure and graphically depict the conventional generic pattern employed in National Geographic feature articles. The resulting visualizations faithfully record the flow of text and images across multiple double-page spreads, a task that can only be accomplished by gauging the design of type, colour and layout. It is such abstract multimodal patterns, in this case on the level of graphic perception, that readers recognize and use for guidance in their daily navigations of the genre universe. The language used in feature articles provides other kinds of generic patterns, e.g., concerning lexical choice, grammar or rhetorical devices.

Despite or because of the immense weight that genre carries in descriptions of multimodal discourse, opinion as to how to best approach it in analysis is divided. One of the problems with genre appears to be that we are seeking to capture a type of knowledge or a type of socio-cognitive semiotic skill. Both are immaterial and can only be accessed indirectly through genre exemplars. The tricky decision in any framework is to select those dimensions of semiotic structure and those aspects of the situation/context that have the strongest predictive power for any one genre and across a large range of genres. Communicative function (also, rhetorical task) usually serves as the primary dimension, the point of departure for any description (cf. Van Leeuwen 2005: 117–122). It is correlated with aspects of the social setting (communicators), the context (medium etc.), the types and sequence of stages, and the style patterns in the use of the individual modes (cf. Stöckl 2019: 59–60). I have advocated establishing the types of multimodal coherence and intertextual/interdiscursive relations as two other important elements in multimodal genre description (Stöckl 2016: 22–25). Bateman’s (2008: 61) GeM-model conceives of 6 layers of description: layout and navigation are a nod to the materiality of genre, while content and rhetoric capture its cognitive underpinnings.

In summary, whichever approach we take, a prime aim of multimodal analysis must be to show how the selection and integrated design of the modes respond to the regulating regimes of the genre.

5.2 Multimodal discourse interpretation

The value of theorizing in semiotics and multimodality research may well be measured by the gains it provides for approaches to real-life communication, be these analyses of individual samples or large-scale collections of artefacts. Kress was praised for the examples he worked with. Van Leeuwen calls his “sense of direction in selecting these texts […] astonishing” and claims they were “the very source of our ideas” (cited in Bezemer et al. 2020: 4). Clearly, Kress’s semiotics was built on case studies that would be ideally representative of contemporary communicative practices and that would instantiate typical phenomena that could then be modelled in general theory.

What Kress produces in his sample analyses with admirable clarity is what in multimodality would be referred to as ‘discourse interpretation’, which simply means that an interpreter is “engaging in abductive discourse hypotheses concerning their ‘meaning’ or signification” [‘their’ referring to the communicative situation and a multimodal artefact, H.S.] (Bateman et al. 2017: 222). Multimodality research today relies less on isolated examples and increasingly moves towards larger collections of data that can be systematically annotated and analysed (cf. Norris 2019; Pflaeging et al. 2021). Pflaeging et al. 2021 (9–16) impressively chart the increase in empirical papers published in the three leading multimodality journals between 2000 and 2020. Their research also indicates that the classic qualitative approaches to multimodal text and interaction are being supplemented by quantitative and mixed methods. At the same time, the share of medium- and large-scale data in empirical studies has grown. If these trends continue, more valid observations can be made about the multimodal structures typical of a genre and more tenable claims be advanced about the overall workings of multimodal communication.

When working with larger-scale, multimodal corpora, the primary aim is to annotate the data for suitable individual properties. In a recent study on print advertising involving computer-generated images (CGI) (Stöckl 2021), for example, I annotated 232 advertisements for image type, visual structure, rhetorical potential, argument type and relational propositions. In this fashion, insights could be gained about the typical and salient patterns of text-image relations that support multimodal argumentation and the role that visual rhetoric plays in this. However, annotation itself presupposes a number of hypotheses about how the discourse we are studying is to be interpreted. Moreover, the larger the number of multimodal properties that are coded in the annotation, the greater the work of piecing all individual patterns together in order to arrive at a plausible overall discourse interpretation. So, there really is a challenge in multimodal corpus studies to not neglect the actual meaning-making and its underlying cognitive mechanisms for the benefit of accumulating isolated multimodal properties. Discourse interpretation is not trivial: it is as much a cognitive task to the analyst as it is to each individual recipient. After all, even if genre conventions constrain the “free pragmatic process” (Forceville 2020: 125), multimodal interpretations still rely on much knowledge, inferencing procedures and rhetorical strategies, which remain in the dark as taken for granted and prototypical.

Forceville (2020: 35–48) has recently drawn attention to the standard but rich pragmatic and cognitive mechanisms involved in multimodal communication, taking up earlier work on logical abduction and inference (cf., Bateman and Wildfeuer 2014). The point of departure here is that signs of whatever kind will always be underdetermined. This is why producing a situationally relevant interpretation of a multimodal piece of communication requires a number of mental operations. These are usually grouped together under inferences but are in fact of various kinds and fulfil different functions. I will briefly illustrate the abductive process suggested by Forceville (2020) along the lines of Relevance Theory (see Sperber and Wilson 1986; RT for short) on a recent social ad for an HIV counselling foundation designed by Illusion Studio, a renowned CGI agency (cf. Figure 7).

Figure 7:

Hyacinth, Harrison & Star, New York & Illusion Studio Bangkok, Lürzer’s Archives 1+2/2021: 101.

1. The dark, obscure and seemingly irrelevant image creates the attention signal and provides a stimulus that must be interpreted in the context of the accompanying copy.

2. In order to arrive at an explicature (Forceville 2020: 46), recipients must perform a number of mental operations, all adding to the sign’s ostensive meaning. Successful interpreters will clarify objects, people, actions, states (reference assignment, i.e., an entrance to a lit front door), resolve doubt about the exact reference and logical form (disambiguation, i.e., the shape of the lit door also represents a syringe/needle), and most importantly, adduce information from the context and consequently generalize or narrow down possible meanings (enrichment).

3. It is at this stage of enrichment that multimodal texts offer a unique meaning-making principle: one mode becomes the context of another and thus acts as a source for disambiguating and enriching the other. In the example, how you got HIV and those facing HIV are textual cues in metonymic relation to the needle. The light by the door metaphorically connects to the help promised (cf., come knock on our door). Moreover, the explicatures of various modes must be related to one another through rhetorical operations (metaphor and metonymy) or relational propositions (the welcoming, helping attitude promised in the text is given evidence in the image). Trans-/intertextual and genre knowledge (HIV-patients need clean needles; social ads offer help) is also activated in enrichment.

4. Finally, explicatures are combined with implicatures, i.e., assumptions not explicitly communicated but drawn from the cognitive environment of the communicators. In the example, the implied connections between the dark outside environment and judgement can easily be inferred as can the link between the warm light inside the house and love.

This process model is anything but simple even though it abstracts from many details. By providing a sensible framework for what seem to be indispensable cognitive operations relative to situational/contextual factors, it sketches tasks that any model of multimodal discourse interpretation will need to cover.