Volume 3, Issue s1 - Special issue: Towards a multimodal construction grammar / Issue Editors: Elisabeth Zima and Alexander Bergs

The meaning-making process in face-to-face interaction relies on the integration of meaningful information being conveyed by speech as well as the tone of voice, facial expressions, hand and head gestures, body postures and movements (McNeill 1992; Kendon 2004). Hence, it is inherently multimodal. Usage-based linguistics attributes language use a fundamental role in linguistic theorizing by positing that the language system is grounded in and abstracted from (multimodal) language use. However, despite this inherent epistemological link, usage-based linguists have hitherto conceptualized language as a system of interconnected verbal , i. e. monomodal units, leaving nonverbal usage aspects and the question of their potential entrenchment as part of language largely out of the picture. This is – at least at first sight – surprising because the usage-based model of Construction Grammar (C × G) seems particularly well-equipped to unite the natural interest of linguists in the units that define language systems with the multimodality of language use. Constructions are conceptualized as holistic “conventionalized clusters of features (syntactic, prosodic, pragmatic, semantic, textual, etc.) that recur as further indivisible associations between form and meaning” (Fried 2015: 974). Given its conceptual openess to all levels of usage features, several studies have recently argued for the need to open up the current focus of C × G towards kinesic recurrences (Günthner & Imo 2006; Deppermann 2011; Deppermann & Proske 2015; Andrén 2010; Schoonjans 2014; Schoonjans et al. 2015; Steen & Turner 2013; Zima 2014a; Zima 2014b, in press; Cienki 2012; Cienki 2015; Mittelberg 2014; Müller & Bressem 2014; Bergs 2015; Valenzuela 2015). Departing from the usage-based foundation of C × G which takes “grammar to be the cognitive organization of one’s experience with language” (Bybee 2006: 219), these studies suggest that the basic units of language, i. e. constructions, may be multimodal in nature. This paper presents some of the current issues for a Multimodal Construction Grammar. The aim is to frame the debate and to briefly summarize some of the discussion’s key issues. The individual papers in the special issue elaborate in more detail on particular points of discussion and/or present empirical case studies.

Language is a symbolic system, whose basic units are arbitrary and conventionalized pairings of form and meaning. In fact, in light of substantive empirical evidence, Construction Grammar approaches advocate the view that not only words but all levels of grammatical description – from morphemes, words, and idioms to abstract phrasal patterns as well as larger discourse patterns – comprise form-meaning pairings, which are collectively referred to as constructions. In this paper, I will discuss the status of multimodal usage-events (multimodal constructs) for the potential entrenchment of multimodal constructions and their implications for human cognition in general. As I will argue, constructionist approaches need to pay more attention to the role of the working memory in assembling and interpreting constructions. Drawing on verbal as well as gesture constructions, I will show that it is essential to distinguish entrenched constructions that are stored in the long-term memory from form-meaning pairings that are assembled in the working memory (online constructions). Once this distinction is made, the precise role of multimodal constructs and the nature of multimodal constructions can finally be disentangled.

Some proponents of the theory of Construction Grammar have been investigating how it might address the nature of spoken language usage as multimodal. Problems confronted in this endeavour include the variability with which gesture is used with speech in terms of its (in)frequency and its (non) obligatoriness: for some expressions a certain kind of gesture is basically obligatory, but for most others it is a variably optional component depending on contextual factors. This article proposes “utterance” as a level of description above that of speech and gesture for characterizing audio-visual communicative constructions. It picks up on earlier proposals to consider constructions as prototype categories with more central and more peripheral features. The language community’s knowledge of a given utterance construction and that of any language user are discussed as “deep structures” (in a non-Chomskian sense) that provide a set of options (some more central and others more peripheral) for expression, whereby any “surface structure” is a metonymic precipitation in context of the construction’s features. An important attentional mechanism proposed that guides production and comprehension (“uptake”) of utterance constructions is the dynamic scope of relevant behaviors. Taken together, this approach may help bring Construction Grammar closer to being a truly usage-based theory.

The purpose of this paper is to address some methodological questions arising in the framework of usage-based construction grammar facing the multimodality of language use. These questions will be discussed with respect to different patterns of head nods accompanying different appositional patterns in spoken German.

The existence of multimodal constructions is highly disputed. One of the most straightforward examples of such constructions includes deictics, among them the German manner adverbial so (‘like this’) (cf. Stukenbrock 2014a, Take the words out of my mouth: Verbal instructions as embodied practices. Journal of Pragmatics 65. 80–102, 2015, Deixis in der Face-to-Face-Interaktion . Berlin, München & Boston: De Gruyter), whose deictic use requires a constellation of grammatical, prosodic and gestural as well as gaze-related features. While this multimodal construction is uncontroversial, this paper tests (and refutes) the broader claim that German so is regularly accompanied by gestures (Streeck 2002: 582, Grammars, words, and embodied meanings. On the evolution and uses of so and like . Journal of Communication 52(3). 581–596). We show that non-stressed, non-deictic so can but need not be coupled with an iconic gesture and hence doesn’t qualify as a multimodal construction.

Taking an Emergent Grammar (Hopper 1998) approach to multimodal usage events in face-to-face interaction, this paper suggests that basic scenes of experience tend to motivate entrenched patterns in both language and gesture (Fillmore 1977; Goldberg 1998; Langacker 1987). Manual actions and interactions with the material and social world, such as giving or holding, have been shown to serve as substrate for prototypical ditransitive and transitive constructions in language (Goldberg 1995). It is proposed here that they may also underpin multimodal instantiations of existential construsctions in German discourse, namely, instances of the es gibt ‘it gives’ (there is/are) construction (Newman 1998) that co-occur with schematic gestural enactments of giving or holding something. Analyses show that gestural existential markers tend to combine referential and pragmatic functions. They exhibit a muted degree of indexicality, pointing to the existence of absent or abstract discourse contents that are central to the speaker’s subjective expressivity. Furthermore, gestural existential markers show characteristics of grammaticalization processes in spoken and signed languages (Bybee 2013; Givón 1985; Haiman 1994; Hopper and Traugott 2003). A multimodal construction grammar needs to account for how linguistic constructions combine with gestural patterns into commonly used cross-modal clusters in different languages and contexts of use.

This paper presents a case study on the English construction [all the way from X PREP Y] and its co-occurrence with manual gestures in multimodal television data from the Red Hen database.

There is little doubt that spatial information underlies a great deal of our processing of temporal information. Research on the ways in which timelines are associated with specific grammatical constructions has just started, and has brought forward new challenges for construction grammar. A true multimodal construction grammar cannot be developed using the same tools and mechanisms applied so far, but will require instead a fresh approach and a careful rethinking of the whole communicative process. It is our belief than in this transition from a narrow conception of linguistic communication to a broader and more complete view of communication “in the wild”, multimodal construction grammar will play an important role.

A range of multimodal form-meaning pairs has arisen to prompt for the generic integration template called blended classic joint attention (BCJA). This article presents examples and principles.

This article explores the role fictive interaction plays in two recent political speeches during the 2012 election year in the United States. The first is actor Clint Eastwood’s keynote address to the Republican National Convention in which he engages in a fictional conversation with President Barack Obama; and the second is by Mark Sanford, former Republican governor of South Carolina and candidate for US Congress who stages a fictive debate with Democratic representative and minority party leader, Nancy Pelosi. Eastwood’s performance was roundly criticized in the media by partisans, pundits, reporters, and analysts across the political spectrum, many of whom openly speculated that the 82-year-old was exhibiting signs of senescence. Sanford’s performance did not receive the publicity of the former but was viewed by voters of South Carolina, and represents a much tighter and effective form of fictive interaction, eventually leading to his election to congress. The factors contributing to the failure of Eastwood’s performance and and those contributing to success of Sanford’s will be the focus of this analysis. Special attention will be paid to the multimodal dimensions of each performance.

This paper offers an analysis of a multimodal pattern of negative assessment, which takes as the starting point a particular recurrent gesture: the Throwing Away gesture. Recurrent gestures are characterized by a stabilized form-meaning pairing. In the case of the Throwing Away gesture, form and meaning are grounded in an “Away-Action-Scheme”: i. e., a socially shared, sedimented experience of removing unwanted objects at, on, or approaching a speaker’s body. Based on a cognitive-linguistic analysis of the gesture and its use in five different verbal contexts, we suggest that the Throwing Away gesture enters a verbo-kinesic construction that consists of the “Throwing Away Gesture + particles/negation/N/V/ADV”. The meaning of the verbo-kinesic construction is grounded in an embodied frame of experience in Fillmore’s sense: i. e., a schematized scene involving mundane actions, here ‘removals of unwanted objects’ (e. g., ‘Away-Action-Scenes’). Referring to Goldberg’s “Scene Encoding Hypothesis”, we propose that the “Negative-Assessment-Construction” designates scenes essential to human experience. With this focus, the paper puts forward a gesture-first account on verbo-kinesic constructions and suggests a possible candidate for such a multimodal pattern.

This contribution focuses on the shrug as it is used in stance taking contexts in face-to-face conversations. The shrug qualifies as a ‘compound enactment’, in which prototypically different gestures are combined: “the eye-brows (which are being raised), the hands (which are turned so that the palms face up), the forearms (which may be lifted), and the shoulders (which are also raised). In addition, the head may be tilted” (Streeck 2009: 189. Gesturecraft. The manu-facture of meaning . Amsterdam: Benjamins). More recent studies show that instead of a head tilt, a headshake can also be a part of the shrug (Schoonjans 2014. Modalpartikeln als multimodale Konstruktionen: Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen . Leuven: KU Leuven dissertation). We report on an empirical study, in which the shrug or some of its gestural components are used to express obviousness on the part of the speaker. Although our data reveal multimodal patterns in the expression of obviousness, this study singles out the gestural dimension. The paper is structured as follows. Section 2 explains the methodological aspects of our study, after which Section 3 presents the formal and quantitative results, illustrated with four examples. The paper ends in Section 4 with a brief discussion of our major findings.

If multimodal work in terms of Construction Grammar is still rare nowadays, this is not just because the upsurge of multimodality in linguistics in general is rather recent. Attempts to include non-verbal layers of expression (especially gesture) in Construction Grammar have uncovered a number of theoretical issues that need to be reflected upon before any serious claims on the existence of multimodal constructions can be made. While some scholars take these issues as reasons for leaving the non-verbal outside of the scope of Construction Grammar, this paper shows that several of these issues are not actually related to multimodality, but rather hint at more general theoretical issues in Construction Grammar. Hence, it is argued that these issues should be seen as an incentive for rethinking and refining the notion of ‘construction’, rather than as a reason for leaving multimodality aside.

A multimodal construction is said to be a conventional pairing of a complex form, comprising at least a verbal and a kinetic element, with a specific meaning or a specific function. Do we need a new constructional approach to account for such multimodal constructions? What are the challenges to account for multimodality? The aim of this contribution is to provide a precise notion ‘multimodal construction’ and, on this basis, to indicate possible pathways for future investigations. The paper opts for cautiously extending the scope of existing constructional approaches in order to include non-linguistic meaningful behavior. In particular, it is argued that even though Construction Grammar invites for treating multimodal on a par with linguistic constructions, there is a huge lack of substantial empirical support to arrive at a more detailed and data-based understanding of the nature of multimodal constructions.