Gradient at-issueness and semiotic complexity in gesture: a response

2.1 Figurative iconic enrichment

Following Ebert et al.’s (2020) proposal, Barnes and Ebert suggest that a concrete iconic gesture refers to an entity, rather than an abstract property that is understood as applying to an entity via some mechanism of multimodal composition. In a recurring example involving a “bottle”, repeated in (1), a big gesture iconically conveys information about size, but still refers to the bottle itself, where all properties of the bottle aside from size are abstracted over. One motivation for this treatment is to create a unified analysis of pointing gestures and other semantically-oriented gestures – they all refer, and some, such as the big gesture, also provide iconic enrichment. The problem here is that multidimensional meaning is reduced to identifying equivalencies between information in the spoken and gestural modes.

This treatment of gestural meaning is only straightforward in cases where the iconic enrichment is a literal physical property of a salient referent. The account becomes problematic when applied to similarly iconic gestures that convey hyperbolic and metaphoric properties of a salient referent. Consider the following two examples.

In (2), the person presumably does not literally want a piece of cake that would fit between their two fingers. More likely, they are dissatisfied with the seemingly standard cake slice, and would really like something smaller. Here, the hyperbolically small gesture is meant to reinforce the request for the smallest piece of cake, not a piece of cake that literally shares the dimensions of the gesture.

In (3), the salient referent (the problem) is an abstract entity that cannot be directly attributed with a physical property like size at all. Instead, there is a metaphoric process by which the problem, as an abstract entity, is understood as a physical object with particular physical characteristics. These particular physical characteristics then elaborate the metaphor, such as by mapping size to importance. The meaning of the gesture itself, however, is the same as in Barnes and Ebert’s bottle examples – it is iconically depicting the literal size of a physical object. It is the contextualized interpretation of the gesture that is different.

If we were to maintain that the gesture in each of these cases referred to “the cake” and “the problem”, we’d be placed in an awkward position. The object referred to by the gesture would be iconically incompatible with the object referred to in speech. All three gestures convey information about size by physically delineating a bound region of space, and as such iconically resemble a physical object that occupies a bound region of space. In the literal case, this resemblance maps directly to the referent in speech (in their case, a bottle). In the two figurative cases, the mapping is indirect – the iconic representation of a literal object with literal spatial dimensions must in some way be related to, in the first case, a literal object with different dimensions, and, in the second case, an abstract object with no spatial dimensions at all. The process by which these gestures are interpreted in context cannot be accounted for by reducing the gesture’s meaning to a salient discourse referent.

2.2 Iconicity in discourse-oriented gesture

There is a common functional distinction drawn between gestures that seem to convey semantic, truth-conditional meaning and those that convey socio-pragmatic or discourse-structural meaning (e.g. Bavelas et al. 1992; Kendon 2004; McNeill 2005; Streeck 2009). Semantically-oriented gestures typically align, temporally and meaningfully, with a co-expressive spoken phrase. As such, these are the gestures that formal multimodal models tend to focus on. Discourse-oriented gestures, on the other hand, often do not have a clearly affiliated phrase in the spoken mode and are often not considered in the creation of formal models. However, despite the differences in function and temporal alignment with speech, there is formal overlap. A gesture that seems to convey iconic semantic meaning (literal or figurative) in one context may serve as a kind of pragmatic marker in another. As an example, consider again the big gesture from Barnes and Ebert. This time, instead of aligning the gesture with a particular referent, imagine that the gesture aligns with an entire utterance, as in (4).

In this case, the big gesture seems to convey something about the discourse topic and what the speaker intends to do with it, rather than literal or metaphoric information about a referent’s size. The gesture may give us the impression that the speaker really wants to pursue the particular topic, or perhaps views the topic as particularly complex. As before, I would argue that the iconic meaning of the gesture itself is the same as in its concrete semantically-oriented use. The gesture, as an independent unit of meaning, conveys spatial information in all cases that is then interpreted differently in context. In the above discourse-structural use, this spatial information provides iconic enrichment for the discourse topic itself. The topic, and the task of pursuing that topic, is reconceptualized as a large, and thus perceptually salient, object.

In this case, it is obvious that the gesture-as-referent account cannot hold. One’s response to this may be that this is explained by the temporal alignment of the gesture with an utterance rather than a DP referent. However, my point here is that when a model of multimodal meaning separates ‘concrete iconic gestures’ from ‘metaphoric gestures’ and ‘discourse-structural gestures’, then the model fails to capture the shared iconic meaning of similar gestures in different contexts. In doing so, I would argue that we run the risk of misrepresenting gestural meaning in our pursuit of a multimodal understanding of language, and in turn risk replicating the models that were designed with only speech in mind.

3.1 Paying attention to gesture

In addition to the two verbal strategies noted by Barnes and Ebert, Cooperrider (2017) suggests two gestural strategies that serve to ‘foreground’ the communicative intent of a gesture – speaker gaze and “conspicuous effort”. Speaker gaze relates to the ability to direct joint attention by shifting one’s gaze rather than by using, for example, a pointing gesture. It has been shown that when speakers look toward their hands while gesturing, their addressee will also turn their attention to the gesture (e.g. Gullberg and Holmqvist 2006). If ‘at-issueness’ is indeed related to the salience of particular information in the unfolding discourse, then this suggests that gaze can independently increase a gesture’s at-issueness by increasing its literal perceptual salience.

Speakers can also direct an addressee’s perceptual focus by putting more physical effort into a gesture, increasing its size or shifting its position. For example, in Enfield et al.’s (2007) work on pointing gestures, they found that the size, duration, and position of a point correlated with whether or not the location of a referent was the main topic under discussion. If the location of a referent was the focus of conversation, pointing gestures often involved full arm extensions, were held for a longer period of time, and were accompanied by a reorientation of the speaker’s gaze and body toward the referent. When the location of a referent was ‘supplementary’, that is not directly under discussion, pointing gestures tended to be smaller, lower in the gesture space, and were less likely to be accompanied by a shift in gaze or bodily orientation. Enfield et al. (2007) suggested that these less perceptually salient gestures provided additional background information, just in case the information was not actually established in the common ground. This interpretation aligns well with Barnes and Ebert’s focus on gradient at-issueness while also suggesting a possible social and communicative purpose for such gradience – gradient at-issueness may be conceived of as an active negotiation of what is present and what should be present in the common ground to achieve communicative goals.

3.2 Paying attention to speech

In addition to there being gestural strategies for drawing attention to gesture, there also appear to be gestural strategies for drawing attention to speech. New and focused information conveyed in speech has been shown to be reliably accompanied by eyebrow raises (e.g. Ebert et al. 2011), larger and more strongly articulated hand gestures (e.g. Gerwing and Bavelas 2004), and more frequent uses of abstract deictic gestures (e.g. McNeill et al. 1993). Beats, small vertical movements of the hands that can occur independently or overlaid on other hand gestures, have even been shown to shift perceived prosodic focus (Krahmer and Swerts 2007).

These correlations between gesture and focused, highly at-issue information in the spoken mode suggests that speakers have access to strategies across modes for directing attention to at-issue information. Rather than thinking of this as a shifting between modes, we can think of this as a joint multimodal effort to direct attention toward information that is most immediately relevant to communicative goals, regardless of the mode in which the information is conveyed.