At-issueness across modalities – are gestural components (more) at-issue in sign languages?

1 Introduction^[1]

In their target article, Barnes and Ebert give experimental evidence for the non-at-issueness of gestural enrichments in iconic co-speech gestures and ideophones and show that the phenomena of iconic enrichment provide evidence for a theory of graded at-issueness. Barnes and Ebert argue that in spoken languages “these phenomena, which deal with the interplay of ordinary descriptive – and arbitrary – meaning contributions on the one hand and depictive – and hence iconic – contributions on the other, are canonical cases of at-issue information interacting with non-at-issue information. Generally, in natural language, meaning is contributed by using language to describe what one intends to convey. Depictive enrichments add another layer. Since this meaning is of a different nature and often occurs simultaneously with what is being transmitted in the main channel (descriptively used speech), it brings in information of a different dimension, which is usually subordinated […] Standardly, verbal information then transmits the main, at-issue information part of an utterance, and gesture the non-at-issue pieces of information” (p. 176).

In this commentary, I argue that Barnes and Ebert’s account of idiomatic gestures and ideophones in spoken language offers also a promising basis for the analysis of gestural components of lexical items in sign languages. In addition, I address the issue of at-issueness and show that the at-issueness of gestural enrichments in sign languages is still open for discussion. In this context, I discuss two competing analyses. On the one hand, the account of Barnes and Ebert provides an interesting modality-independent explanation of the at-issueness of gestural enrichments in spoken and sign languages. According to this account, the at-issueness of iconic enrichments in spoken and sign languages can be derived from the same general principles. On the other hand, the competing account is based on the assumption “that sign languages can bring special insights into the foundations of semantics” (Schlenker 2018: 127). Hence, one might argue that depictive enrichments in sign languages, unlike depictive enrichments in spoken languages, do not add another layer since they are of the same nature as what is being transmitted in the main channel. Following this account, we can assume that iconicity in sign languages is fundamentally different from iconicity in spoken languages because of the gestural basis of sign languages and the special status of iconicity and iconic demonstrations in sign languages. Consequently, iconic enrichments can be assumed to be generally more at-issue in sign languages than in spoken languages.

The paper is organized as follows: In the next two sections, I first discuss two examples from German Sign Language (DGS) that can be compared to the iconic co-speech gestures and ideophones discussed in Barnes and Ebert’s target article and show that similar kinds of iconic enrichments are also attested in sign languages. In Section 2, I compare iconic co-speech gestures to different kinds of classifiers. In Section 3, I show that DGS has a special lexical class of expressive signs that can be – to a certain extent – compared to ideophones in spoken languages. I also show that iconic enrichments in classifiers and ideophone-like signs can be analyzed within the account developed by Barnes and Ebert for ideophones in spoken languages, that is, the analysis provides a general explanation of iconic enrichments in both modalities. In Section 4, I turn to the issue of modality and at-issueness and discuss two alternative argumentations that make conflicting predictions. I show that both explanations can make good arguments and argue that we need representative empirical studies to verify the conflicting predictions made by the two alternative explanations.

2 Iconic gestures and classifiers

The first example discussed in Barnes and Ebert is speech-accompanying iconic gestures that express size and shape properties of entities (see their discussion of example (9)). These iconic gestures correspond to size and shape specifiers (SASSes) in sign languages, which are used to depict (or trace) the size and/or shape of an entity (Zwitserlood 2012). Unlike iconic gestures in spoken languages, these SASSes cannot be temporally aligned with the nominal expression they specify since both, the noun and the SASS, are produced manually and thus use the same modality. Consequently, the noun modified by the SASS and the SASS have to be signed separately in a specific temporal order (i.e. depending on language-specific rules, the SASS either precedes or follows the noun). Moreover, SASSes function as adjectival modifiers that are syntactically highly integrated. Given these properties, we expect SASSes in sign languages to be more at-issue than iconic gestures in spoken languages. An interesting point is, however, whether the gestural demonstration of the actual size or shape of an entity (i.e. the actual movement of the hands that outline the size and/or shape) is a meaning component that is non-at-issue or at least less at-issue since this specific gestural demonstration cannot be part of a conventionalized form of the SASS. One can argue that SASSes have unmarked basic forms that are used to express specific shapes or sizes (e.g. round, rectangular, big, small, …). These forms can then be modified with additional gestural demonstrations to express more specific size and shape properties of the entity specified. Following this line of argumentation in the context of Barnes and Ebert’s analysis, the second meaning component, the non-conventionalized gestural demonstration, can be argued to be a non-at-issue component (or at least a meaning component which is less at-issue). We will come back to this distinction in a minute in the context of other kinds of classifiers and ideophone-like signs.

Sign languages do not only use classifiers to specify the size and shape of an entity but also classifiers that represent whole entities (so-called whole-entity classifiers, we-cl) and body-parts of entities (so-called body-part classifiers, bp-cl). These classifiers are anaphoric, i.e. they refer back to a prominent discourse referent, and they are typically used to express the movement or location of entities from different perspectives (Barberà and Quer 2018; Zwitserlood 2012). This is illustrated by the two images in Figure 1, which are taken from a DGS version of the fable ‘The lion and the mouse’ (Steinbach 2021, 2023).

Figure 1:

Two images of a running and climbing small animal (Steinbach 2023).

In the picture on the left the signer uses a two-handed body-part classifier to demonstrate the running action of the mouse from the character’s perspective (character viewpoint). This demonstration is accompanied by a gestural facial expression and a change in body posture to depict the movement of the mouse. In the picture on the right, the signer changes perspective (observer viewpoint) and uses a whole-entity classifier (dominant left hand) to trace the movement of the mouse, who is climbing over the head of the lion. Interestingly, the non-dominant right hand still produces the body-part classifier used in the left picture to mark topic continuity.

Body-part and whole-entity classifiers combine systematically conventionalized linguistic features with gestural demonstrations. The handshapes of both kinds of classifiers are lexically specified categorial features and can be subject to typological variation (Perniss et al. 2007: 9–10). By contrast, the movement and location of the hand(s) cannot be part of the lexically specified phonological form of the classifier. Since these features specify the movement of the entity in the real or fictional world (in our case the movement of the mouse who climbs over the head of the lion), they cannot be part of the lexical entry but must be part of a gestural demonstration (for more discussion, see Brentari et al. 2012; Cormier et al. 2012; Goldin-Meadow and Brentari 2017; Schembri et al. 2005, for a discussion of the morphosyntax of classifiers, see Benedicto and Brentari 2004; Zwitserlood 2012).

This distinction is also reflected in the glosses of the two classifiers: The small caps (bp-cl _SMALL-ANIMAL and we-cl _SMALL-ANIMAL ) reflect the linguistic (conventionalized) part of the classifier and specify the semantic class of the entity referred to (here ‘small animal’). The gestural demonstration is indicated in italics (run and climb) with verbs that describe the action performed by the mouse. Following Barnes and Ebert’s analysis of ideophones, Davidson’s analysis of role shift and Maier’s theory of unquotation in action role shift in sign languages (Davidson 2015; Maier 2018; Steinbach 2023), the semantic representation of the whole-entity classifier (produced with the dominant right hand) in the right picture in Figure 1 can be analyzed as shown in (1) (ignoring the fact that the classifier is an anaphoric expression, which refers back to the mouse in this context):

(1)

[e] ∧ move p (e) ∧ [x] ∧ agent(e, x) ∧ form(e, ‘we-cl SMALL-ANIMAL ’) ∧

[d] ∧ d = d ∧ sim p* (e,d )

The gestural demonstration uses a linguistic element, the whole-entity classifier for small animals, which is unquoted in (1) (i.e. ‘form(e, ‘we-cl _SMALL-ANIMAL ’)’), and combines it with a simultaneous gestural demonstration d that adds movement and location features to the classifier. These features then depict the movement of the mouse over the head of the lion. Body-part and whole-entity classifiers are thus an even more obvious example for linguistic items used in sign languages that combine categorial linguistic features with gestural iconic features than SASSes, which are more similar to iconic gestures used in spoken languages.

The discussion reveals that all three kinds of classifiers differ from iconic speech-accompanying gestures used in spoken languages. Classifiers are syntactically integrated and have specific adjectival or verbal functions. At the same time, they involve gestural demonstrations of the size, shape, location or movement of an entity. In this respect, they correspond to iconic co-speech gestures. However, with body-part and whole-entity classifiers, the gestural demonstration is produced simultaneous and in the same modality. It is thus not a coincidence that the analysis of these two kinds of classifiers is based on Barnes and Ebert’s analysis of ideophones. They do not only share several properties with iconic gestures, but also with ideophones, which we discuss in the next section before we turn to the main question of this commentary, the impact of modality on the at-issueness of gestural demonstrations in spoken and sign languages.

3 Ideophones and idiomatic signs

The second type of iconic meaning in spoken languages discussed by Barnes and Ebert are ideophones, which have been defined as expressive items that form “an open lexical class of marked words that depict sensory imagery” (Dingemanse 2019: 16, see Barnes and Ebert, and Dingemanse 2012 for a discussion of the key properties of ideophones). In addition to the properties discussed in Dingemanse, Barnes and Ebert show that ideophones involve a depictive demonstration which contributes non-at-issue content or content which is at least less at-issue (see also Barnes et al. 2022).

While iconic gestural components can be easily identified also in sign languages as discussed in the previous section using the example of different kinds of classifiers, counterparts of conventionalized iconic lexical expressions such as ideophones are more difficult to identify in sign languages. The main reason for this difficulty is that sign languages have a gestural (iconic) basis which is still visible in the grammar and lexicon of sign languages (Aronoff et al. 2005; Goldin-Meadow and Brentari 2017; Pfau and Steinbach 2011; Strickland et al. 2015; Van Loon et al. 2014). As a consequence, iconicity is a rich resource in sign languages which has an impact on all levels of the linguistic system, including the lexicon (Meier 2012).

Most importantly in this context, many lexical signs are to a certain degree iconic in many if not all sign languages, that is, unlike in spoken languages, the form-meaning relation is to a much lesser extent arbitrary (Perniss et al. 2010; Taub 2012). In a norming study with 313 DGS signs, deaf native signers assigned most signs a very high iconic value (between 4 and 7 on a scale from 1 to 7, Trettenbrein et al. 2021). Figure 2 gives an example for a DGS sign with high iconicity (book, 7), with medium iconicity (awful, 3,8) and with low iconicity (boy, 2,8). In the norming study, almost all signs received a higher iconicity rating than awful and were thus rated as very iconic (for a discussion of the relation between iconicity and transparency, see Trettenbrein et al. 2021).

Figure 2:

Three still images of signs that received different iconicity ratings (Trettenbrein et al. 2021).

However, unlike the iconicity of co-speech gestures and ideophones in spoken languages, the (high) iconicity of signs is semantically not relevant. The highly iconic sign book in DGS, for example, denotes a set of books but does not iconically depict any context-dependent specific size, shape or handling features of the corresponding set of entities or a specific book referred to in a context. This means that the iconic properties of conventionalized lexical signs such as book are not relevant semantic features that enter the semantic representation on the non-at-issue level as gestural demonstrations: “Iconicity seems to be ‘optional’ in daily language use of lexical signs” and “seems to play no role in acquisition, recall, or recognition of fixed lexical signs in daily use” (Taub 2012: 405, 408).

Unfortunately, the observation that many lexical signs have iconic properties makes the identification of signs that correspond to ideophones in spoken languages difficult since the key property of ideophones, i.e. the depiction of sensory imagery, cannot be a sufficient criterion in sign languages that distinguishes unmarked conventional lexical signs from marked expressive items such as ideophones. This does, however, not mean that sign languages do not have corresponding marked expressive items that can be compared to ideophones and analyzed as the visual counterparts of ideophones.

Interestingly, there is a special class of signs that has been discussed especially in the context of sign language teaching and in sign language communities as an important and indispensable part of competent everyday communication (similar to idioms in spoken languages). In the few publications on this topic, these signs are called ‘multi-channel signs’, ‘special signs’, ‘Spezialgebärden’, ‘polyseme’, ‘Rede-/Gebärdenwendungen’ and ‘(signed) idioms’ (Brennan 1992; Konrad 2011, 2014; Schütte 2014; Wrobel 2017). These ‘special’ signs share some properties with ideophones in spoken languages and can be – to some extent – analysed as their sign language counterparts. In the following discussion, I will use the term ‘idiomatic sign’, which is especially used by the deaf community in teaching contexts. ‘Idiomatic sign’ is meant to denote an open lexical class of marked signs that have a figurative expressive meaning component.

The terminology used by teachers and linguists already expresses that these idiomatic signs – like ideophones in spoken languages – are somehow special (‘special signs’). In addition, they systematically combine manual and nonmanual features (‘multi-channel signs’) and the meaning of idiomatic signs is flexible and context dependent (‘polyseme’, ‘idioms’). It is thus not a coincidence that idiomatic signs are usually translated with phrasal figurative expressions into spoken languages (‘Rede-/Gebärdenwendungen’).

The DGS dictionary (Kestner 2021) lists 92 idiomatic signs (‘allgemeine Idiome/Redewendungen’) and the DGS corpus (Konrad et al. 2020) 250 idiomatic signs (‘Spezialgebärden’) with more than 4.000 tokens. These two compilations may contain signs that do not belong to the class of idiomatic signs as defined here. There is, however, some informal agreement on the core group of idiomatic signs in the sign language community and the dictionary and corpus teams. Figures 3 and 4 show six examples of idiomatic signs in DGS that are representative for this special lexical class.

Figure 3:

Three still images of idiomatic signs in Kestner (2021).

Figure 4:

Three still images of idiomatic signs in Finkbeiner and Meister (2019, 2022.

Idiomatic signs can thus be defined as an open lexical class of ‘special’ signs which are to some degree conventionalized within a sign language community (and subject to sociolinguistic variation, Thomas Finkbeiner, p.c.). In addition, idiomatic signs are marked items with a strong expressive meaning component. The examples in Figures 3 and 4 illustrate a typical property of all idiomatic signs, i.e. the combination of iconic features which are expressed manually as well as nonmanually. Especially the nonmanual markers (facial expression) contribute an important expressive meaning component. As opposed to conventional lexical signs that may also include manual and nonmanual iconic features as illustrated in Figure 2 above, the marked expressive features are an integral part of the meaning of idiomatic signs, that is, the figurative gestural demonstration enters the semantic representation of these signs (for lexical nonmanuals, see Pendzich 2020). Therefore, an idiomatic sign cannot be simply replaced by a (sequence of) semantically equivalent conventional sign(s) without a loss of meaning and expressive power. Not surprisingly, the meaning of idiomatic signs is usually paraphrased with different phrasal figurative expressions.

Because of these similarities between ideophones in spoken languages and idiomatic signs in sign languages, Ebert and Steinbach (2023) argue that idiomatic signs, like ideophones, have two meaning components: (i) the conventionalized meaning of the lexical item and (ii) the expressive meaning of the gestural (iconic) demonstration. The second meaning component is what makes idiomatic signs special. Note that with conventionalized lexical signs, iconic features do not enter the semantic representation. The iconic properties of book might be the gestural basis of the emergence of this sign and signers are still aware of the iconic features (which are still visible in the phonological form of the sign) when asked to rate the iconicity of this sign. However, these features are irrelevant for the semantic interpretation: The sign book is associated with the concept BOOK and denotes a set of books without any special expressive component or without an iconic depiction of size, shape and handling features of specific books.

Following this line of argumentation, both modalities obviously have an open class of conventionalized marked expressions that combine a descriptive with a depictive expressive meaning component. As opposed to ideophones in spoken languages, the iconic features are, however, not sufficient indicators for this special class of signs in sign languages. Nevertheless, the specific expressive status of the depictive features and the figurative meaning contributed by these features distinguishes conventionalized lexical signs from idiomatic signs. Only idiomatic signs involve an iconic demonstration that triggers a depictive enrichment. Following Barnes and Ebert’s analysis of ideophones, the semantic representation of the second idiomatic sign in Figure 3 would look as follows:

(2)

[e] ∧ be-surprised p (e) ∧ [x] ∧ experiencer(e,x) ∧ [d] ∧ d = d ∧ sim p* (e,d )

The conventional meaning of the idiomatic sign in (2) is that a person x is surprised. In addition to this basic meaning component, the idiomatic sign has an expressive meaning component which adds the iconic demonstration of a surprised person, i.e. a person whose jaw is dropping. This means that we can give a modality-independent analysis for marked expressive words (i.e. ideophones) and marked expressive signs (i.e. idiomatic signs). Note finally, that as opposed to the semantic representation of classifiers in (1), the semantic representation of idiomatic expressions in (2) does not contain an unquoted linguistic item which is combined with or modified by a context-dependent gestural demonstration. Idiomatic signs are fully conventionalized gestural depictions with a descriptive meaning component.

4 Modality and at-issueness

The discussion of classifiers and idiomatic signs in DGS has shown that gestural components of utterances can be accounted for with modality-independent analyses of gestural demonstrations. The same tools used by Barnes and Ebert for iconic co-speech gestures and ideophones in spoken languages can also be used to account for the phenomena discussed in the previous sections. The most important question of this commentary is now whether the second (depictive) meaning component of classifiers and idiomatic signs is also non-at-issue or at least less at-issue in the visual-gestural modality (as has been shown for iconic gestures and ideophones in spoken languages) or whether we find a general difference between the two modalities in this area. The following discussion will reveal that both options are plausible answers to this question and that experimental data are necessary to decide which alternative is on the right track.

Recall that the experiments discussed by Barnes and Ebert show on the one hand that gestural demonstrations are less at-issue than corresponding linguistic descriptions. On the other hand, the experiments also show that iconic enrichments in ideophones seem to be more at-issue than iconic enrichments in speech accompanying gestures because the former occur in the same modality, i.e. ideophones are words (and thus part of the linguistic system) with different degrees of syntactic and prosodic integration in the utterance. Note that the degree of integration corresponds to the degree of expressiveness: Less integrated ideophones are more expressive and consequently less at-issue (Dingemanse and Akita 2017).

Table 1 illustrates some similarities and differences between the different kinds of iconic enrichments in spoken and sign languages discussed in this commentary. The table compares the two examples discussed by Barnes and Ebert (iconic gestures and ideophones) with the two examples discussed in Sections 2 and 3 (classifiers and idiomatic signs). The first property concerns the conventionalization of the iconic demonstration. While ideophones and idiomatic signs involve a fully conventionalized iconic component, the iconic components of classifiers (movement and location features) are not conventionalized (unlike the handshape features). Likewise, iconic gestures do not have a conventionalized iconic component. The second property refers to the modality of the iconic demonstration, i.e. whether it uses the same modality as the linguistic description or whether it is produced in a different modality. The third property specifies the syntactic integration of the corresponding item. While iconic co-speech gestures are obviously not integrated, classifiers, ideophones and idiomatic signs seem to be an integral part of the syntactic structure as adverbial or adjectival modifiers, predicates or fully propositional items. The last property states whether the iconic demonstration appears independent of linguistic items (irrespective of temporal alignment) or whether it is typically combined with a linguistic item (word, sign, handshape).

The table illustrates on the one hand that iconic co-speech gestures clearly differ from classifiers, ideophones and idiomatic signs. On the other hand, the table also shows that classifiers differ from ideophones and idiomatic signs in the degree of conventionalization. The iconic features of classifiers that express the size, shape, movement or location of an entity are not conventionalized. By contrast, ideophones and idiomatic signs have conventionalized iconic features which are specified in the lexicon.

Following the argumentation of Barnes and Ebert, we expect that the iconic enrichment in idiomatic signs corresponds to the iconic enrichment in ideophones. The iconic enrichment is thus expected to show a higher degree of at-issueness than the iconic enrichment in co-speech gestures. It uses the same modality as the linguistic description and it is internal. Moreover, the idiomatic sign that triggers the iconic enrichment is usually integrated to a certain degree into the utterance (either as predicate or propositional expression). In addition, the gestural demonstration of idiomatic signs is a conventionalized part of the sign which is specified in the lexical entry of each idiomatic sign (and only subject to modest modifications which, for example, might be used to express different degrees of surprise).

By contrast, the degree of at-issueness of the iconic enrichment of classifiers is expected to be somewhere between the degree of at-issueness of the iconic enrichments of idiomatic signs and iconic gestures. Comparing classifiers to idiomatic signs, classifiers are expected to be less at-issue since the idiomatic demonstration is not conventionalized. Like iconic co-speech gestures, classifiers can be produced with a basically endless variety of trajectories and locations to depict the size, shape, movement or location of an entity. Comparing classifiers to iconic gestures, the higher degree of syntactic integration (either as an adjectival modifier or as a predicate), the dependence on a linguistic component (the conventionalized classifier handshape) and the use of the same modality suggest that classifiers are more at-issue than iconic co-speech gestures. This argumentation presupposes that iconic features enter the semantic representation in the same or at least in a similar way in both modalities based on modality-independent principles. This is a big conceptual advantage of Barnes and Ebert’s approach.

An alternative line of argumentation is based on recent formal semantic analyses of iconic features in sign languages which argue that iconicity in sign languages makes a different contribution to the semantic representation than in spoken languages. Kuhn and Aristodemo (2017: 38) argue, for example, “that there is no fundamental opposition between iconic properties and formal properties; there’s no problem in allowing an iconically defined predicate to be incorporated directly into a logical definition.” Similarly, Schlenker (2018) argues that sign languages are more expressive than spoken languages because iconicity systematically interacts with the logical core of language: “Sign languages make use of expressions that simultaneously have a logical/grammatical function and an iconic semantics, defined as a semantics in which some geometric properties of signs must be preserved by the interpretation function” (Schlenker 2018: 129). Consequently, in recent formal semantic analyses, many iconic phenomena have been directly implemented in the at-issue-dimension of the semantic representation (Aristodemo and Geraci 2018; Kuhn and Aristodemo 2017; Pfau and Steinbach 2016; Schlenker et al. 2013; Steinbach and Onea 2016; Wilbur 2003, among others).

Following this line of argumentation, iconic features are expected to provide (at least more likely) at-issue information in sign languages than in spoken languages. The difference in the at-issueness of iconic components in iconic co-speech gestures, on the one hand, and ideophones, on the other, seems to provide some evidence that modality matters. Even in spoken languages, the status of iconic components seems to depend on the channel which is used for the production of the iconic component (i.e. visual vs. auditory). Iconic demonstrations that are produced in the same modality as linguistic descriptions seem to be more at-issue. Since sign languages use the same modality as manual and nonmanual iconic demonstrations and since in sign languages, iconic demonstrations generally depend on and strongly interact with linguistic descriptions, as has been shown in this commentary by the examples of classifiers and idiomatic signs, iconic demonstrations might be generally more at-issue in sign languages than in spoken languages.

So far, both explanations sound plausible. A modality-independent analysis of iconic enrichment combined with a theory of graded at-issueness is as convincing as a modality-specific explanation of the special status of (visual) iconicity and iconic demonstrations in sign languages. Therefore, representative empirical studies on the at-issueness of iconic enrichments in different kinds of classifiers and idiomatic signs are necessary to verify the opposing predictions made by the two alternative explanations discussed in this section. Obviously, the differences between the different degrees of (a graded) at-issueness require very precise experimental methods. Otherwise, the subtle differences in the degree of at-issueness of different kinds of iconic demonstrations in sign languages will be lost in the noise of the data and the basic question of this commentary would remain unanswered.