A cross-linguistic comparison of reference across five signed languages

1.1 Language ecologies and semiotic strategies in signed language interaction

How do these signers, from five different signed language communities, establish and maintain reference as part of their communication with others? To answer this question, we first take the position that a wider spectrum of communicative practices available to signers must be taken into account, not just fully lexicalized forms. It is well-established that signers (and speakers) recruit a semiotically diverse set of strategies for meaning-making, and for signers this partially manifests as different types of manual and non-manual signs. Along with physical artifacts, we have at our disposal the coordinated use of multiple bodily articulators which we use to produce multimodal actions (Enfield 2009; Goodwin 2000; Kusters et al. 2017; Mondada 2007). Importantly, these multimodal actions are situated both within particular instances of use as well as in wider communicative ecologies, with particular temporal, spatial, physical, and socio-historical characteristics: under these conditions, referencing emerges (Goodwin 2000; Haugen 1972; Keevallik 2018).

Communicative ecologies include not only the physical places where signers and speakers exist and interact, but they also reflect the history of interactions between people and their environment. They are also continuously shaped by future interactions (Hodge 2014; see also Linell 2009). People create meanings with each other within these environments, recruiting the whole range of resources available to them, i.e., their “semiotic repertoire” (Kusters et al. 2017). Semiotic repertoires may be shared across a community, but they are also individual, reflecting the unique trajectories of interaction each person has over their lifetime. These micro- and macro-patterns of meaningful interaction intersect, as individual encounters shape larger scale practices and vice versa (Agha 2005: 12; Barth et al. 2021; Hodge and Goswell 2021). It is this complex diversity within and across language ecologies that partially motivates the use of multivariate statistical methods in the current study, details of which are discussed below.

The types of signs available to signers vary in their degree of conventionalization and semiotic composition. In the following paragraphs, we introduce these various signs and characterize them against a backdrop of neo-Peircean semiotics (Peirce 1955; see also Enfield 2009; Frick 2014; Kockelman 2005; Mittelberg 2019; Parmentier 1994). In particular, we draw on Clark’s (1996) proposal that language use is essentially a system for signaling symbols, indices, and icons through acts of describing, indicating and depicting, respectively (see Ferrara and Hodge 2018). From this perspective, signed language signs are considered polysemiotic ensembles created through a combination of these three signaling methods (see e.g., Capirci et al. 2022; Johnston 2013, Ferrara and Hodge 2018; Hodge and Ferrara 2022). In this way, all signed language signs exhibit symbolic, indexical, and iconic properties to varying degrees, and these properties all contribute to how signs are used to achieve reference. Below, we introduce and illustrate the referential nature of these signs using example utterances from the five signed languages involved in this study. More details are also provided in Section 2, but the reader is also directed to more in-depth discussions of signed language signs provided by Johnston and Schembri (1999, 2010.

1.1.1 Fully lexical signs

We begin with fully lexical signs, which are highly conventionalized form-meaning pairs. These signs are symbols in that it is “pre-agreed” that X means Y, while also exhibiting indexical and depictive qualities (albeit to different degrees). Lexical signs are the closest equivalent to the words of spoken languages. Signers can use lexical signs or phrases to name a referent. Consider the utterance in Figure 1 from Auslan, the main deaf community signed language used in Australia.^[1] Here the signer produces lexical signs to refer to the referents boy and rock in the story (see the images of signs outlined in black and glossed in English as boy and rock). Lexical noun signs like these, alone or as part of larger lexical phrases, are a common strategy that signers use to identify referents.

Figure 1:

An utterance in Auslan that makes reference to boy and rock via lexical signs, co-produced with mouthed English words (Johnston 2008; SSNA2c7a.mp4; 00:01:54.5-00:01:56.75).

One type of lexical sign functions somewhat differently to the noun signs mentioned above and is considered separately here. These signs, called indicating signs, are often analyzed as verbs, and signers can change the direction of movement of these signs in the signing space to index arguments. In this way, they also effectively index the referents of these arguments. Referencing with indicating signs is illustrated with an utterance in Finnish Sign Language in Figure 2. The signer is explaining that the boy from the story she is retelling finds his lost frog hiding out with a family of baby frogs. At the end of the story, the boy then takes or is given one of the babies to take home with him. In this particular utterance the signer produces the indicating sign give two times. Signers can change the beginning and end positions of this verb to indicate its arguments. The sign here begins, each time, out in the signing space and then moves towards the signer’s body. This directional movement towards the signer indicates that the ‘boy received a frog’ (as the signer is used as the locus for referent boy). In other instances, we may observe signers moving the sign from their body towards another area of space to indicate that the referent at the signer’s locus (including the signer themselves) gives something to a recipient. Again, signers interpret the start and end positions of these signs (according to convention) as indications of the sign’s arguments, and thus referents in the discourse. This referential work often succeeds whether or not the sign occurs in an utterance with additional lexical signs that name the referents explicitly (as we see in Figure 2).

Figure 2:

An utterance in Finnish Sign Language that refers to the referent boy via an indicating sign (University of Jyväskylä, Sign Language Centre 2019; CFINSL2014_020_05_CAM3_0.mp4; 00:07:02.160-00:07:04.160).

1.1.2 Fingerspelling and mouthing spoken language words

In addition to these lexicalized signs, the current study also considers two forms of language contact: fingerspelling and mouthing words from spoken language. Mouthing is the voiceless articulation of spoken language words that produces a visual signal that captures some aspects of the spoken form (Quinto-Pozos and Adam 2015). Fingerspelling is the practice of spelling out words from a spoken/written language using a set of signs that index individual letters. These forms are considered here to be instances of describing (because they are symbols), albeit derived from an ambient spoken language, and they also index these words and depict some aspects of their form.

We saw an example of a signer using mouthed English words to refer in Auslan in Figure 1. As the signer produced the Auslan lexical sign boy, he also mouthed the English word boy. This mouthed word, a conventionalized, lexical word in English, effectively names the referent—similarly to the co-produced Auslan sign. Then as the signer produces the phrase rock ds:round-rock, he also produces a mouthing of the English word rock, again effectively naming the referent using the ambient spoken language, while simultaneously depicting and describing the referent in Auslan. As we will see later, mouthings are often co-produced with manual lexical signs in referring expressions across the five signed languages investigated here.

Fingerspelling can also be used to identify referents. In Figure 3 an Irish signer refers to the dog from the story he is retelling by fingerspelling the English word dog, produced as a series of three handshapes representing the letters d, o, and g (see the first three images from the left in Figure 3). We can note that all five signed languages investigated here have fingerspelling systems, but the conventions around the use (frequency, distribution) of this practice vary across languages and individuals.

Figure 3:

An utterance in Irish Sign Language that refers to the referent dog via a fingerspelled English word (Signs of Ireland Corpus 2004; 17-Kevin(Dublin)-Frog Story.mov; 00:00:44.93-00:00:44.45.78).

1.1.3 Depicting signs

In the preceding paragraphs, several types of conventionalized signs along with two strategies for expressing spoken language words in order to refer were introduced. Now, we move on to survey the less conventionalized types of signs observed across many signed languages, including the five signed languages investigated here. One type of sign that is only partly lexicalized and available to signers is depicting signs. As the name suggests, these signs exploit the semiotic mode of depicting to iconically profile aspects of their meanings (although, as with all the other types of signs, depicting signs may also describe and indicate meaning to varying degrees). Depicting signs are produced with the hands in the space in front of the signer. They often provide a scaled-down or zoomed-out view of objects and events. They are used to demonstrate how referents move, the size and shape of referents, or the location of referents (Bergman and Wallin 2003; Johnston and Schembri 2007; Liddell 2003). Some researchers have compared the similarities between depicting signs to the iconic hand gestures produced by speakers while talking (Kendon 2004; Schembri et al. 2005).

As these signs, particularly the handshapes of these signs, depict referents, they are considered here to also contribute to reference (see also Hodge et al. 2019; Pizzuto et al. 2008). We see how these signs do this with an example from Norwegian Sign Language in Figure 4. The signer explains that the frog from the story she is retelling was inside a jar. She introduces this new referent jar via a lexical adjective glass and then uses her hands to trace the outline of an object that is cylindrical in shape and about the height of a person’s torso (see the image outlined in black and glossed as ds:shape-jar in Figure 4). We understand this depicting sign to identify the container that the frog was in, understanding the referent to be a glass jar.

Figure 4:

An utterance in Norwegian Sign Language that refers to the referent jar via a depicting sign (Ferrara in prep.; KNTS_B_CNT_Frosk.mp4; 00:00:09.913-00:00:11.226).

1.1.4 Pointing and indexing actions

Finger pointing, which is one way that signers index locations and referents, is another type of partly lexical sign. Finger pointing directs attention towards a meaning or referent by being spatially or temporally connected to it. Finger pointing also describes and often depicts in various instances, through both conventional characteristics and an ability to create perceptual and structural resemblances (e.g., the sign glossed pt:path in Figure 4, which is a point that iconically traces the path to locate the frog sitting in a jar). An example of a finger point that identifies a referent is presented in Figure 5 (see the image outlined in black and glossed as pt). Here a Swedish signer points to his right to indicate the referent frog. This part of the signing space had been linked to the frog now for several utterances. The signer goes on to explain that the frog was with his frog family (the signs of which are also oriented towards the signer’s right side, further underscoring that the signer is talking about the frog).

Figure 5:

An utterance in Swedish Sign Language that refers to the referent frog via a finger point (Mesch et al. 2012; SSLC02_331_S033_b.mp4; 00:02:28.36-00:02:31.10).

While finger pointing is considered a prototypical type of indication, signers also point with other bodily articulators, e.g., eye gaze, body shifts, and through signs being placed or directed to locations in the signing space (as mentioned above with indicating signs). Both manual pointing (described in the previous paragraph) and non-manual pointing contribute to reference. Of particular note are the various ways signers index referents with their hands, face, and body to signal the presence of “invisible referents” in the signing space. These indexing actions prompt interlocutors to conceptualize referents within portions of the signing space, as if they were physically present. These activated spaces, sometimes called invisible surrogates (see Liddell 2003 for more on this concept), often manifest as part of complex, visibly constructed actions (see below). The spaces themselves are not a type of sign per se, but they are the result of a confluence of indicating actions. They are considered here in this study because signers create invisible surrogates and these spaces effectively reference otherwise non-present referents. To illustrate invisible surrogates and their referential import, we continue on from the utterance first presented in Figure 4, where the Norwegian signer explains that the frog is in the glass jar. Immediately following that utterance, she produces the utterance provided in Figure 6. Here her body, eye gaze and face are all directed to an area of space in front of her stomach. These indexing actions prompt interlocutors to conceptualize the frog in the jar in this location, and in this way, the signer refers to the frog and the jar (illustrated in Figure 6 by the white line drawings on the images).

Figure 6:

An utterance in Norwegian Sign Language that refers to the referent boy via constructed action and to the referents frog and jar via indexing actions that create an invisible surrogate (Ferrara in prep.; KNTS_B_CNT_Frosk.mp4; 00:00:11.28-00:00:12.20).

1.2 Referencing

In signed languages, reference, generally thought to be fundamental to our understanding of language (Levinson 2006), has been extensively investigated using narrative retellings and other data elicited from signers of a range of signed languages (e.g., Ahlgren and Bergman 1994; Barberà 2015; Bergman and Wallin 2003; Engberg-Pedersen 1993, 2003; Garcia and Sallandre 2020; Neidle and Nash 2012; Perniss and Özyürek 2015; Pizzuto et al. 2008; Swabey 2002; Winston 1991). Many of these studies show (1) that, cross-linguistically, signers often associate referents with locations in the signing space, e.g., by combining lexical noun signs and pointing within a noun phrase, and (2) that anaphoric reference can be maintained through the placement or direction of manual signs, such as verbs and pointing, towards these locations to refer to these referents. These studies also observed signers enacting referents with their bodies and depicting referents with their hands (Ahlgren and Bergman 1994; Frederiksen and Mayberry 2016; Garcia and Sallandre 2020; Swabey 2002; Pizzuto et al. 2008).

While each of these studies contributes important knowledge about how signers do reference in often similar ways across signed languages, there are limitations. Many studies have focused on only a selection of possible sign types or have conflated different sign types together (Barberá and Zwets 2013; Kibrik and Prozorova 2007; Perniss and Özyürek 2015; Pizzuto et al. 2008). Secondly, most studies to date have been qualitative and/or have provided only minimal descriptive statistics on a small dataset, or in some cases the data analyzed are not explicated in any depth (Barberá and Zwets 2013; Barberà and Quer 2018; Garcia and Sallandre 2020; Pizzuto et al. 2008; Swabey 2002, 2011). These types of analyses have led to more or less categorical statements about how referencing is done in signed languages. These characterizations are often then applied cross-linguistically without further empirical description to test the validity of such applications. One justification of this has been due to the general assumption that signed languages are all very similar due to iconicity and the use of space (Johnston 1991; Meier 2002; Talmy 2003; cf. Perniss 2012; Perniss and Özyürek 2008).

Recent work however has engaged different methods to reveal more detailed findings that include a wider range of sign types and which undertake more complex statistical analyses (e.g., Fredriksen and Mayberry 2016; Hodge et al. 2019). Hodge, Ferrara and Anible (2019) used mixed-effects modeling and clustering analysis to investigate how signers do reference using a wide range of sign types in Auslan. Their clustering analysis revealed, for example, that Auslan signers tend to refer to maintained animate referents using constructed action and invisible surrogates, while inanimate maintained referents were often expressed with depicting signs and invisible surrogates. These tendencies were contextualized by variation across utterances and signers. The current study continues with this approach to investigate the referencing practices of a similar cohort of deaf signers across five Western signed language ecologies.

The studies mentioned above highlight the strong relationship between cognitive accessibility and degree of conventionality of referential forms. Signers tend to produce longer referring expressions that include more conventionalized forms (such as lexical signs with fingerspelled and/or mouthed words) more often with new and re-introduced referents, while shorter expressions with less conventionalized forms (such as enactments and pointing actions) are used to express maintained referents (see e.g., Frederiksen and Mayberry 2016; Hodge et al. 2019; Pizzuto et al. 2008; Swabey 2002, 2011).

These findings reflect the earlier canon of reference in spoken languages, which found that referring expressions are shaped by the activation status of the referent (new vs. reintroduced vs. maintained) (e.g., Ariel 1991; Chafe 1976; Gundel et al. 1993; Silverstein 1976/2016). Activation status, or cognitive status, is based on the “assumptions that a cooperative speaker can reasonably make regarding the addressee’s knowledge and attention state in the particular context in which the expression is used” (Gundel et al. 1993: 275). Cross-linguistic evidence from spoken languages has shown that referents with the highest cognitive accessibility are often expressed with minimal phonetic form (Ariel 1991; Givón 1983; Gundel et al. 1993; Levinson 1987). However, it should be noted that much of this work has focused almost solely on specific types of referring expressions (in speech or writing), namely full noun phrases, pronouns, and zero anaphora, with only some recent studies considering the wider semiotic repertoire of speakers interacting face-to-face (e.g., Perniss and Özyürek 2015). One aim of the current paper is to show the advantages of widening the scope of linguistic typology to include less conventionalized forms that primarily index and depict meaning.

In addition to activation status, there is also evidence that animacy affects choice of referential expressions cross-linguistically (Dahl and Fraurud 1996). Details regarding granularity and classification of animacy are debated however (Dahl 2000; DeLancey 1981; Fraurud 1996; Silverstein 1976). Fraurud’s (1996) cognitive ontology of animacy is one model, based on the idea of individuation, which emphasizes that referents can be construed in different ways depending on the context. For example, as a referent becomes more known in the discourse it becomes more individuated (i.e., it becomes an Individual, see Section 2.2) and thus more susceptible to naming (motivated by the need to be able to identify recurrent referents, Fraurud 1996: 82). We found this model to be well suited to accommodate the data investigated for this study, which is based on narratives elicited from a children’s story book, where three referents—a boy, a dog and a frog—are the main characters. We begin with the three broad categories Human, Animate, and Inanimate, but as will be shown in Section 2.2, we assigned referents animacy categories according to the functional use of the referents in the context of a children’s story.

In this study, we use corpus methods to analyze how fluent signers of five Western signed languages do reference, and how referring expressions are influenced by factors of animacy and activation status. Direct comparison of data from five signed languages will help us to determine if similarities and differences are language-specific, modality-specific, due to social factors that have been shown to influence the heavily conventionalized aspects of language use, and/or whether they reflect more widespread and potentially universal patterns of face-to-face signed communication.

2.1 Data and participants

The data for this study^[2] come from corpora of five deaf community signed languages (SLs): Auslan (Johnston, 2008), Irish Sign Language (ISL) (Signs of Ireland Corpus 2004), Finnish Sign Language (FinSL) (University of Jyväskylä, Sign Language Centre 2016, 2019), Norwegian Sign Language (Ferrara and Bø 2015; Ferrara and Halvorsen 2021; Ferrara in prep), and Swedish Sign Language (Mesch et al. 2012). The data comprises video-recorded retellings of the children’s picture book Frog, Where Are You? (Mayer 1969) produced by 10 signers of each language. The data is annotated in ELAN^[3] (Crasborn and Sloetjes 2008). The overall duration of the data investigated is 2:11:51, with the longest subset coming from the FinSL data (0:31:37) and the shortest from Swedish SL data (0:21:47). The mean duration of all the retellings is 0:02:35 min (SD = 0:00:58). Table 3 in Section 2.3 provides details about the duration characteristics of the study corpus, which were calculated by totaling the durations of all composite utterances in the datasets.

The deaf signers represented in the study corpus were not strictly controlled for sociolinguistic variables. This was due to the small participant size of the study corpus, and the main aim of conducting a preliminary quantitative cross-linguistic investigation. However, the participants are balanced for gender across the languages. The mean age of all signers is 44.52 years old (SD = 5.21). All signers typically acquired their respective signed language early in childhood, but their exact age of signed language acquisition is not comparable across corpora, because different corpora used different cut-off points for early signed language acquisition. Even so, all signers can be characterized as fluent signed language users, and all prefer to use their respective signed language as their primary language. All signers are bilingual or multilingual with some other signed, spoken and/or written language(s), including the ambient majority language(s).

2.2 Annotation

The annotation method used for the current study, briefly summarized in this section, is taken from the earlier study of reference in Auslan (Hodge et al. 2019). Please refer to Hodge et al. (2019) and the OSF deposit of the current study for full details of the annotation method.

The data analyzed here had been previously annotated for manual sign tokens to varying degrees. These annotations were helpful in creating the annotations for the current study, although they were not directly used in the analysis. Using these annotations as a starting point, the signed retellings were further annotated for composite utterances, referring expressions, and the finer details of the semiotic composition of all identified referring expressions.

Composite utterances were annotated following the guidelines for annotating clause-like units outlined in the Auslan Corpus Annotation Guidelines (Johnston 2019: Section 2.2.2). These utterance level units usually contain multiple signs that are held together via articulatory, semantic, and structural features. Composite utterances are multimodal and contain different types of signs. They generally predicate something about the world and move an interaction forward (Enfield 2009). These utterances were the basis on which referring expressions were analyzed and annotated for activation status and semiotic composition detailed in the following paragraphs.

To investigate the potential effects of animacy on referring expressions, we categorized the 12 referents examined for the current study as human, animal (non-human animate), and inanimate (see Table 1). We did this by taking into consideration the nature of ontological categories described by Fraurud (1996) and the context of the narrative re-telling investigated here. This led us to “re-categorize” two of the referents (the dog and the frog) as “human,” given their role as main characters whose individual identity over time is important for the story, and thus align with Fraurud’s (1996) Individuals.

Annotations identifying these referents were created on a Narrative Referent tier when explicitly referred to with visible bodily actions. In many cases, referring expressions span multiple signs or other non-manual behavior within a single composite utterance. All of these were collected into a single referent annotation. Three copies of this tier were created to accommodate multiple referents being expressed at the same time. In addition to annotations on the Narrative Referent tiers being coded for referent (as listed in Table 1), further tags were added to indicate the activation status of the token referent.^[4] The suffix -NEW was added to referent tags if it was the first time the referent was mentioned in the story. If a referent had been identified also in the immediately preceding utterance, it was tagged as maintained, by adding no suffix. If a referent had not been mentioned in the immediately preceding utterance, and there was mention of another referent in this preceding utterance, the referent was tagged as re-introduced with the suffix -REIN. Indeterminate or uncertain tokens either for the referent itself or its activation status were further tagged with a question mark. An illustration of this coding is provided in Figure 7, using the two Norwegian SL utterances first shown in Figures 4 and 6.

Figure 7:

Annotating for referring expression, activation status, and semiotic composition (Ferrara in prep, KNTS_B_CNT_Frosk.eaf, 00:00:9.9-00:00:12.2).

After referents and their activation status were identified, the semiotic composition of these referring expressions was tagged on the Narrative Referent Sign Type tiers. Again, three copies were created to accommodate simultaneous referring expressions. Annotations were coded for the sign type(s) produced as part of the referring expression. In many cases multiple sign types were produced as part of one referring expression, which meant that all such signs were tagged within the annotation (see Figure 7). The sign types and their tags are listed and explained in Table 2 (based on Hodge et al. 2019: 39).

The resulting annotations were created and proofed across three or more parses (see Hodge and Crasborn 2022 and Johnston 2019 for more about the stabilization, rather than validation, of corpus annotations). First, initial annotation of each dataset by proficient language users was carried out. This initial annotation was then revisited one or two times to address typos and other uncertainties. A third review of the data was conducted by the first and third author. Any discrepancies or uncertainties were identified, and these were discussed with the initial annotators and resolved. Sometimes this involved an additional parse of the dataset by both the original annotator and the first or third author. The final annotations of referring expressions and their semiotic composition were extracted from the ELAN files, proofed again, and then further organized and analyzed in R (R Core Team 2019).

2.3 General overview of the data

The annotation methods described above resulted in the annotations that were used to investigate how signers of five signed languages do reference. Each language data subset contained composite utterances, manual signs, and referring expressions which provided the foundation for the analysis presented here. Summaries of these annotations are reported for each language in Table 3. It is important to note that while annotations of composite utterances and referring expressions were a part of annotation work undertaken for this study, and checked across multiple parses, the manual sign annotations were not. Thus, the numbers of manual sign tokens reported in Table 3 for each language are based on previous annotation work, particular to each language corpus, and should be treated as rough estimates only.

Referring expressions included the full range of semiotic strategies and semiotic modes described in Sections 1.1.1–1.1.5. Table 4 provides cumulative frequency counts of the semiotic strategies engaged across the five datasets, ordered according to the presumed conventionalization (more to less conventional) of the strategy, in addition to the counts for referring expressions produced only on the weak hand. These figures reveal that signers engage less conventionalized depictions (constructed actions and depicting signs) and more conventionalized descriptions (lexical signs and mouthing) to refer to entities. Indexicality is also important in referring expressions considering the frequency of invisible surrogates and pointing.

These referring expressions and their semiotic compositions constitute the data for the following analysis, presented below as two studies. First, we analyzed the five signed languages together to compare the distribution of individual semiotic strategies (types of signs) across languages. Then, in the second study we provide individual language profiles of the five signed languages, detailing how semiotic strategies co-occurred across referring expressions.

3.1 Method of Study 1

All analyses were performed in the R programming environment version 3.3.0 (R Core Team 2019). The number of referring expressions were counted by referent and by semiotic strategy. We then modeled the token counts of semiotic strategy inferentially by regressing them onto the interaction of our three levels of animacy and activation, additionally adding language as a control variable. These categorical variables were treatment coded. Bonferroni adjusted pairwise comparisons for interactions are also reported to correct for Type I errors. Separate models were constructed for each semiotic strategy, to systematically limit mixed model complexity. We included the average number of referring expressions per referent as an exposure variable to these Poisson regressions to account for much higher tokens in referents such as boy and dog. Tokens of weak hand use and uncertain coding were excluded in this analysis as we did not have specific predictions for them. Additionally, too few observations of lexical noun phrases and gestures were observed to perform statistical analysis. Each model was tested for overdispersion using the pscl package (Jackman 2020). Where overdispersion was found, a negative binomial regression was performed instead using the MASS package (Venables and Ripley 2002).

3.2 Summary of results for Study 1

Findings from Study 1 show no notable differences for specific semiotic strategies between languages. Generally, the same patterns observed in the previous study of Auslan (Hodge et al. 2019) were also observed in our cross-linguistic sample. Lexical noun phrases, fingerspelling, and mouthing occurred with high frequency when referring to less cognitively accessible referents, compared to more accessible referents. Highly indexical strategies such as pointing, indicating signs and invisible surrogates were used in many different contexts, but occur often with referents that had lower animacy. Depictive strategies such as lexicalized depicting signs, depicting signs, and constructed action were used primarily for more cognitively accessible referents and also for referents that have a higher degree of unique and persistent identifiability (high animacy).

Further, we observed that depicting signs were also used in expressions introducing a subset of inanimate referents (cliff and rock primarily, but bees and hive as well). Signers sometimes used depicting signs within larger phrases to introduce these new referents (for an example of this see Figure 1, where the signer introduced the referent rock with the lexical sign rock followed by the depicting sign ds:round-rock, ‘a rock, shaped-so.’). This pattern suggests a separate pattern of use for depicting signs that differs from their expected occurrence in maintained contexts.

A few interlanguage differences were found for lexicalized depicting signs, but given their lower token count overall, this is likely attributable to intercoder uncertainty in this category. We also observed interlanguage differences in fingerspelling frequency across languages. Notably Norwegian, Swedish, and Finnish SLs had little to no fingerspelling token counts. However, ISL and Auslan did not differ in terms of fingerspelling frequency for the animacy or the activation status of the referent.

These general similarities are visualized in Figure 8, where we calculated the likelihood of each semiotic strategy occurring in each of the nine possible combinations of our activation and animacy categories in each language, and adjusted the strategy count by the average number of referring expressions per referent. This ratio was then adjusted across the observed likelihoods using range standardization. For example, Swedish SL signers using mouthing to express new inanimate referents occurred 41 times. The average number of referring expressions across this subset tagged as new inanimate was 172.37, yielding 0.24 as the frequency of mouthing in this condition. This was the maximum strategy frequency value and therefore is displayed as 1.00 on the likelihood scale. The resulting heatmap in Figure 8 was plotted using the ggplot2 package (Wickham et al. 2019). The findings from Study 1 support our hypothesis that the languages in our study corpus have very similar distributions of semiotic strategies across referring expressions tagged according to activation status and animacy of the referent.

Figure 8:

Heatmap displaying likelihood of semiotic strategy within activation status and animacy level by language. Empty cells indicate categories without observations.

4.1 Methods of Study 2

A heatmap for each language was constructed using hierarchical clustering to show annotated referents on the right and annotated strategies on the bottom. Along the top, semiotic strategies are ordered by patterns of co-occurrence called “signatures.” Along the left, referents are also grouped into “clusters” that reflect the similarity of how the referents are expressed by semiotic strategy. In both cases, dendrograms visualize how distinct each grouping is through a network of branching lines. For example, Figure 9 shows that depicting signs for inanimate referents in Norwegian SL are more likely to be expressed with invisible surrogates than they are with other strategies such as pointing or indicating signs, and they are distinct enough to warrant their own unique signature.

Figure 9:

Unbiased heatmap comparisons of semiotic strategies in Norwegian SL retellings measured for selected narrative referents by activation status. Mean strategy counts per referent were standardized across referents (cell values from 3 to −3). Ward (1963) clustering was used to determine clusters 1–3 (cluster 1, n = 8; cluster 2, n = 8; cluster 3, n = 19). Dendrograms for strategies and referents are ordered by height from the origin.

The optimal number of clusters was identified by calculating multiple clustering indices in the NbClust package (Charrad et al. 2014) and selecting the most frequent clustering solution. Heatmaps were generated using the pheatmap package (Kolde 2019). In these heatmaps the value for an individual cell is scaled so that the average strategy occurrence for that referent is centered and divided by the standard deviation for all observations of that referent. For example, the average proportion of mouthing occurring during introductions of the bees (beesNEW) in Norwegian Sign Language was 0.60. Normalizing across all strategy averages recorded for this referent (beesNEW) returned a cell value of 1.41, i.e., meaning mouthing was observed for beesNEW half as much again as other strategies.

4.2 Results of Study 2

4.3 Interim summary

We found that Auslan, Swedish SL, and FinSL are all best described with four clusters of generally similar sets of co-occurring strategies. Norwegian SL is best described with three clusters and ISL with five clusters. Yet the total number of contrasts between our three levels of activation (new, reintroduced, maintained), and animacy (human, animal, inanimate) returns nine possible clusters. It appears there is no good reason to assume that nine levels of granularity are necessary to understand how the signers of each signed language do reference in their retellings.

The best explanation is that all five languages make differential use of a two-way distinction for both activation status (new vs. maintained/reintroduced) and animacy (human/animal (animate) vs. inanimate). This claim is strongest for the Auslan, Swedish SL, and ISL data, where the clusters map onto these distinctions with only marginal outliers. The ISL data includes two clusters for maintained and reintroduced inanimate referents, but without a readily apparent difference relating to increased specificity in levels of either animacy or activation status. The clustering solution for Norwegian SL, however, stops just short of achieving a new versus maintained and reintroduced distinction for human/animal referents. In FinSL the distinctiveness in co-occurring strategies between new and old human/animal referents is also not strong enough to present as a cluster. Even with these differences between languages, we generally observe that new human/animal referents are expressed with a higher occurrence of lexical signs and mouthing, while maintained and reintroduced human/animal referents are expressed with constructed action. In contrast, new inanimate referents are often expressed with depicting signs, and maintained and reintroduced inanimate referents with depicting signs and invisible surrogates.

5.1 Similarities observed across five signed languages

The results from the two studies suggest the signers of these five signed languages do reference in quite similar ways when retelling Frog, Where Are You? For example, we see lexical signs and mouthing clustering together across each of the languages to consistently introduce new referents. In contrast, constructed actions are often used to reference maintained animate referents—especially the boy, who was the main character. We suggest that these similarities are not exceptionally surprising considering that the languages examined here are all signed languages and that the datasets are all narrative retellings of the same children’s picture book. In fact, in earlier studies, Swabey (2002) on American Sign Language and Pizzuto et al. (2008), who studied reference across three signed languages (American Sign Language, French Sign Language, and Italian Sign Language), also found across a corpus of narrative retellings that signers introduced referents with lexical signs and then preferred constructed actions in non-new contexts. Other early studies have also mentioned that mouthing plays an important role in the introduction of new referents (see e.g., Boyes Braem 2001; Ebbinghaus and Hessmann 1996).

It has often been claimed that the pervasive iconicity of the visual-gestural modality of signed languages is a key reason for formational similarities observed cross-linguistically in signed language lexicogrammars (Hwang et al. 2016; Johnston 1991; Meier 2002; Schembri et al. 2005; Talmy 2003). Researchers have also long observed that the iconic and spatial affordances of signed languages lend themselves well to the visual art of storytelling (see e.g., Bauman 2003; McCleary and Viotti 2010). Studies across signed languages have consistently shown that signers frequently recruit depiction (constructed action, depicting signs) and indication (including the meaningful use of space) during narratives (e.g., Aarons and Morgan 2003; Ahlgren and Bergman 1994; Cuxac 1999; Ferrara 2012; Ferrara and Johnston 2014; McCleary and Viotti 2010; Pizzuto et al. 2008; Quinto-Pozos 2007). The similarities identified in the current study regarding the use of constructed action and depicting signs, as well as indexical strategies, are therefore expected.

An example from our data illustrates this type of similarity. In Figure 10, one signer from each language is re-telling the part of the story where the boy (and dog) look at the frog in the jar. These moments are very similar visually, although they are not identical. All of these signers enact the boy on their heads, face, and torso looking downwards towards the invisible surrogate |frog in jar|. The Auslan and Irish signers produce signs depicting the jar with their hands, while the Swedish and Finnish signers are producing signs meaning ‘look.’ In these cases then we see all signers depicting and indicating with their hands and body, with more or less describing (e.g., the manual sign look in Swedish SL and FinSL). We suggest that there are only so many ways a person could physically enact this event, which explains the similar actions we see across the signers from different languages. We also suggest that this narrative event is well-suited to depiction and indication, even though description is also possible and attested in this dataset.

Figure 10:

Signers re-telling how the boy (and dog) look at the frog in the jar.

There is one major caveat to the similarities observed here: the datasets for these analyses were not large enough to balance for sociolinguistic factors, and there is still much individual variation between signers of each language to be accounted for (see Barth et al. 2021 on the importance of accounting for individual variation in cross-linguistic typology and how corpus methods can do this). Studies with larger datasets that can account for individual as well as cross-linguistic factors are needed. For example, a recent study of constructed action in FinSL showed that age is a significant factor affecting this strategy during storytelling (Puupponen et al. 2022). Individual signers can vary greatly in their use of constructed action while storytelling for many other reasons too (Ferrara 2012; Hodge and Ferrara 2014; Jantunen 2017; Puupponen et al. 2022). In ISL, the mouthing of English words is more common among younger signers and female signers (Mohr 2014). Age has also been shown to be a significant factor affecting the frequency and distribution of depicting sign types in New Zealand Sign Language (McKee et al. 2021). These differences are not trivial, and more work will be needed to integrate this variation into typological studies on signed languages.

5.2 Local ecologies and contact with ambient spoken languages shape potentials for difference

The previous section considered the similarities in doing reference observed across the five signed languages, while noting there were some differences that underscore the specific local ecologies of these communities. In Study 2 an investigation revealed additional differences between how multiple types of signs co-occur within referring expressions and how these patterns of (non-)occurrence are conditioned by animacy and activation status across each individual language. While the Auslan, Swedish SL, and ISL data suggest a two-way distinction of activation and animacy, the Norwegian SL and FinSL data do not. Instead, the Norwegian SL data showed some patterning for human/animal (animate) vs. inanimate referents in maintained and reintroduced contexts. However, the third cluster included all types of referents in all activation settings, which suggests a more complex picture. In the FinSL data, some patterning was observed for human/animal (animate) referents in reintroduced/maintained contexts. However, the other clusters were more heterogenous. While the current study focused on the variables of animacy and activation status, findings here and elsewhere suggest that additional factors influence the choice of referring expression across these five signed languages. We suggest that the (non-)patterns observed also reflect in part the evolution of these five signed languages within their own specific socio-historical contexts.

Related to specific semiotic strategies, the largest difference was observed among signers’ use of fingerspelling. Fingerspelling was mostly likely to be produced by ISL and Auslan signers to refer to new inanimate and animate referents. Figure 8 showed that the signers of the other three signed languages were much less likely to use fingerspelling to do reference. FinSL and Swedish SL signers had some likelihood of fingerspelling new human and animate (animal) referents along with maintained animate referents. There was also some likelihood for Swedish SL signers to fingerspell new inanimate referents and maintained human referents. However, there was zero likelihood that Norwegian SL signers would use fingerspelling as a strategy for identifying referents.

We interpret the various fingerspelling practices observed in the data to reflect the historical, educational, and political practices as well as language ideologies of these signed language ecologies. For example, Auslan signers fingerspell for many different and strategic reasons, such as to emphasize a topic, and fingerspelling is generally perceived positively when done in culturally acceptable and comprehensible ways (Johnston and Schembri 2007). In contrast, the use of fingerspelling in FinSL has been framed more negatively, characterized as artificial and external to natural signing, and much aligned with speaking and writing (Rainò 2014; Tapio 2012). These specific contexts create the potential for cross-linguistic differences that have rarely been considered in discussions of signed language grammar and use (though they have been considered in the fields of Deaf Studies and Deaf Anthropology, e.g., Kusters et al. 2020).

To add another caveat to the study, we acknowledge that differences between languages may also have resulted from categories of semiotic strategies that are more difficult or controversial to annotate, such as lexicalized depicting signs and invisible surrogates (see also Hodge et al. 2019: 45). Invisible surrogates in particular rely heavily on inferencing by an interactant, rather than on any overt lexis or morphosyntax, and thus may be interpreted differently by different interactants, including annotators viewing the video data. We therefore conservatively assume that some of these observed differences could be attributed to variation in annotator decisions, rather than in actual differences in referencing between the signers of each language. Overall, findings do suggest that earlier categorical claims about how signers do reference should not be automatically assumed to characterize all signed languages, and that these claims should be tempered with statistical generalizations. The data clearly show there is a need to better understand the diverse semiotic strategies used to do reference in signed languages, and respect any observed differences across languages.

We hypothesize differences may pattern in other ways in non-storytelling contexts, including more naturalistic conversations. For example, Ferrara et al. (forthc) examined fingerspelling in Norwegian SL conversations and determined that signers do indeed fingerspell, even though it is infrequent (2.9% of all manual signs). An examination of these fingerspelled tokens further revealed a strong preference to fingerspell proper nouns. The lack of fingerspelling in the dataset of Norwegian SL retellings analyzed here is therefore unsurprising, because these retellings did not include the specific names of places or people (only generic identifiers). Analysis of other discourse contexts and text-types will shed further light on how semiotic strategies pattern together within and across different signed language ecologies. For example, corpus research has shown that Auslan signers typically use English mouthing with fingerspelled and lexical nouns, regardless of whether the signer is retelling a story or engaging in conversation (Johnston et al. 2016).

In addition, the frequent use of constructed action observed in the current study may also manifest differently in conversational contexts, where it is observed to occur much less frequently in comparison with narrative retellings (Ferrara 2012; Puupponen et al. 2022). Signers may instead coordinate more conventionalized and/or indexical strategies when doing reference. Further empirical research is needed in order to detail how signers do reference in more spontaneous and non-narrative contexts.