Finding continuers in Swedish Sign Language

1 Introduction

Languages arise from the need to communicate and they form an indispensable tool for building and maintaining social bonds. One of the most central aspects of language is thus its use in interaction with others. Humans have developed many linguistic strategies to achieve efficient and successful communication, such as ways of signaling understanding with backchannels, serving as feedback signals from the addressee, as well as signaling a lack of understanding with repair, flagging misunderstanding or need for clarification (e.g., Dingemanse 2024; Dingemanse and Enfield 2024; Micklos and Woensdregt 2023; Schegloff et al. 1977). Such communicative strategies illustrate how conversation is not one-directional, but rather a continuous and simultaneous negotiation between the interlocutor who is currently (more) active in producing an utterance (the speaker/signer) and the interlocutor to whom it is directed (the addressee), a negotiation which renders both interlocutors active participants in the communication taking place (Bavelas and Gerwing 2011; Bavelas et al. 2000; Goodwin 1986a, 1986b). In this conversational negotiation between interlocutors, the feedback signal expressing comprehension and encouragement for the speaker/signer to go on (e.g., the English interjection uh-huh) is called a continuer (Goodwin 1986b). Continuers are an important tool at the addressee’s disposal to maintain the communicative flow, expressing that “one unit has been received and that another is now awaited” (Goodwin 1986b: 208). Naturally, this function could be expressed also with signals other than sounds, such as a head nod (see Lutzenberger et al. 2024). Using corpus data from a sample of genealogically diverse spoken languages, Dingemanse et al. (2022) illustrated that distributional patterns of words in a conversational corpus can be used to identify communicative devices, such as continuers, in a noncircular and language-independent way. By looking at sequences of repeated, identical turns by the addressee, interspersed with the primary speaker’s utterance(s), the authors could identify words, in the individual languages, that functioned as continuers. Thus, the words used in this core communicative function were shown to be identifiable through distributional patterns alone, in the annotated corpus data from spoken language interactions.

Languages arise through natural interaction between humans, wherever and whenever needed, and in whatever modality is available. For spoken languages, the primary channel of transmission is the auditory one, where spoken words are uttered by the speaker and perceived as sound waves and interpreted as language by the addressee. Spoken language interaction naturally involves other modalities, too, such as the visual channel (e.g., through bodily gestures and facial expressions) and the tactile channel (i.e., physical touch). Sign languages make use of the gestural-visual modality, being articulated by the signer with the hands, face, and body and perceived visually (or, alternatively, tactilely) by the addressee (also known as the “signee”). The modality is one of the reasons why it has been challenging to represent sign language data for linguistic analysis: whereas video is the only format that truly represents the language production, most research has been limited to written transcriptions that do not capture all aspects of production (see Crasborn 2015; Frishberg et al. 2012; Johnston 2014; Miller 2006). With the increased availability of video recording and storage technology, but also trends in linguistic research (cf. Ferrara and Hodge 2018; Vermeerbergen and Nilsson 2018), sign language researchers in a variety of countries started building sign language corpora, using software suitable for the synchronized display of multiple video files and machine-readable annotations (Fenlon and Hochgesang 2022). The possibility of creating and utilizing sign language corpora has revolutionized sign language linguistics, giving researchers the data and tools to investigate many linguistic phenomena based on, for example, distributional patterns (Börstell 2022a), and to research naturalistic, dyadic, conversational signing. As such, we now have the resources to utilize tools and methods from corpus linguistics and answer questions about language use in conversation, much like what has been done in the linguistic study of spoken languages for decades. In addition, even more recent technological advances have enabled computer vision-based approaches to analyzing sign language data (e.g., Börstell 2023; Kimmelman and Anželika 2023). These methods allow for some features of sign language production to be extracted directly from the video files (i.e., the recorded signing), without the need for time-consuming, manual annotation work.

The use of language in human interaction has been studied extensively within several subfields of linguistics, but like most linguistic research, there is a heavy skew towards spoken languages as the object of study. With the recent development in corpus building within sign language linguistics, there have been some advances in interaction studies on sign languages as well, although this is still sorely under-explored on the whole. Many of the existing studies on sign language conversation analysis have looked at the coordination of turn-taking in visual and tactile signed conversations (e.g., Baker 1977; Berge and Raanes 2013; Bjørgul 2022; Casillas et al. 2015; Coates and Sutton-Spence 2001; de Vos et al. 2015; Groeber and Pochon-Berger 2014; Iwasaki et al. 2022; McCleary and De Arantes Leite 2012; Mesch 1998; Van Herreweghe 2002), whereas other studies have focused on conversational repair (e.g., Byun et al. 2018; Lutzenberger et al. 2024; Manrique 2016; Manrique and Enfield 2015; Mesch and Schönström 2023; Safar and de Vos 2022; Skedsmo 2020, 2023). A number of studies have identified specific signs that are used extensively in conversation-regulating and discourse functions, among them pointing signs (Ferrara 2020, 2022; Lepeut and Shaw 2022) and the gesture-like palms-up (flat hands, palms facing up; see Figure 1). The latter has been shown to exhibit many discourse functions across sign languages, such as pause-filling, backchanneling, expressing questions, and politeness (e.g., Engberg-Pedersen 2002; Gabarró-López 2020, 2024; Hoza 2011; Lepeut and Shaw 2022; McKee and Wallingford 2011; Roush 2007; Ryttervik 2015), and also across many spoken languages (e.g., Cooperrider et al. 2018; Kendon 2004; Müller 2004). While there has been some debate around the division between signs and gestures in sign languages (e.g., Ferrara and Hodge 2018; Kendon 2008), I do not make any categorical distinction between them and consider items like palms-up to be signs.

Figure 1:

Palms-up PU@g in STS (Svenskt teckenspråkslexikon 2023: 18717).

Both pointing signs and palms-up have been shown to be highly frequent items in conversational signing across sign languages (e.g., Börstell et al. 2016; Fenlon et al. 2014; Johnston 2012; McKee and Kennedy 2006), and Mesch (2016) found that these signs occur as manual backchannel responses in conversations in Swedish Sign Language (STS; svenskt teckenspråk). In her study, Mesch (2016) annotated 35 min of STS conversations from the STS Corpus (Öqvist et al. 2020) to identify the most common types of backchannel responses in the language. The study focused on manual signs since there is very limited annotation of nonmanual activity in the STS Corpus. However, mouthings, mouth activity reminiscent of silently articulated spoken language words (see Bisnath 2024), and mouth gestures, mouth activity not associated with spoken language (see Crasborn and Bank 2014), are frequently used in STS, including as backchannels (cf. Mesch and Schönström 2021; Mesch et al. 2021). Mesch (2016: 39) found that palms-up (glossed PU@g in the STS Corpus; see Figure 1) is one of the most frequent backchannels in STS, and Ryttervik (2015) found that it can be used to acknowledge or question the content uttered. However, the most frequent manual backchannel sign in STS was found to be the two variants for signing ‘yes’ (see Figure 2): originally JA@b (‘yes’; “@b” referring to bokstavering ‘fingerspelling’), derived from a clear fingerspelling of the letters J and A (for the Swedish word ja ‘yes’), but reduced as a backchannel into JA@ub (the “u” referring to uppbackning ‘backchannel, support’). Mesch (2016) found that the reduced backchannel JA@ub is often articulated with a different orientation (palm facing inwards), movement (repeated opening and closing of the fingers), and/or location (held at a lower location, e.g., closer to a resting position in the lap). While manual backchannels were frequent, Mesch (2016) found that most backchannel responses in her dataset consisted of nonmanual markers, similar to recent findings for British Sign Language (BSL), where head nods were found to be the most common expression (Lutzenberger et al. 2024).

Figure 2:

Signs JA@b and JA@ub in STS (Svenskt teckenspråkslexikon 2023: 9027, 11810).

Mesch (2016) looked at backchannel responses in a wider sense, as in any utterance produced by the addressee in the conversation. However, there is a distinction to be made between continuer (generic) and assessment (specific) responses, in that the former is not commenting on the conversational content, but simply signals understanding and that the primary speaker/signer should continue their turn (Bavelas et al. 2000; Goodwin 1986b). In this paper, I look at continuers as a subset of backchannels, following the approach by Dingemanse et al. (2022) in attempting to identify them by distributional patterns alone. In their study, they searched for continuer candidates in a crosslinguistic sample based on their distribution in conversational corpus data, identifying “streaks of non-unique conversational turns that occur in frequent alternation with unique turns by other participants [… with] a minimum streak length of 3” (Dingemanse et al. 2022: 161). In a second step, the authors compared the form properties of the identified continuers against other high-frequency words and found that continuers have minimal form (albeit not shorter than top-frequency items) and often contain nasals and partially reduplicated form (e.g., English mhm; Dingemanse et al. 2022: 164).

The aim of this paper is (i) to test whether we can employ distributional patterns to identify continuer candidates in sign language corpus data from Swedish Sign Language (STS) in a similar fashion to what Dingemanse et al. (2022) did for spoken languages, and (ii) to make use of computer vision methods to analyze form properties of identified candidates from video data.

2 Methodology

The data for this study comes from the STS Corpus (Mesch et al. 2012; Öqvist et al. 2020) as stored in The Language Archive^[1] online repository in July 2023, consisting of 298 corpus files (dyadic interactions of narrative and conversational texts) comprising 189,679 sign glosses across 42 signers and over 23 h in total duration. The interactions are video-recorded from multiple angles and annotated with ELAN (Wittenburg et al. 2006) for individual signs articulated by each signer’s two hands (see Figure 3 for an illustration of the corpus interface layout; note that two-handed signs are only annotated on a single tier in the STS Corpus, rendering the nondominant hand tiers empty in this excerpt). The main data includes ELAN annotation files and corresponding metadata files for the entire STS Corpus.

Figure 3:

Illustration of the STS Corpus video player and annotation tiers (Öqvist et al. 2020): https://teckensprakskorpus.su.se/video/sslc01_241.eaf?t=358880.

In order to find continuers according to the distribution-based method used by Dingemanse et al. (2022), the sign annotations in the corpus data had to be grouped into units corresponding to turns. Since the STS Corpus does not have any such type of unit in its annotations (see Börstell 2024), turns had to be inferred. Three methods to inferring “turns” were applied, named interval, quantity, and sequential. The interval method uses pauses to infer turns: anytime a pause between sign annotations is longer than a set threshold (here: 500 ms), a segmentation of turns is made – it is thus possible for a single signer to have multiple consecutive turns. The quantity method calculates which of the two signers has the most signs in each window of three chronologically ordered signs, assigning this signer the turn for that stretch of signing, regardless of any interspersed sign by the other signer. The sequential method simply assigns a new turn as soon as there is a change of signer in the temporal sequence of chronologically ordered signs, even if it is only a single sign by one signer within a longer stretch of signs by the other. These three methods are applied separately to the corpus data and continuer candidates are identified for each method based on the definition of a streak of three nonunique (i.e., identical) turns, following Dingemanse et al. (2022: 161).^[2]

Additionally, a fourth method was used, simply looking at the frequency distribution of signs in the context of overlapping and nonoverlapping signing, and whether or not there is a signer change at that location. Here, the frequency distribution based on these two variables (overlap and signer change) was used to calculate weighted log odds (Schnoebelen et al. 2022) of signs occurring in these contexts. This method thus identifies signs that are statistically overrepresented in a particular context – for example, whether a sign often occurs during signing that overlaps with the interlocutor’s signing, but does not result in a change of primary signer (i.e., an expected context for continuers).^[3]

A subset of the data is used for the computer vision analysis. This subset consists of 15 of the corpus files covering 30 of the 42 signers, with 30 video files showing each of the signers from a front-facing angle (see Figure 3), comprising 13,507 sign glosses and about 2 h and 45 min in total duration. The subset video data was processed with the Python implementation of computer vision software MediaPipe (Lugaresi et al. 2019). MediaPipe is used to extract landmarks on the body of the signers for body-pose estimation, such that the coordinates of these different points on the body can be tracked frame by frame throughout a video. In this study, I focus on the landmarks representing the articulating hands, effectively extracting manual activity over time in the videos (see, e.g., Börstell 2023 on its application to sign language articulation). This method gives you data on where the hands are in the horizontal (x) and vertical (y) dimensions in each frame of the videos (i.e., location), and consequently how far the hands have moved (i.e., distance) between frames.

The corpus data and the coordinates from the video analysis were processed, analyzed, and visualized using R v4.3.2 (R Core Team 2023) and the packages glue (Hester and Bryan 2022), ggrepel (Slowikowski 2023), ggtext (Wilke and Wiernik 2022), here (Müller 2020), patchwork (Pedersen 2024), scales (Wickham et al. 2023b), signglossR (Börstell 2022b), slider (Vaughan 2022), tidylo (Schnoebelen et al. 2022), tidyverse (Wickham et al. 2019), and xml2 (Wickham et al. 2023a).

The data and scripts used for this study can be found at https://osf.io/hyxrp.

3 Results

3.1 Finding continuer candidates

Using the three methods for estimating turns (or utterance units) in the corpus data, there are only two signs that appear in the intersection of all three methods, namely JA@ub (Figure 2) and PU@g (Figure 1). For each of the three methods, JA@ub occurs in six to seven times more streaks than PU@g. These results would immediately corroborate the findings of Mesch (2016), who also identified these two signs as the most frequent individual manual backchannels in a subset of the STS Corpus. This also shows the crosslinguistic and crossmodal validity of the approach of Dingemanse et al. (2022), illustrating how this distribution-based method for identifying continuer candidates appears to be successful also when applied to the STS Corpus data.

The second approach to identifying continuers in the STS Corpus data was based on the distributional frequency patterns in contexts of overlapping/nonoverlapping signing and possible signer change, calculated with weighted log odds (Schnoebelen et al. 2022) in order to see overrepresentation of signs in a particular context. Figure 4 shows the results of this calculation, across the contexts of (i) overlapping signing without signer change, (ii) overlapping signing with signer change, and (iii) no overlapping signing with signer change – only signs with a token frequency above 20 for each context are included, in order to remove spurious low-frequency items. The first (left-most) context in Figure 4 would represent the prototypical context of a continuer: the addressee inserts a single response without attempting to take over the signing. The other two contexts are where there is a signer change.

Figure 4:

Weighted log odds of sign token frequency in three distributional contexts based on signing overlap and signer change. Only signs with a context token frequency >20 are included.

As Figure 4 shows, the two continuer candidates are represented in the top 10 signs for most of these contexts. However, JA@ub is the only sign to stand out from the rest in any of the contexts, namely in the one expected for continuers (i.e., signing overlap without signer change), a context for which PU@g ranks only number 12, with low log odds. Based on the distribution in both continuer streaks and overlap/signer-change contexts, JA@ub appears to be the more prototypical continuer among these two candidates, with PU@g being more prominent in other contexts. Several other common backchannels are also found among the top signs in the context of signing overlap without signer change, such as EXACTLY (originally glossed PRECIS), ACCURATE (originally STÄMMA), and OH-REALLY (originally JASÅ), but these are assessment-type backchannels rather than generic continuers. Interestingly, even though the data generally concerns manual signs, and only very few nonmanual expressions have been annotated, one mouthing is included in the continuer context in Figure 4. This is the annotation “mtyp:ja”, which is a mouthing of the Swedish word ja (‘yes’) without any accompanying manual sign (Mesch and Wallin 2021: 38–39; see also Mesch et al. 2021). Since the weighted log odds calculation accounts for distribution relative to frequency, “mtyp:ja” can appear in the top 10 despite having relatively few tokens in total (n = 75), compared to JA@ub (n = 2,054) and PU@g (n = 3,283), because it is particularly associated with this context – compare this to the most frequent signs in the corpus: PRO1 (first person point; n = 10,846) and IX (originally PEK ‘point’; n = 6,925). Pointing signs are among the most frequent lexical items in STS discourse (Börstell et al. 2016) and Figure 4 shows that they are among the top signs occurring in contexts with signer change, but less so in the expected continuer context – that is, they appear to be frequent at the beginning and end of (extended) turns, but not as stand-alone turns.

Based on these two approaches to identifying continuers, both JA@ub and PU@g appear to be relevant candidates. However, whereas the streak method identified both of the signs as continuer candidates, JA@ub appears to be significantly more frequent in the expected continuer contexts in terms of overlap and signer changes, compared to PU@g and other high-frequency signs. Figure 5 shows an example of the distribution of the two candidates across three corpus files.

Figure 5:

Location of JA@ub (yellow) and PU@g (blue) in three corpus files. Bands show the stretches of signing for each signer and triangles show the location of continuer candidates.

As Figure 5 shows, both JA@ub and PU@g occur frequently, sometimes as short responses during a longer stretch of signing by the primary signer, but also at end points of longer turns. Dingemanse and Liesenfeld (2022) compare the frequency of continuer responses in corpora across languages and find that 8 % of turns are continuers in English and 21 % of turns are continuers in Korean. In the STS Corpus data, about 6.5 % of (sequential) turns consist of a single JA@ub response, and about 2.5 % of a single PU@g response, putting STS closer to English than Korean when looking mainly at manual responses and disregarding nonmanual ones (cf. Lutzenberger et al. 2024; Mesch 2016).

3.2 Form properties of the continuer candidates

Börstell et al. (2016) showed that STS patterns as expected, based on spoken language corpora, in that more frequent lexical items are shorter (based on annotation durations). Figure 6 shows the correlation between token frequency and mean duration in the whole STS Corpus. Similar to what Dingemanse et al. (2022) found across spoken language corpora, Figure 6 shows that the most frequent items are almost an order of magnitude more frequent than the identified continuer candidates. However, both PU@g and – even more so – JA@ub appear to have longer durations than expected based on their frequency. This discrepancy was discussed by Börstell et al. (2016) as well as Mesch (2016) with regard to JA@ub, with its longer duration attributed to the fact that it is often articulated with multiple sequential repetitions. While Dingemanse et al. (2022) note that spoken language continuers are somewhat longer than top-frequency lexical items, Figure 6 seems to point to longer-than-expected durations for the two candidates, especially JA@ub.

Figure 6:

Token frequency and mean duration of all signs in the STS Corpus, with the continuer candidates highlighted. Dashed lines show means of token frequency and duration across all signs. Signs with a mean duration ±3 standard deviations from the global mean are removed.

In order to look at form properties of the continuer candidates in more detail, we can turn to the subsample that was analyzed using computer vision, estimating articulatory features of the signing. Figure 7 shows the estimated hand height (vertical position of the wrist) and the total distance traveled by wrist and index finger combined for the articulating hand during signing. The values have been z-scored relative to the signer, and only across video frames time-aligned with an annotated sign (i.e., stretches of rest and transport movements are removed).

Figure 7:

Mean estimated height and distance traveled by the active hand for each sign. Continuer candidates are highlighted. Signs with mean values ±3 standard deviations from the global mean (dashed lines) are removed.

What Figure 7 shows is that both JA@ub and PU@g have lower vertical articulation and generally travel a slightly shorter distance throughout the sign than most signs in the subsample. Mesch (2016) noted that many manual backchannel responses in the STS Corpus are articulated lower down, close to the signer’s lap (where the hands are often placed in a rest position when seated), attributed to a desire “to not direct attention away from the primary signer” (Mesch 2016: 32). Thus, it is unsurprising that the vertical articulation of JA@ub and PU@g is generally lower on average. However, the distance traveled is only slightly shorter than signs on average, which may be linked to Figure 6 and their longer-than-expected durations. Thus, Figure 8 shows the same values as Figure 7, but the distance traveled has been recalculated to show the relative distance traveled by dividing the total distance by the duration (i.e., distance traveled per time unit). From Figure 8, we can see that JA@ub shows a shorter relative distance traveled than most signs, whereas PU@g now falls slightly above the global mean. In fact, among signs with more than five occurrences in the subsample, thus removing low-frequency items, there are only four signs that have a lower articulation height and shorter relative distance traveled than JA@ub, and they all happen to be so-called buoys. Buoys are signs articulated on the more passive hand, often spanning multiple signs by the other hand, used for discourse reference and cohesion (see Liddell et al. 2007; Nilsson 2007; Vogt-Svendsen and Bergman 2007). This indicates that JA@ub is among the bottom-most signs in terms of visual prominence (low and stationary over time).

Figure 8:

Mean estimated height and distance traveled per time unit by the active hand for each sign continuer candidates and signs with values lower than the candidates’ and >5 occurrences are highlighted. Signs with mean values ±3 standard deviations from the global mean (dashed lines) are removed.

4 Discussion

The goals of this study were (i) to see whether distributional patterns (cf. Dingemanse et al. 2022) could identify continuer candidates in STS, and (ii) to make use of computer vision to analyze form properties of identified candidates.

With regard to the first goal, it was shown that distributional patterns can be used to successfully identify continuers in the STS Corpus, even when turns are inferred from the data rather than manually annotated. The streak method based on Dingemanse et al. (2022) singled out the two signs that Mesch (2016) identified as the top manual backchannels in a manually annotated subsample of the STS Corpus, namely JA@ub and PU@g. A second approach calculated weighted log odds (Schnoebelen et al. 2022) for the distribution of signs in different conversational contexts: whether there was overlap between interlocutors’ signing and whether there was a signer change around the occurrence. Here, the results point to JA@ub being the more dedicated continuer, whereas PU@g occurred relatively more frequently in the other contexts. Based on previous work, PU@g has been shown to be used in a multitude of functions in both STS (Ryttervik 2015) and other languages (e.g., Cooperrider et al. 2018; Gabarró-López 2024; Lepeut and Shaw 2022), and should be categorized as a wider backchannel rather than a dedicated continuer. Thus, this combined approach of using both sequences of signs (streaks) and contextual frequencies (occurring with or without overlap or signer change) may be a way to validate potential candidates from multiple angles and spot nuances in their distribution. The context approach would also have other applications, such as identifying turn-regulating signs from signing context alone. For instance, Figure 4 showed that pointing signs are more prominent when there is a signer change, which mirrors findings from Norwegian Sign Language, where index points can be used to coordinate turn-beginnings (Ferrara 2020, 2022). A more comprehensive annotation of the nonmanual expression of backchannels and conversation-regulating devices (e.g., head nods, mouthings, eye gaze) should be a fruitful future direction for investigating continuers in STS (cf. Lutzenberger et al. 2024). In principle, with annotations of such expressions in a corpus, the methods for identifying continuers employed in this study should be applicable. The current results should thus be seen as an incomplete picture, as the data is almost entirely focused on the manual channel.

Several form properties were identified through sign durations as well as the articulatory features extracted with computer vision. While preliminary, these results point to similarities in the patterns found across spoken languages, stating that optimal continuers are “(i) easy to plan and produce, (ii) unobtrusive, and (iii) sufficiently distinct from regular words to be seen as ceding the conversational floor” (Dingemanse et al. 2022: 161). Based on the results of this study, the sign JA@ub is articulated lower in height with the hand traveling a shorter distance (i.e., moving less) than in most other signs. These properties make the sign easy to produce and simultaneously less visually prominent by having a lower articulation activity, hence rendering the sign less obtrusive when overlapping in conversation. Although a more detailed phonetic analysis is needed to measure distinctiveness of form across more aspects of the sign than just height and movement (e.g., handshape and palm orientation in combination with repeated local movements), these preliminary form characteristics may potentially distinguish JA@ub from other signs.

In conclusion, this study has shown that the distributional patterns and form-based properties used to identify and compare continuers across spoken languages can be transferred successfully to sign language data. The identified candidates align with those signs observed in previous research based on manual annotation of turn-taking and backchanneling in STS, thereby validating the search methods. The form (e.g., unobtrusive) and distributional properties (e.g., frequent, repeated sequences) of the STS continuer candidates, particularly JA@ub, are similar to continuers identified in various spoken languages. The signed and spoken modalities may come with slightly different affordances. For example, extended production overlap between interlocutors may have lower signal interference in the visual modality compared to the auditory one, thus potentially allowing for longer continuers in sign languages. Nonetheless, the articulation of JA@ub, while often longer than expected based on spoken language counterparts, seems to adhere to the idea of unobtrusiveness by exhibiting lower manual activity. This highlights the modality-independence of continuers as a linguistic concept and illustrates their central role in communication, which expands on our understanding and definitions of continuers as a conversation-regulating device across individual languages and different modalities.