Constructed dialogue in signed-to-spoken interpreting: renditions as a reconfiguration of utterance semiotics

2.1 Depiction, description, and indication as semiotic discourse strategies

Depictions are linguistic expressions that in some way depict the intended referent (Clark 1996; Clark and Gerrig 1990; Ferrara and Hodge 2018). In a broad sense, depictions are linguistic utterances in which information is presented in a “show-you” format (Beal-Alvarez and Trussell 2015; Johnston 2019). Constructed dialogue is depictive because it shows dialogue (Dudis 2011): it prompts the signer or speaker to enter the role of the person they are talking about, mimicking some historical or fictional part of a conversation. Depictions are typically dependent on context for correct interpretation, contrasting with descriptions, discussed next.

Depiction represents but one facet of the resources of constructed action. The lexical words or signs of these utterances represent descriptive resources. These are highly conventional parts of discourse, without an apparent motivated link between form and meaning (Dingemanse 2015; Ferrara and Hodge 2018). In constructed dialogue sequences, conventional lexical items serve as descriptive signaling method semiotically. Historically, descriptive resources have been the easiest to account for and categorize and have therefore been the focus of most research on language (Linell 2005). In discourse, descriptions are linguistic utterances where meaning is conveyed in a “tell-you” format (Beal-Alvarez and Trussell 2015; Dingemanse 2015; Johnston 2019). Thus, while constructed dialogue depicts discourse, it is achieved through the use of descriptive resources.

Indicative acts entail directing the interlocutor’s attention towards something. This joint attention can be achieved by physical pointing, or gazing towards something, but we can also indicate with lexical items like deictic expressions. The deictic expressions (like “this” or “she” in English) represent a hybrid concerning conventionality: they are highly conventional lexical items that require context to make meaning. Further, indication is a strategy for anchoring discourse in time and place (Clark 1996; Ferrara and Hodge 2018). In discourse containing constructed dialogue, gaze has been reported as an important resource for indicating the (imagined) addressee of the utterances (Metzger 1995).

Importantly, the descriptive, depictive, and indicative resources often co-occur, demonstrated by the resource of constructed dialogue: a linguistic strategy that depicts discourse with the descriptive resources of conventional grammar and lexical items, often indicating an imagined addressee with body posture and gaze.

2.2 Semiotic complexity in a visual language ecology

Janzen (2017) investigates how the composite nature of languages might apply to signed languages, where everything occurs in the visual modality: “For a group of languages that share the same bodily medium of expression with gesture, understanding how utterances might be composite in this way seems a more difficult task” (Janzen 2017: 512). He continues to establish the presence of gesture in signed languages and how the concept of multimodality might apply in a visual language ecology:

In a strict sense, signed language use may not be considered multimodal because it typically is not a combination of speech of gesture, and because signs and gestures are thought of as expression in the same medium or modality. But perhaps such a conclusion is misleading … if we consider “modality” to stretch beyond a simple distinction between two articulatory systems – speech and gesture/signing – and treat multimodality as true multiplicity and not duplicity, then both spoken and signed language can be analyzed under the same rubric. (Janzen 2017: 519–520)

Thus, investigations of multimodality in spoken languages often focus on combining different resources, such as speech and gesture. Janzen (2017) suggests the topic-comment structure as a representation of the multimodal and composite nature of signed language utterances. This discourse structure involves interpreting lexical items in sequences with varying semiotic signals, embodying both compositeness and multimodality. In topic-comment structures, the topic is understood as indicating relevance to the subsequent comment. Constructed dialogue is another discourse structure where participants must interpret lexical items through different semiotic lenses. In the following, I demonstrate how the semiotic approach to languages can be used to treat constructed dialogue as a multiplicity of semiotic resources.

2.3 Constructed dialogue depicts discourse

The term constructed dialogue was coined by Deborah Tannen (1986), working with spoken languages. It is a linguistic device of representing dialogue, extensively explored in both spoken and signed languages (e.g., Dudis 2011; Ferrara and Bell 1995; Hodge and Cormier 2019; Metzger 1995; Mohammad and Vásquez 2015; Tannen 1986; Thumann 2011; Young et al. 2012). This resource of communication is also referred to as reported speech in the literature (e.g., Holt and Clift 2006). However, Tannen (1986) observed that the term obscured the fact that we can never report truthfully exactly what has been said by someone (or oneself) in the past, we construct dialogue according to our memory. While Tannen reported from spoken languages, Metzger (1995), working with American Sign Language (ASL), added the term constructed action as signed language utterances often include the construction of peoples’ action and not only their dialogue. Notably, although the term “constructed action” originates from observations of a signed language, spoken languages also employ action-like modes of expression (Kendon 2014).

As previously stated, constructed dialogue utilizes an important depictive resource by means of its “show-you” format (Beal-Alvarez and Trussell 2015; Dudis 2011). An important backdrop for the current study is that language ecologies have different use of semiotic strategies. The notion of a language ecology considers languages as part of larger ecological systems (Garner 2004; Haugen 1971). In this study, an ecological lens of inquiry is useful to help us understand how a visual language ecology (like NTS) differs from spoken language ecology in its sophisticated use of depictive resources (e.g., Brennan 1999; Hodge 2014). When a language is perceived and produced in a visual space in front of the signer, the ecological surroundings have allowed depictive resources to develop into a sophisticated system of meaning-making. The interpreted interaction investigated here, demonstrates how the “show-you” format of constructed dialogue exists with different ramifications in a spoken and a visual language ecology. Thus, the transition between language ecologies requires a nuanced semiotic adjustment. In this study, to highlight the semiotic complexity of source utterances, as well as the semiotic work carried out by the interpreter, I refer to this adjustment as the “reconfigured semiotics of utterances.”

Having established a semiotic and ecological approach to language practices, I now turn to the specific ecology surrounding the explored events of this study: the ecology of an interpreted event.

2.4 A semiotic approach applied to interpreting studies

The acknowledgement that languages are multimodal and semiotically complex has implications for how we understand the task of an interpreter. There is a growing body of work that considers the different functions of multimodal resources of interpreted interaction (Arbona et al. 2022; Davitti 2019; Davitti and Pasquandrea 2017; Janzen et al. 2023; Janzen and Shaffer 2013; Mason 2012; Poignant 2021; Tiselius 2022). Recognizing that the multimodal resources of language practices are also part of a specific language ecology, presents the possibility of viewing the act of simultaneous interpreting as replacing a specific set of semiotic resources and strategies so that renditions fit into the ecology of another language. This semiotic approach highlights that lexical choices are only part of the interpreting process.

A few studies have applied a semiotic approach to the analysis of interpreted material. Without explicit reference to semiotics, Halley (2020) explores if there is a directionality effect in the rendering of depiction strategies of ASL (American Sign Language) and English. In an experimental context, he investigates the work of one interpreter in both directions with the conclusion that the interpreter needs to “render a wide variety of forms of depiction in both directions, which further complicates the interpreting process” (Halley 2020: 34). Drawing on the (post-)Peircean semiotics also applied in the current study, Meurant and colleagues (2022) explore reformulations in signed and spoken discourse, wherein signers or speakers repeat their discourse using different terms to clarify an expression. They find that reformulations may play a key role in the interpreters’ process. Also, they argue descriptions and depictions are combined in reformulations, as “signers frequently produce reformulations by combining conventional signs and structures with signs and structures whose meaning is constructed depictively” (Meurant et al. 2022: 327). They also mention the time constraints of simultaneous interpreting which poses a specific challenge to the interpreted event and may result in extra clarity requirements.

While varieties of multimodal resources are highlighted in several studies on interpreting, few studies on dialogue interpreting have explicitly differentiated between semiotic strategies (although, see Stone and Hughes 2020). The current article explores resources in renditions with the lens of a semiotic approach to language practices: The strategies of depictions, descriptions, and indications.

3.1 Participants and data

The data for this study consists of two interpreted informal lunch conversations (ca. 84 min in total) that have been recorded and annotated. The sites for recording were chosen based on my professional network and availability. It was important that the hearing non-signing participant was in the minority, for two reasons: (1) Directionality: This study primarily focuses on spoken language renditions. Therefore, it was important to ensure that there were ample instances of signed-to-spoken renditions, and (2) Adaptation: Deaf individuals are typically experts at calibrating their language to meet communication needs (Moriarty and Kusters 2021), particularly in interpreted interaction (Haug et al. 2017). This expertise may result in adapting into a slower, clearer discourse; thereby facilitating comprehension for (hearing) individuals with different levels of proficiency in NTS. By ensuring a majority of deaf, signing individuals, I aimed to reduce alignment with L2 users because it could potentially alter the character of the signing discourse. Finding workplaces where deaf professionals are in the majority, limits the potential objects of study. However, while 84 min of recorded material is a small sample, it provides sufficient example sequences for a multimodal in-depth analysis (Davitti and Pasquandrea 2017; Goodwin 2017; Halley 2020; Mondada 2018).

All interpreters in this study are familiar with the workplace and most of the participants. The data are semi-naturalistic as informal settings require an appointment to film participants, and there were two or three cameras and a researcher in the room. On arrival, all participants were provided with the necessary information to give their consent to participate in the project. The project description was sent to the workplace for participants to read beforehand, and in addition information was given in Norwegian Sign Language and Norwegian to deaf and hearing participants, respectively. The current project was approved by the Norwegian Centre for Research Data (NSD; #600573). All participants have signed consent forms, stating their willingness to participate in the project, to be video-recorded, and for the data to be used in research and teaching outputs. The data was annotated in ELAN, a software widely used for signed and spoken language analyses (Crasborn and Sloetjes 2008).

Open non-anonymized data is an important contribution of the current study, as it provides the field opportunities of extended discussions on real-life interpreted interaction. For an overview of the data, see Table 1.

The interpreters are L2 learners of NTS. They are trained interpreters both with more than ten years of interpreting experience. Both interpreters frequently interpret in workplaces 1 and 2, respectively, so they are familiar with the other participants in the study. As topics discussed were not instructed but occurred naturally, there were limited possibilities of preparation for the interpreters.

3.2 Annotation and analysis

The transcription of multimodal, embodied discourse is necessarily a selective activity that depends on the relevance in connection to research questions (Mondada 2018) and theoretical standpoints (Ochs 1979). To answer the research question of how constructed dialogue in the signed source utterance is situated in a spoken language ecology, it was important to establish what semiotic strategies were used in both the original (signed language) utterance and the Norwegian rendition. Involving a spoken and a signed language in the same study, annotations represent two languages that represent markedly different transcription traditions.

Annotation systems of signed language data have developed since the beginning of signed language research (Stokoe 1960, 1996), and have been refined through the efforts of linguists working with signed language corpora for different signed languages (Cormier et al. 2012a; Johnston 2003). Multimodal interaction analysis considers linguistic and pragmatic resources (e.g., Goodwin 2000, 2010; Mondada 2013) and how all these different resources interact and contribute to joint meaning-making (Clark 1996). In what follows, annotation conventions used for signed and spoken discourse in this study are presented.

3.3 Presenting data

Framing interpreting as reconfiguring the semiotics of utterances requires a comparison of the distribution of semiotic resources and strategies in the source utterances with resources and strategies of the renditions. Consequently, each extract contains images both from the source utterance and from the interpreters’ renditions. For readability purposes, the interpreters’ renditions are highlighted in gray. The constructed dialogue sequences are identified with the prefix CD: <CD: WITHIN BRACKETS> on both the GLOSS tier and the Norwegian tier. For a complete list of annotation guidelines, see the Appendix. In what follows, four sequences will be analyzed and discussed.

4.1 Maintaining the semiotics

In the first example (Extract 1), the interpreter renders constructed dialogue in NTS as constructed dialogue in Norwegian. Image #1 shows gaze as co-establishing the enactment of constructed dialogue (Beukeleers and Vermeerbergen 2019; Ferrara 2019) and is thus an example of a depictive strategy. Gaze is often described as constituting an instance of constructed dialogue and constructed action in signed language research (Cormier et al. 2013; Dudis 2011; Metzger 1995; Shaffer 2012; Young et al. 2012). In the source utterance, a participant exemplifies how it would be typical in the deaf community to describe a person visually. The sequence is introduced with a conventional discourse marker, for example (line 01), after which she shifts into the depicting constructed dialogue sequence. In this sequence, she constructs the dialogue of an imagined person (lines 01 and 03, Figure 1).

Figure 1:

Extract 1.

The constructed dialogue sequence is realized by means of gaze direction (indicating imagined person in signing space), a forward head gesture, and a squint (image #1). The squint contributes to depicting the facial expression the imagined deaf person would have in the effort of describing someone. The forward head gesture may be an embodied signal: placing the head in a slightly different position to signal the dialogue stems from someone other than the utterer. After the depictive constructed dialogue sequence, there is a descriptive sequence: we know who that is (line 03). This sequence is descriptive due to the semiotic character of resources deployed: Interlocutors are no longer required to “imagine what it is like to see the thing depicted” (Dingemanse 2015: 950), rather, highly conventional lexical items with corresponding grammar is the main source of meaning making. The shift between the depictive constructed dialogue and this descriptive sequence is evidenced by facial expression and head position returning to neutral, while making eye contact with one of the other deaf interlocutors (indicated by the arrow, image #2).

Working into spoken Norwegian, the interpreter maintains the descriptive character of the discourse marker that introduces this sequence (glossed ONE EXAMPLE, line 01), rendering it for example (line 02). In addition to maintaining the depictive device of constructed dialogue, the interpreter also incorporates some of the embodied resources from the source utterance: she squints and displays a slightly side-tilted head position (image #3). The squint and head tilt accompany the whole constructed dialogue sequence (you know the large one with glasses, right, the one with the big curly hair, you know who that is?) in the spoken language rendition. Also, the interpreter produces a gesture that copies elements from the source utterance, and thus might be labelled an alignment gesture (Rasenberg et al. 2020) or gestural mirroring (Shaw 2019). Note the simultaneous interactional work done by the interpreter: signaling alignment with the deaf participant while rendering an utterance directed to the hearing participant (Dittmann and Llewellyn 1968; Haug et al. 2017; Llewellyn-Jones and Lee 2014).

The depictive constructed dialogue source utterance is preceded by a descriptive discourse marker (glossed ONE EXAMPLE) and followed by a descriptive utterance then you know who that is, we are visual, you know (glossed KNOW WHO PERSON VISUAL). This way of preparing and concluding the constructed dialogue sequence is also found in the rendition, making the semiotic character of the source and target language similar (see images #1–#4).

While the referential content of then you know who that is, we are very visual, you know, is quite similar to the lexical parts of the NTS source utterance (glossed KNOW WHO PERSON VISUAL), some discourse markers are added in the rendition: jo (‘yes’) twice, and ikke sant (‘right’). The pragmatic particle jo (‘yes’) has been described to suggest that something is a given, or mutually manifest, and has the function of reminding the interlocutor of their “shared conclusion about the world” (Berthelin and Borthen 2019). Thus, two different discourse markers (and an accompanying hand gesture) mutually work to establish common ground (Brennan and Clark 1996; Clark and Schaefer 1989; Clark 2005). It might be seen as “securing the orientation of a hearer” (Goodwin 2000: 1499) as part of the process of properly situating an utterance in a different ecological environment.

4.2 Maintaining the semiotics with expanded descriptive sequence

In the second example (Extract 2), Participant 2 (P2) gives an example of what audism (discrimination on the basis of hearing status) can look like in practice, that an imagined employer would come up with reasons not to hire a deaf applicant. Resources of the constructed dialogue sequence include gaze towards the signing space, facial expression (furrowed eyebrows) and leaning to the right (extract 2, line 01). Participant 2 displays readiness to yield the floor towards the end of this sequence, evidenced by resuming eye contact with her interlocutor (line 03). I suggest she simultaneously signals a shift of main semiotic strategy, from depiction to description; she uses a conventional discourse marker that is glossed SO-ON (line 03) and the facial expression and body lean that were used as depictive resources are now neutral (line 03). See images #1 and #2 for demonstrations of the depictive and the descriptive semiotic strategies, respectively (Figure 2).

Figure 2:

Extract 2.

Again, we find gaze as a constitutive resource of the constructed dialogue sequence. However, note the gaze going back and forth between the interlocutor and the signing space (lines 01 and 03), before resting at the interlocutor when the sequence of constructed dialogue is finished (line 03). The subject could indicate some interactional processes: P2 is explaining to P3 (hearing participant), how a hearing employer is likely to treat a deaf applicant. Thus, the pattern of gaze behavior closely aligns with Janzen’s (2019) description of view pointed spaces in ASL narrative discourse. Janzen argues this is an intersubjective choice “in that the signer in effect asks the addressee to see the story as she is seeing it” (Janzen 2019: 256). Seeing linguistic choices as contributing to intersubjectivity is in line with treating semiotic strategies as interactional. In this specific situation, it is difficult to conclude whether the deaf participant’s gaze is directed towards the interpreter or the hearing interlocutor. Thus, the gaze behavior could also reflect an increased need to monitor interpreters, previously reported from a study on deaf leaders’ strategies when working with interpreters (Haug et al. 2017: 120).

In the rendition, the depictive resource constructed dialogue from the source text is rendered as constructed dialogue, i.e., the semiotic strategy of depiction is maintained (lines 02, 04, and 05). The interpreter’s forward lean reflects a similar body lean behavior from the source utterance (lines 02 and 04, image #3). The shift of main semiotic strategy from the source utterance is also present in the rendition following the constructed dialogue sequence, which is evidenced by (at least) three visual resources: A hand gesture (line 05, image #4), a shift of facial expression, (raised eyebrows to neutral, see images #3 and #4) and a backwards movement of the head (lines 05 and 06). The shift of eyebrow position corresponds with that of the NTS source utterance, with a slight difference: they are not raised to neutral as in the source utterance but furrowed to neutral (images #1 and #2).

The observed differences in head position, eyebrow position and the gesture, are in this study framed as contributing to an embodied shift of semiotic strategy, from depictive constructed dialogue to a descriptive sequence.

Note that the rendition of the single sign concluding the source utterance (glossed SO-ON, line 03) is substantially longer: right, one has such attitudes to it (line 6). While the verbal rendition has undergone significant changes compared to the source utterance (SO-ON), the descriptive character of the discourse marker is maintained. SO-ON may indicate a given, that further elaboration is unnecessary. In the rendition, the Norwegian discourse marker does similar work: ikke sant (‘right’) has been described as establishing common ground with the interlocutor (Svennevig 2008). However, the continuing sequence has an explanatory character: one has such attitudes to it. One might say this prolonged rendition signals the opposite: that more work needs to be done until sufficient common ground is attained. I suggest the interpreter detects an ecological mismatch as the depictive strategy of constructed dialogue is more prevalent in spoken Norwegian discourse among adolescents (Opsahl and Svennevig 2012). Hence, she might have incorporated the extended descriptive rendition to situate the utterance more appropriately. The descriptive part of the rendition is accompanied by a neutral body position (from a forward lean), and then a salient backwards head movement (line 06).

Summing up, the reconfiguring of embodied resources from the source utterance reflects the semiotic shift between a depictive and a descriptive main strategy with slightly different realizations in form. The verbal component of the concluding descriptive sequence is extended compared to the source utterance, possibly as an attempt to situate a depictive resource into a language ecology with differing norms for depictive strategies. While constructed dialogue in NTS is unmarked as adult discourse, it may signal more adolescent discourse in spoken Norwegian. The interpreter must take this difference into account when making her choices.

4.3 Altering the semiotics: mainly descriptive discourse

In this third example (Figure 3), the constructed dialogue sequence in the source utterance is short; it consists of only three signs. Again, this sequence is preceded by a descriptive utterance in which participant 1 explains what it was like for her parents, finding out that she was deaf. She introduces the sequence with: At the time I was born (…) (line 01). This descriptive sequence is mostly accompanied by eye contact with the hearing interlocutor. She describes how the field of deafness was completely unfamiliar and new to her parents, enacting how her parents would ask what deafness is: what is this deafness all about? (NTS utterance glossed WHAT DEAF WHAT) (line 03).

Figure 3:

Extract 3.

In line 03 (image #2) the gaze is directed towards signing space, where she has earlier defined an area that represents deafness (glossed AREA). Also, the head movement, the repeated head shake (line 03), contributes to expressing the parents’ stance towards deafness, they were not familiar with it. This sequence exemplifies how, in discourse, one might swiftly shift into and out of constructed dialogue. Although two semiotic strategies can be identified in this sequence (mainly descriptive or mainly depictive), the shift into the depictive strategy seems to be preceded by a preparation phase: The gaze towards the signing space and the tilting head movement start a few signs earlier than the actual constructed dialogue sequence: both the head movement and fixed gaze towards the signing space emerge with a negating sign (glossed NEG) (line 03).

Turning to the interpreter’s rendition, the constructed dialogue sequence is not rendered as constructed dialogue, but altered into a descriptive sequence, again substantially longer than the source utterance: my parents were totally unfamiliar with the situation, they knew nothing about deafness and such (lines 04–06). The interpreter’s visual cues are consistent throughout this utterance, not reflecting the semiotic shift that occurs in the source utterance.

The only reflection of a depictive device deployed by the interpreter might be read into her slightly furrowed eyebrows (image #3), which might be interpreted as an expression of the unfamiliar stance (lines 02, 04, 05, and 06). Throughout the utterance, there is no real shift to be detected regarding eyebrow position or other facial expressions, and the verbal rendition is consistently representative of descriptive resources. Thus, the depictive and descriptive resources from the source utterance are reconfigured into a mainly descriptive rendition with some depictive visual resources co-occurring.

4.4 Altering the semiotics: joint utterance

The last example is also short: participant 3 demonstrates how he would subtly puff his cheeks to refer to someone who was overweight (Figure 4). This mouth gesture [puffed cheeks] is conventionally part of a lexical adjective with a manual part, referring to the size of someone or something (image #1). Wishing to remain subtle about this characterization of an individual, he explains that the manual part of the sign in some situations will be omitted. Thus, the constructed dialogue consists of two units: [puffed cheeks], and one sign glossed UNDERSTAND, asking an imagined interlocutor if they understand who is being referred to. Again, gaze is an important resource, fixed towards a point in signing space slightly before the puffed cheeks, see Figure 4.

Figure 4:

Extract 4.

After the constructed dialogue sequence, eye contact is resumed with the interlocutor with a descriptive explanation: then they understand (line 03).

Rendering this last sequence, the interpreter alters the depictive resource into descriptive and indicative resources. If I just do like this with my face contains highly conventional lexical items, which are descriptive in their semiotic character. Note however, the deictic expression like this which needs an indicative resource to be meaningful. The interpreter indicates the mouth gesture with a hand gesture towards her own face, without copying the mouth gesture. Since this hand gesture is completely invisible for the hearing non-signing participant (her gaze and head are directed towards the signing participant, leaving the interpreter behind her, see image #3), it seems she combines the visual resource from the source utterance [puffed cheeks], with her own indicative resource, a deictic expression (like this). The interpreter incorporates resources (a depictive facial expression) from the source utterance, indicating it with a deictic expression, making the rendition a joint utterance between them. This joint utterance is created in a specific ecology of the interpreted event. Note that the spoken Norwegian rendition does not overlap with the mouth gesture [puffed cheeks], as reflected in the annotation (see line 01 and line 04). How the interpreter ensures that the deictic expression like this is indicative of [puffed cheeks] thus remains unclear. The only evidence suggesting that the hearing participant makes the connection is the absence of any orientation towards it as problematic.

The ecology of an interpreted event includes the physical condition of seating arrangements: In this case sitting parallel to the signing participant. Thus, this example demonstrates that the choice of rendering constructed dialogue as a descriptive utterance with an indicative resource is based on a mix of linguistic, pragmatic, and physical conditions, i.e., the ecology and semiotic affordances of the interpreted event.