Identifying and describing functional discourse units in the BNC Spoken 2014

2.1 Phase I – speech events and categorical functions

Our inspiration for the segmentation coding framework came from the Hymesian notion of a speech event; we began by assuming that conversations could be segmented into speech events that existed within speech situations, above the level of speech acts and utterances. Speech events are “activities, or aspects of activities, that are directly governed by rules for the use of speech” (Hymes 1967: 19; see also Hymes 1972, 1974). Hymes never provided an operational definition for a speech event, so we turned to other sources for that. We used aspects of Gumperz’s (2015) definition of a speech event as a starting point and began by assuming that each speech event will constitute a ‘text’, hence we incorporated Egbert and Schnur’s (2018) definition of a text into our operational definition of speech event:

1. Recognizably self-contained

2. Sequentially bounded with detectable beginnings and ends

3. Thematically and functionally coherent

We used these three criteria and attempted to segment conversational files into speech event units.

Three of the study’s authors attempted to independently segment several texts and then discussed agreement in their placement of speech event boundaries. This was moderately successful as there were some instances where unanimous agreement was achieved by all raters on the boundaries of speech events, and many instances where majority agreement was achieved. While the independent coding was useful, we discovered early on that multiple coders working through a conversation file together was also an effective way to make progress towards a framework. One source of disagreement among raters resulted from differing decisions about the most appropriate level of granularity for speech events, or whether to ‘lump’ or ‘split’, when in doubt. To address this, we established a minimum length of four utterances for speech events (later changed to five utterances). While the minimum threshold was inevitably arbitrary, it seemed to work well based on several rounds of coding. When a choice had to be made regarding whether to split a string of utterances into two speech events or lump them into a single speech event, we opted to favor splitting rather than lumping in an effort to fully identify and describe boundaries between functional units.

Subsequent attempts to identify speech event boundaries, both independently and in pairs, showed improvement in our agreement. To further improve reliability, we added an explicit guideline that a speech event boundary must coincide with a shift in communicative goal, not simply with a topic shift. This further improved our agreement. We quantified inter-rater reliability at this point using Pearson’s correlations, which we averaged across the purpose categories, to achieve r = 0.49.

At this stage, we had two primary goals: (1) attempt to identify recognizable shifts in communicative purpose to identify speech event boundaries, and (2) attempt to determine the primary communicative purpose for each speech event. Initially we did not have a taxonomy of communicative purposes to choose from. Rather, we were keeping notes on observed communicative purposes that could later be used for classification. Based on these observations, we compiled a list of observed communicative purposes to serve as a starting point for the functional annotation. We benefited from the speech event categories in the taxonomy of speech event functions proposed by Goldsmith and Baxter (1996), though many of the categories in their framework were defined at a more granular level than we were interested in, so we only included communicative purposes that we observed in the coded BNC files. For example, they had separate categories for ‘morning talk’ and ‘bedtime talk’. They also had categories that were defined topically, not functionally, such as ‘class information talk’ and ‘current events talk’.

We began with 12 communicative purposes, which changed many times before the final version. From this point forward, rounds of independent coding included two major steps: segmenting the files into speech events and characterizing those speech events by assigning a communicative purpose category to the segment. It is important to note here that in this phase we only allowed for one communicative purpose per speech event, coded dichotomously (present or absent). We also developed a scheme for XML markup that we used to annotate corpus files for segment boundaries and communicative purpose code. We also introduced the option of coding a file segment as a non-functional speech event to annotate sequences of utterances that did not satisfy the three criteria for a speech event.

Further coding led us to modify, based on our experience with the data, the definitional criteria for a speech event to:

1. functionally coherent: A speech event is a sequence of utterances characterized by a single dominant communicative goal.

2. sequentially bounded: A speech event has an identifiable beginning and end.

3. length requirement: A speech event must be a minimum of five utterances or 100 words.

Based on these changes, we carried out further rounds of coding. After each round of coding, the coders would meet to discuss and review differences in segmentation and annotation decisions. Coding was done using an XML markup scheme that required coders to add opening and closing tags for every speech event, with a single communicative purpose code added to each opening tag.

We arrived at a set of 10 communicative purposes,^[4] with abbreviated codes, not including categories for ‘unknown’ and ‘non-functional speech event’. All functions were informed by observations of the distinctive functions in the conversations themselves. The segmentation agreement among coders was quite high, with our segment boundaries falling within 1–3 utterances the majority of the time, but we found it extremely difficult to reliably assign speech event segments into a single communicative purpose category. One reason for this is that it often seemed plausible there could be more than one communicative purpose for a single speech event, leading us to seriously reconsider our approach and enter into the second phase of our framework development.

2.2 Discourse units and continuous purposes

Phase I established an operational definition for speech event, tested methods for reliably identifying speech event boundaries, and developed a taxonomy of communicative purposes that were observed in conversations in the BNC-S 2014. As a result of our observations during Phase I, we decided to make major adjustments which would refine and improve our methodological framework for the segmentation and characterization. With regard to segmentation, we began to hypothesize that the construct of ‘speech event’, as it was used in previous literature, was inappropriate for the functional units we were observing in conversational discourse. Based solely on the writings of Hymes and Gumperz, it is unclear what the differences are between ‘speech events’ and ‘speech acts’. Moreover, most researchers adopting these terms have done so with the goal of establishing speech event types, where each of the types has a single purpose or function (e.g. Friginal et al. 2017; Gilner 2016). So we made the decision to abandon the term speech event as it was apparent we were now doing something quite different than others using that term, adopting instead the term Discourse Unit (DU) to refer to the focus of this study, i.e., functional segments of conversation.

With regard to the characterization, our observations during Phase I also raised questions about the validity of characterizing functional units using a single communicative function category. So we revisited recent research related to hybrid texts (e.g. Biber et al. 2015) and continuous situational parameters (e.g. Biber and Egbert 2018; Biber et al. 2020). Based on our experience of coding segments in Phase I, we considered the possibility that functional units could be better described using multiple communicative purposes coded continuously on an ordinal scale. We believed that this may help with agreement where coders identified different, yet plausible, communicative purposes. Relatedly, we considered the possibility that these communicative purposes would not all play equal roles in accomplishing the overarching goal of the DU. Thus the ordinal scale represents the degree to which each communicative purpose is used to accomplish the overarching communicative goal of the segment.

Based on these revisions to our framework, we developed a modified set of parameters for DU. A DU is a sequence of utterances that is:

1. Coherent for its overarching communicative goal, which is both the primary objective of a DU and the task that the interlocutors are doing with language in the DU. This goal is typically coupled with a single topic or theme. Each DU has one communicative goal (e.g. complaining about annoying co-workers; making plans for buying Christmas gifts). There is an open-ended set of specific communicative goals, and these are not coded or labeled in our framework.

2. Characterized by one or more communicative purposes, where a communicative purpose is a finite set of actions that serve to help accomplish the communicative goal of a DU. Communicative purposes are coded in our framework. A DU may rely on one or more communicative purposes. When present, communicative purposes are coded on a scale from 1 to 3.

3. Recognizably self-contained: A DU has an identifiable beginning and end.

4. Length requirement: A DU has a minimum of five utterances or 100 words.

We began piloting this new method, relying on the set of observed communicative purposes developed in Phase I but allowing for multiple functions to be coded on a single DU. Initially, we used a 6-point scale, based on Biber et al. (2020). However, after the first round of pilot coding we determined that six points was too many to make the necessary distinctions and achieve reliable results. Therefore, we adjusted to a 4-point scale for each of the communicative purposes, where:

• 0 = not present

• 1 = minor function

• 2 = major function

• 3 = dominant function

The primary difference between a score of ‘2’ (major function) and ‘3’ (dominant function) is that a ‘3’ could only be used with one communicative purpose per DU, designating the purpose with a score of ‘3’ as the purpose with the most important role in accomplishing the communicative goal. Not every DU was required to have a dominant purpose. Regardless of whether a DU had a ‘dominant purpose’ or not, there was no limit to the number of communicative purposes that could be coded with a ‘1’ or a ‘2’ so long as they were functioning to help accomplish the overarching communicative goal of the DU.

This new method was applied in two major steps: (1) Segmentation: identify the boundaries of a DU, and (2) Characterization: code each communicative purpose for the degree to which it is actively helping to accomplish the overarching communicative goal of the DU (0–3).

Excerpt 3 below is an example from the BNC-S 2014 of a DU with an overarching goal of learning about each other’s children. This includes multiple purposes that are present to varying degrees. These purposes include figuring out the exact age of Speaker B’s toddler, describing his fascination with a vacuum cleaner, and a narrative from Speaker A about similar behavior from their child in the past. Although these purposes are distinct from each other, they converge in this segment to create a single DU that is coherent for its overarching communicative goal.

Feeling satisfied with our framework as refined in Phase II, we attempted a systematic evaluation of inter-rater reliability for files coded by four of the study’s authors. This presented several challenges. Methods for inter-rater reliability assume that the objects being rated have fixed boundaries. As disagreements on DU boundaries rendered it impossible to directly compare sequences of utterances, at this stage we did not evaluate the reliability of segmentation boundaries. However, we did measure inter-rater agreement in communicative purpose categorization at the utterance level, simultaneously accounting for boundaries and classification reliability. While this did not directly provide information about the two constructs of segmentation and characterization separately, it did allow us to measure inter-rater agreement across all of the communicative purposes, and for each of them separately.

Inter-rater agreement was measured using Krippendorf’s alpha and Pearson’s correlations. Krippendorf’s alpha measured agreement across all four raters in dichotomous terms: presence or absence of a code for each communicative purpose. Pearson’s correlations measured agreement between every possible pair of raters in continuous terms: the degree of association in scores (0–3) for each communicative purpose. Correlation values were averaged across all of the possible pairs of raters for each of the communicative purposes. While these measures did not directly account for DU boundaries, taken together they did provide some insight into agreement among coders.

In our first pilot, we achieved an overall Krippendorf’s alpha value of 0.29, a “fair agreement” (Landis and Koch 1977), and a moderate mean Pearson’s correlation of 0.49. We also noted, though, that there was extreme variability in inter-rater agreement across communicative purposes, leading us to make refinements to the operational definitions for several purposes. A second round of pilot coding resulted in modest improvements to both Krippendorf’s alpha (0.33) and Pearson’s correlation (0.50) and much less variability across communicative purposes. For both pilot rounds, we noted that some of the communicative purpose categories were much more common in some files than others. Thus, the agreement estimates for the less frequent purposes were less robust.

We combined the quantitative measures of agreement with a qualitative, manual review of agreement across files. This was done using a vertically aligned spreadsheet for each file that had one column for each rater, one row for each utterance, and the communicative purpose code assigned by that rater for that utterance. This allowed us to evaluate (mis)alignment in DU boundaries and (dis)agreement in communicative purpose codes, helping us to make sense of the quantitative reliability results and determine systematic areas of disagreement. Crucially, it revealed some important trends that could not be observed from the quantitative measures. First, minor disagreements in DU boundaries had a strong negative effect on both quantitative measures of agreement, even though boundaries were not accounted for in any direct way. This was mostly due to entire DUs being assigned the same set of communicative purposes. Second, the difference between a communicative purpose code of 0 and 1 had a much stronger effect on the alpha statistic than a difference between 1 and 2 or 2 and 3 because this measure only accounted for presence or absence, not the full ordinal scale. Third, and most importantly, there were many cases where raters disagreed, yet after closer inspection and discussion we determined that both coding decisions were equally plausible.

This final observation led us to the conclusion that quantitative measures of agreement or reliability were not accurately reflecting the nature of the task as they assume that there is one objectively correct answer that raters are attempting to converge on. Our observations up to this point in the project suggest that (1) there is considerable evidence that there is an objective reality to functional DUs, but (2) there is a permissible degree of flux that exists in the boundaries between DUs and the degree to which communicative purposes are present within those units. The constructs in our study, and the way we have operationalized them—DU boundaries, presence of communicative purposes, extent to which purposes are helping to achieve the overarching communicative goal of the unit—are all, to some extent, dependent on the observer’s perception of the nature of the interaction and the intents of the speakers involved. Hence, we decided that the ultimate goal was not perfect agreement among independent coders. Rather, our aim was for independent coders, trained using the same framework, to make coding decisions that other trained coders would deem plausible. So our goal is for raters to achieve segmentation boundaries and communicative purpose ratings that would be deemed plausible by another trained rater. Thus, plausibility checks are the basis for feedback to raters-in-training and for evaluating the degree to which segmentation and characterization has been successful. Moving forward, however, we also plan to conduct large-scale evaluations of inter-rater agreement. This will be done using the methods presented in Section 2.2, as well as other methods we are currently exploring that will allow us to account for agreement in (1) segmentation boundaries and (2) the dichotomous (presence/absence) of communicative purposes.

Additional rounds of pilot coding were carried out to test our new plausibility criterion and to make further refinements to the coding framework. We are quite satisfied with the plausibility criterion. It can be usefully applied at all three steps of the coding process: (1) plausibility of DU boundaries, (2) plausibility of communicative purpose presence, and (3) plausibility of communicative purpose degree. Checks at these three stages have been an effective means of providing feedback to new coders during the training process and are a useful method for more experienced coders to periodically calibrate their coding with others to ensure that there is no drift in their understanding and application of the coding framework and communicative purposes.

The final stages of pilot coding resulted in small changes to the parameters of communicative purpose. We also began annotating where both recording-related talk (language that refers to the task of recording the conversation) and foreign language (any language other than English). These do not take the place of communicative purposes, rather they are added as XML markup to the unit where the recording-related talk or foreign language were observed. These annotations can be used in both DUs and non-functional DUs. As noted, in the early stages of developing our coding framework we observed that most, but not all, normal conversation is structured and organized as functional units. We consistently observed stretches of conversation that did not function as a coherent DU. This is not merely an artifact of our length requirement. While non-functional DUs are often small, they can also be more extended. We believe there are important theoretical implications of both the discoveries that (1) most utterances in a conversation can be organized into larger units of conversation and (2) this is not always the case and there can be portions of conversation that do not function together as larger units. An example of a non-functional DU can be seen in Supplement C.

2.3 Final coding framework

We now describe the final coding framework, introduce and illustrate the communicative purposes, and provide an example of conversation coded using the framework. Supplements A and B represent the sum total of our work over many months to develop a comprehensive method for identifying and describing functional units of conversational discourse. Our final coding framework and instructions are included in Supplement A. This document contains the full set of guidelines that coders use to (1) segment transcribed speech files into DUs, and (2) characterize those DUs according to their communicative purposes. These guidelines are divided into three sections. Section one provides operational definitions for DUs, communicative goals, and communicative purposes, and contains a listing of the set of communicative purposes in the framework. Section two contains the full set of steps that coders follow when segmenting and characterizing DUs. Section three establishes the XML markup coders use when annotating the corpus files.

The full set of communicative purposes are contained in Supplement B. This contains labels and definitions for each of the communicative purposes in the framework. We will briefly describe and illustrate each of these nine communicative purposes here. It should be noted that the examples are meant to demonstrate when a particular purpose would be present. The degree to which the purpose is present, as well as the possible presence of additional purposes, varies from example to example.

1. Situation-dependent commentary (sdc). Occurs when speakers in a conversation are commenting on people or objects that are present, or events that are occurring in their shared situational context. Examples include (1) commentary on the unsafe driving practices of another driver at the petrol station where they are waiting for their turn at the pump, and (2) conversation about rules and strategies in a board game as it is being played.

2. Joking around (jok). Conversation that is intended to be humorous, including both lighthearted and darker humor. It also includes good humored banter, teasing and flirting. Examples include (1) a hyperbolic comparison between a bad tasting pie and sawdust, and (2) one speaker teasing another because her blouse was being worn inside out.

3. Engaging in conflict (con). Includes disagreement of any type, including lighthearted debate as well as more serious quarreling. Examples include (1) a debate over which key on a key ring fits which door in a house, and (2) a friendly disagreement over which of the two speakers is more likely to become rich one day.

4. Figuring things out (fto). Discussion aimed at exploring or considering options or plans, including discussion about how things work and what the best solution to a problem may be. Examples include (1) discussion about the appropriateness of visiting in-laws after a spouse’s death, and (2) attempts to understand and explain the recent behavior of a mutual acquaintance.

5. Sharing feelings and opinions (fel). Discussion about feelings, opinions, and beliefs, including the airing of grievances and the sharing of personal perspectives. Examples include (1) an explanation to justify a speaker’s preference for an item of clothing, and (2) a discussion about political views.

6. Giving advice and instructions (adv). Occurs when one speaker offers directions, advice, or suggestions to another speaker. Examples include (1) one speaker helping another to navigate a website to order tickets by giving step-by-step instructions during the process, and (2) one speaker offering suggestions to another on the best kind of copy paper to purchase

7. Describing or explaining the past (pas). Narrative stories about true events from the past or other references to people or events from the past. Examples include (1) a speaker telling stories from a favorite vacation, and (2) two speakers reminiscing together about the past while sorting through boxes stored in the attic.

8. Describing or explaining the future (fut). Descriptions or speculations about future events and intentions, including those that are planned and those that are more hypothetical. Examples include (1) one speaker describing plans for a date with a significant other, and (2) two speakers sharing their plans for life after graduation from university.

9. Describing or explaining (time-neutral) (des). Descriptions or explanations about facts, information, people or events where time (past or future) is either irrelevant or unspecified. Examples include (1) a speaker responding to another’s questions about the progress of house renovations, and (2) a description of the difference between two products.

Each coder in the study participated in an extensive training process. During this process, one or more of the project leaders described the broad aims of the study and the design of the BNC. Coders were then introduced to the coding framework and the list of communicative purposes, and saw multiple examples of speech files, both uncoded and coded. Each coder then performed multiple rounds of coding. After each of these rounds, project leaders reviewed the coding for plausibility and offered feedback during follow-up meetings. Once a coder’s work was consistently achieving high levels of plausibility, they were assigned a batch of files to code independently.

Excerpt 4 below is an example of a sequence of two DUs and a non-functional DU. For readability, all XML markup has been excluded. The same segment is included in Supplement C with all XML markup.

In DU 1 two speakers are commenting on an advertisement they see in a shop for a cow that is for sale. In addition to the dominant purpose of ‘situation-dependent commentary’, this DU also includes the purposes of ‘figuring things out’ and ‘joking around’. The speakers then briefly give their attention to commenting on something different, but this was coded as a ‘non-functional DU’ because it did not meet the minimum length requirement and it does not function as part of either of the two adjacent DUs. In DU 2, the speakers each share similar experiences from earlier that day when a mutual acquaintance, presumably a young child, waved at them. This was coded with the dominant purpose of ‘describing or explaining the past’, along with the major purpose of ‘sharing feelings or opinions’ and a minor purpose of ‘joking around’.