Corpus-based discourse analysis: from meta-reflection to accountability

3.1 The initiative

Making code available arguably constitutes a relatively recent trend in corpus-based DA and is hampered by a computational skills shortage. Even where the code is made available, the question is how many corpus-based discourse analysts have the skill to read, evaluate, and/or implement this code? To address this skills shortage and to increase transparency, researchers in Australia are collaborating on the Australian Text Analytics Platform (ATAP). While ATAP does not exclusively focus on corpus-based discourse analysis, we want to take this opportunity to introduce this initiative as potentially helpful to this field.

In brief, ATAP is an initiative which fosters cooperation among data and text analytics users and providers, and supports researchers from diverse academic backgrounds, including beginners, in adopting user-friendly code-based text analysis. It involves collaboration among the Universities of Queensland (including the Language Technology and Data Analytics Laboratory/LADAL), Sydney (Sydney Corpus Lab, Sydney Informatics Hub), and Australia’s Academic and Research Network/AARNet. It is also connected to the Language Data Commons of Australia HASS RDC through which it collaborates with additional institutions/organizations and supports language-related technologies, data collection infrastructure, and Indigenous capability programs. ATAP’s primary goal is to enhance research flexibility, offer valuable upskilling resources, and improve research workflow transparency and reproducibility. It thus offers a range of resources for text data collection, processing, visualization, and analysis (e.g. training, office hours, collaborations). Code-based resources include the development of user-friendly, shareable, transparent, and interactive Jupyter notebooks. Before we introduce an ATAP notebook that may be of particular interest to corpus-based discourse analysts, we provide an explanation of such notebooks given that they may be unfamiliar to many researchers in this field.

3.2 Jupyter notebooks

Jupyter notebooks are free open-sourced online web applications that are designed to create and share documents which combine text with executable code and the output of the code. These notebooks support a variety of programming languages including Python, Julia, and R. They are often stored on GitHub and other online repositories. In September 2018, there were more than 2.5 million public Jupyter notebooks hosted on GitHub alone (Perkel 2018: 145). Jupyter notebooks can be uploaded to online platforms which allow to run the code in notebooks independent of personal computers or they can be run locally (i.e. without connecting to the Internet), although extra setup is required for this. It is also possible to convert Jupyter notebooks to other file formats (e.g. html, LaTeX, and pdf), which makes them shareable outside of wherever they are stored (Beg et al. 2021: 38).

In general, a notebook is composed of cells (i.e. areas of the notebook), of which there are three types: code cells, text cells, and raw cells. Code cells contain code which can be executed to run analyses, output results from analysis, produce visualisations, etc. Text cells make use of markdown, a method for formatting text. Raw cells are mainly used for configuration, e.g. for specifying if a notebook should be converted into a pdf or html document (Pimentel et al. 2021: 4). Thus, Jupyter notebooks typically contain both computer code (in the relevant programming language) and text elements (paragraphs of text, links, equations etc.). Code cells are (usually) preceded by explanatory text (e.g. description of the code). What the code produces (the output) is presented just below the code cells. This means that notebooks are simultaneously human-readable and executable documents. Thus, a notebook can be summarised as “an interactive literate programming document and an application that executes the document” (Pimentel et al. 2021: 3).

Jupyter notebooks are used in ATAP because they have several advantages: they make it easy to document, share, and reproduce codes used for analyses (Pimentel et al. 2021; Shen 2014). With notebooks, it is possible to carry out an entire study within a single document while maintaining a complete and executable record of the processes involved (Beg et al. 2021: 38, 40). Due to the ability to contain text and tables, notebooks can also provide information about annotation schemas, provide or discuss examples as well as automatically store and document technical information (Beg et al. 2021: 40).

The combination of explanation and code allows analysts to describe the intended process and outcomes of the code which, in turn, allows others to better understand how it works. This is helpful for troubleshooting if any errors are encountered when the code is adapted. Finally, it is possible to modify and then immediately execute/run the code. This means that it is possible to see the results of the modifications which lead to a better understanding of the code. Because of these features, Schweinberger and Haugh (in press) argue that the use of notebooks in linguistics can assist in making corpus analyses and the workflows they are based on more transparent, reproducible, and efficient.

To date, ATAP has developed a range of Jupyter notebooks and is developing more. A full list is available at https://www.atap.edu.au/tools/, with tools addressing different researchers including those interested in corpus-based DA, but also those interested in stylistic or literary analyses or those interested in natural language processing (NLP) techniques such as topic modelling, sentiment analysis, document classification, geolocation, etc. Some notebooks can be considered as upskilling tutorials (e.g. those on the LADAL platform which cover a broad range of areas), while others are stand-alone tools. The latter tool suite includes:

1. Document Similarity Tool – enables the identification of duplicated content among the texts in a corpus and permits the exclusion of duplicates after manual review.

2. Semantic Tagger – can be used to add semantic tags to the texts in a corpus, undertake analysis/visualisation and export tagged texts.

3. Keywords Analysis – keywords analysis on two or multiple corpora, as well as statistical tests to investigate word use across corpora.

4. Quotation Tool – identifies who is quoted as well as quoted content in newspaper corpora, integrating named entity recognition.

While most of the notebooks are relevant to different text types, the Quotation Tool (Jufri and Sun 2022) targets newspaper texts – a very common data source in corpus-based DA. Given that quotation extraction is a type of analysis not normally enabled in “classic” off-the-shelf corpus software but is of clear relevance to corpus-based DA, we will introduce this tool in more detail below.

4.1 Introducing the Quotation Tool

We have chosen the Quotation Tool as a sample notebook because of the relevance of quoted speech to corpus-based DA. In Baker et al.’s (2008: 282) list of strategies of representation (adapted from Wodak 2001), quotation falls under the strategies identified as “Perspectivation, framing or discourse representation”. As they note, “Quotation patterns […] play an important role in implementing particular perspectives, and hence, ideologies” (Baker et al. 2008: 295). Therefore, it is important to analyse “who is written about, how much space they are given and whether they are directly or indirectly quoted” (Baker et al. 2008: 294), and whether sources are represented negatively, for instance as “inarticulate, extremist, illogical, aggressive or threatening” (Baker et al. 2008: 295). In other approaches to discourse analysis, reported speech and thought is also regarded as important, in particular in newspaper discourse (see Bednarek 2016: 31–32; White 2012). Fairclough (1988) is a classic CDA paper that introduces some of the relevant issues. Bednarek (2016) suggests that there is general agreement that the following questions can be differentiated in relation to the integration of voices (sources) in news discourse:

Whose voice is it (who is the source)? How are they identified? Where are they located and how do we access them? How are these voices integrated? What reporting expression, if any, is used to introduce the content?

The Quotation Tool is of clear relevance to these questions, because it allows users to upload a news corpus and to automatically identify who is cited (the source), to classify this source (as entity type), to identify which reporting expression is used to cite them (e.g. according to, said, claimed), the type of quotation used to integrate the reported content (e.g. direct, indirect), and to extract the quoted content itself (for additional analyses, e.g. manually or as corpus in another tool). It then becomes possible for discourse analysts who examine larger datasets (which would not be amenable to manual analysis) to identify what sources are cited, how these sources are identified, whether readers hear/experience what sources said or a paraphrase by the journalist, and which sources and which content is associated with neutral versus other reporting expressions (e.g. those expressing attitude like admit, warn or those expressing potential doubt like claim). This enables analysis of whose perspective is included and whether there is any bias in any quotation patterns. For discursive news values analysis (Bednarek and Caple 2014, 2017), the Quotation Tool can inform analysts of whether the cited sources are “ordinary people” (news value of personalisation) or “elites” (news value of eliteness).

Before we illustrate some of the results from using the Quotation Tool, we first briefly explain its basic features, using a small training dataset/corpus comprising 100 articles (utf-8 encoded text files) from the University of Sydney’s student newspaper Honi Soit. We collected one article per week from the newspaper’s online News section (http://honisoit.com/category/news/) over a period of 2 years (2021–2022), as explained in Lee (2024). The Honi Soit editors gave us permission to use this corpus for training purposes.

The Quotation Tool notebook has been kept deliberately simple and contains just five steps:

• Step 1 – Setup: imports and initiates the Quotation Tool and the necessary libraries;

• Step 2 – Load the data: uploads the corpus file, including previewing its contents (cf. Figure 1);

• Step 3 – Extract the quotes: extracts the quotes and applies named entity recognition to speakers and quoted content (but not other elements); includes preview of the extracted data (not shown here);

• Step 4 – Display the quotes: shows the results of the analysis in one text (Figure 4); allows analysis of whole corpus for the most frequent (top) entity names (e.g. John Doe) and entity types (e.g. PERSON) in speakers and quoted content; visualisations of the latter can be exported (Figures 2 and 3);

• Step 5 – Save the quotes: saves the results into a spreadsheet for downloading to users’ computers and additional analysis outside the Quotation Tool notebook environment.

Figure 1:

Previewing corpus contents in the notebook.

Figure 2:

The five most frequent speaker entity names.

Figure 3:

The five most frequent speaker entity types.

The quotation extraction has been adapted (with permission) from code developed by Maite Taboada’s Gender Gap Tracker team (e.g. Asr et al. 2021), while other tools used in the notebook include spaCy (https://spacy.io/), Natural Language Toolkit (https://www.nltk.org/index.html), pandas (https://pandas.pydata.org/) and Jupyter Widgets (https://ipywidgets.readthedocs.io/en/stable/). Before we demonstrate uses of the Quotation Tool further (in Section 4.3), we provide a brief discussion of some of the notebook features with respect to transparency of analysis (falling within the second row “Methods, techniques, analysis” in Table 1).

4.2 Reflection on accountability and transparency

How does this notebook foster accountability in relation to transparency of analysis? On a general level, such analytical transparency is facilitated by the fact that this notebook contains both explanations and code. This allows users to understand what action the relevant code performs and would be particularly important for users who do not read code. The explanations also include information on how the code can be changed by users, for example changing the number of rows that are previewed by changing the “n” variable. This makes the code more transparent, as the “n” variable is both explained and made adaptable. However, it must be acknowledged that some of the code will still be non-transparent to those who cannot read it. They can understand what the code achieves, but cannot evaluate the code itself for how it achieves the outcome. A final general feature is the use of previews, which allow users to see the extracted data and identify any potential issues with it. Such previews can also make the code more transparent, as results are displayed and can be inspected (cf. Figure 4).

Figure 4:

Extract from text preview showing results of quote extraction and named entity recognition.

In addition to these general features, this notebook includes several links to resources where users can read about (aspects of) the underlying code used to extract the quotes. Figure 5 shows the link to the relevant team’s GitHub page as well as to an article that evaluates the Quotation Tool’s accuracy. Elsewhere in the notebook, users are able to directly access an appendix (from the Gender Gap team), which contains technical information about the quotation extraction pipeline.

Figure 5:

Introduction to the Quotation Tool notebook.

The notebook also includes various coloured boxes which provide information about the libraries that are installed (Figure 6) and the tools that are used (Figure 7). In some cases, additional information about the relevant tool is also provided, for instance with respect to how texts are split into tokens (Figure 7) or with respect to capitalization (Figure 8). This is important so that users can understand the choices that are integrated within the tools, to be able to interpret the output, and to identify potential limitations.

Figure 6:

Information about installed libraries.

Figure 7:

Information about tools and tokenization.

Figure 8:

Information about capitalization.

Other coloured boxes contain important information for users, such as providing a definition of each column header in a table or information about how to deal with particular issues from a technological perspective (e.g. large file upload).

In sum, transparency is addressed through several different measures, either directly within the notebook or by providing users with links to relevant documentation outside the notebook. When users do not change the code but use it with all default values, this analysis should be reproducible (with the same dataset) and replicable (with a different dataset). Where users change the code – as they are able to – they would need to take responsibility for documenting the changes that have been made in a transparent way to ensure reproducibility and replicability. One remaining disadvantage is that some notebook users will not be able to read and evaluate the code itself, but will have to rely on the accompanying material and its accessibility to them. In addition, there is a trade-off between including too much and too little information in the notebook itself. Too much explicatory text within the notebook may overwhelm the notebook user, while links to documents outside the notebook require additional effort by the user and may deter them from engaging with the relevant information. In general, we have opted for a mix of both.

4.3 Potential uses in corpus-based DA

Having discussed its general features, this section now illustrates some potential uses of the Quotation Tool in corpus-based DA. One of the obvious uses of the notebook is to analyse a specific corpus for its quotation patterns and to interpret these patterns. This seems a relatively straightforward use of the tool, which we only briefly discuss here. To give some examples, the downloaded spreadsheet (produced in step 5) could be filtered by reporting expression and all sources and propositions could be retrieved that are cited using negative attitudinal reporting verbs such as WARN or ADMIT. These could be compared to those cited with neutral reporting expressions. Potential classification systems that can be used as starting point for identifying reporting expressions can be found in Caldas-Coulthard (1994) or Bednarek (2006: 57–58). Similarly, one could compare sources cited directly with those cited indirectly, or one could identify and classify all the sources and types of entities that are cited. Thus, relevant quotation patterns could be retrieved in a corpus of news about a particular topic and analysed with respect to bias and ideology.

Additionally, the Quotation Tool can be used to obtain information about journalistic norms or conventions that may hold across different newspaper corpora. This can be useful as background information for discourse analyses of various types. We will illustrate this use of the tool by extracting quotes from five different newspaper corpora:

1. The Honi Soit training dataset introduced above (100 articles from an Australian student newspaper);

2. A subset of the AusBrown corpus (Collins and Yao 2019), namely 169 articles from the Press section (139 articles from 1990; 30 articles from 2006);

3. The Cycling corpus described in Bednarek and Caple (2017: 138–143) containing 1,687 news items about cycling from 12 Australian, US, and UK newspapers in the years 2004, 2005, 2009, 2013, 2014;

4. The Diabetes News Corpus described in Bednarek and Carr (2019), consisting of 694 newspaper articles about diabetes (577 news items, 117 other items) from 12 Australian newspapers (2013–2017);

5. The Australian Obesity Corpus, including approximately 26,000 articles from 12 Australian newspapers that mention the words obese or obesity (both news and other items; see Vanichkina and Bednarek 2022).

These datasets all contain Australian newspaper items, although some include non-news genres (e.g. opinion) and some also include UK and US items. We used the Quotation Tool to compare a range of aspects across these corpora, with the aim of identifying the frequency of entities (as sources and in quoted content), of quotation types, and of reporting expressions. All results in Tables 2 –5 were retrieved by working with the downloaded spreadsheet, but can now be easily retrieved in summative form (see Australian Text Analytics Platform 2024).

We start by examining trends in entities (raw frequencies and ranks): Tables 2 and 3 show the results for speaker entities (sources) and quote entities (entities mentioned within quoted content) in the five corpora (excluding blank cells). These results are based on the six spaCy entity types included by default in the notebook (PERSON = people; ORG = companies, agencies, institutions, etc.; GPE = countries, cities, states; NORP = nationalities, religious or political groups; FAC = buildings, airports, highways, etc.; LOC = non-GPE locations, mountain ranges, bodies of water). Numbers derive from the named entity recognition, where the same source is sometimes classified as more than one entity, for instance in the case of role labels (e.g. NSW Police Minister David Elliott: David Elliott = PERSON; Police = ORG; NSW = GPE) and were retrieved using a text filter. It is evident that the three most highly-ranked speaker entities are the same across all corpora, with only small differences among the remaining entities. No doubt there are errors in the entity recognition; however, the ranking conforms to expectations, given that we would expect people to be sources more often than other entity types and given that ORG and GPE/NORP are likely associated with the construction of the news values of eliteness and/or proximity (Bednarek and Caple 2017: 82–83; 91–93). For quote entities, there is similar overlap in the rankings, with three newspaper corpora having identical rankings for all entity types and two corpora (Cycling; Diabetes) having slight variations. Thus, there are journalistic norms concerning the kinds of entities that are quoted as sources as well as those that are mentioned in quoted content.

We now consider the quotation structure, with Table 4 listing all quotation types and their (raw) frequency and rank in each corpus. The categories come from the quotation extraction developed by the Gender Gap team and are explained in Asr et al. (2021)’s supporting information. Briefly, AccordingTo = use of according to, S = speaker, V = verb, C = content of quote, Q = quotation mark; floating quote = quotation by same speaker without new reporting expression. As can be seen, there are also definite norms at play considering quotation types: SVC is ranked first across all corpora, while most other quotation types also show rankings that are either the same or similar (within one to two ranks) across corpora. Only CSV (ranks 2, 4, 5) and SVQCQ (ranks 5, 6, 8, 9) show slightly more variation, some of which may derive from the diversity of news genres/topics in each corpus.

It is also interesting to identify frequency patterns in the reporting expressions that are used, because such expressions can be tied to evaluation/attitude (e.g. Bednarek 2006; White 2012). Again, there is much overlap concerning the ranking of the ten most frequent reporting expressions in each corpus (cf. Table 5, excluding blank cells). For instance, said is ranked first in all corpora and is thus the most commonly used reporting expression, while the present tense form says is ranked second or third across all. Told (ranks 2, 3, 5) and according [to] (ranks 3–6) are also important in all datasets, while say has the same rank (#4) in all but the student newspaper (Honi Soit), which may not consistently follow established journalistic norms. Other differences may arise from genre or topic variation. In any case, there are clear usage norms that can be identified with the Quotation Tool, allowing us to identify common and uncommon quotation patterns in news against which a particular news corpus could then be compared.

In addition, we can use the notebook for the purpose of triangulation (of methods). We will illustrate this briefly with the Diabetes News Corpus. In a previous study (Bednarek and Carr 2021), we used WordSmith (Scott 2020) to identify the four most frequent reporting expressions in this corpus (said, says, according to and say; cf. also Table 5), and to classify each source (excluding pronouns) that occurs with these four expressions using a coding manual (https://osf.io/jrhx2/). The results of this original classification are shown in Table A.1 in the appendix. We can now use the output from the Quotation Tool to triangulate this analysis. To do so, one of the authors (Kelvin Lee) used the same coding manual to classify all speakers occurring in the “speaker” column in the downloaded spreadsheet. We filtered out pronouns (he, I, it, she, some, someone, that, them, they, this, we, which, who, you), blank cells, and instances associated with said, says, according to and say – leaving us with 658 instances to analyse (resulting in 659 codings, with one instance double-coded). The purpose of the triangulation was to see if the patterns would be similar with respect to additional reporting expressions; hence the exclusion of the expressions that were previously analysed.

Since we are reporting these results here purely for the purposes of illustration, inter- or intra-rater reliability was not measured. In addition to the categories from the coding manual we added two additional categories: “error” and “unclear”. The “error” category refers to cases where speakers were incorrectly identified by the Quotation Tool (e.g. mice), while the “unclear” category refers to cases where the spreadsheet output did not suffice to enable speaker classification. In such cases one would have to use an additional tool for analysis (e.g. a concordancer) or check the relevant text in the corpus. The results of this qualitative analysis are presented in Table A.2 in the appendix and show that frequency patterns of source categories differ somewhat, indicating that particular reporting expressions may be associated with particular source categories. When the results of both codings are combined (Table 6), however, the ranking of the source categories is virtually identical to that based on the most frequent expressions, with only health advocacy groups and lay people changing places (but very close to each other). Triangulation with the Quotation Tool thus gives us additional confidence in the original findings. All three tables are included in the appendix for further consultation.

A final use of the notebook for corpus-based DA is that we can use the quoted content itself as corpus to identify linguistic characteristics of such content. To do so, we take the column that contains the quoted content, extract this content, and turn it into a corpus. We can then analyse the five quoted content corpora for frequent features, for example part-of-speech tags, semantic tags, grammatical words, etc. (cf. Tables A.4–A.6 in the appendix). There is no room here to discuss these results adequately and further analyses would be necessary, but these tables indicate some similar trends regardless of the topic of the corpus. For instance, common POS tags (Table A.4) across all or most corpora include singular/mass nouns, prepositions/subordinating conjunctions, determiners, adjectives, personal pronouns, singular proper nouns, plural nouns, adverbs, verbs (in base form; past participle; singular present tense; past tense; gerund/present participle), and coordinating conjunctions. Excluding the grammatical bin and unmatched cases, the top 15 semantic tag lists (Table A.5) show a preference for Pronouns, Existing, Time: Period, Getting and possession, Personal names, Evaluation: Authentic (all corpora) as well as People, Likely, and Numbers (four corpora). The overlapping grammatical words (Table A.6) likely reflect general frequency trends in English, but do inform us about the importance of first person plural pronouns (we) in quoted content, as they occur among the top 15 most frequent grammatical words in all but one corpora. That these tendencies occur regardless of the topic informs us about general language trends in newspaper quotes on the whole.

In sum, the Quotation Tool has a range of potential uses for corpus-based discourse analysis of English-language newspaper texts, and we hope that this notebook – alongside other ATAP notebooks – will prove helpful to other researchers. Of course, manual analysis will be more accurate and is preferable for small text collections. In addition, the tool does work with medium-to large datasets (e.g. 26,000 articles) but may be slow or require several attempts, and it currently does not work with “big data”. Thus, the tool may be most suitable for medium-to large corpora that are not amenable to manual coding.