This is how to recognize conspiracy theory–based alternative news media: a comparative analysis of style, text structure, and argumentation in alternative news media and mainstream news media

1 Introduction

During the COVID pandemic, politico.com editor Zack Stanton (2020) described the present time as a “golden age of conspiracy theories”. This resonated with many people who felt that more people seemed to be open to or interested in alternative explanations for events and phenomena, including explanations involving malicious plots by powerful groups. Douglas et al. (2019]: 4) define conspiracy theories as “attempts to explain the ultimate causes of significant social and political events and circumstances with claims of secret plots by two or more powerful actors”. Belief in these conspiracy theories can result in negative consequences on both individual and societal levels. For individuals, conspiracy beliefs are associated with lower economic and social well-being and poor health behavior (Van Prooijen et al. 2023). Conspiracy beliefs have also been associated with negative outcomes like disengagement from mainstream politics and radicalized political views (Douglas and Sutton 2023). This does not only impact individuals, but affects society as a whole as well, thwarting the democratic process and contributing to polarization in society. With these serious consequences and – according to polls – almost 90 % of the US public endorsing one or more conspiracy theories (Hanley et al. 2023) it is clear that conspiracy theories play an important role in today’s society and are considered a major concern.

Studies on how to recognize and deal with conspiracy theories are conducted at an unprecedented rate (e.g., Demata et al. 2022a; Mahl et al. 2022; Tam and Chan 2023). Recent language technology studies have focused on automatic detection of fake news and conspiracy theories (Chen et al. 2023; Choraś et al. 2021; Samory and Mitra 2018). This results in useful models that computationally predict the veracity of texts, help fact checkers, and provide information about sources’ credibility. However, these models do not help people assess the reliability of online or offline texts while reading. Educational interventions can help to improve readers’ media and information literacy and in doing so improve their resilience to conspiracy theories. To develop these interventions, a more comprehensive and in-depth analysis of conspiracy theories’ linguistic features that can be recognized by nonexpert readers is required.

One context in which these readers encounter conspiracy theories (in liberal democracies) is in alternative news media (ANM). These ANM are often contrasted with mainstream media (MSM). MSM are characterized as bringing “routinized and professionalized journalism”, by “legacy news media organizations” (Holt et al. 2019: 861). These websites present what is often described as “authentic news”: “outlets that generally adhere to journalistic norms including attributing authors and correcting errors; altogether publishing mostly true information” (Hanley et al. 2023: 4). However, people who are skeptical towards MSM and feel that they are not represented by it, are likely to turn to ANM for their information (Schulze 2020). These people might see epistemological authorities (like governments and media) as “part of the conspiracy itself”, and seek out alternative narratives (Demata et al. 2022b: 2). ANM present themselves as “correctives of the mainstream news media” (Holt et al. 2019: 860). Vogler et al. (2024]: 1) add that ANM usually have an “affinity toward conspiracy myths”.^[1]

It is important to note, however, that because information from ANM is widely distributed via social media, even people who do not choose to visit ANM websites will probably encounter information from these websites online. How can we help these readers recognize ANM-based news? We decided to explore whether there are linguistic features that are specific to ANM or to MSM, or that are used in a distinctive way by one or the other. To do so, we created two sets of news articles from Dutch websites with an obvious ANM or MSM status. We explored the possibilities offered by linguistic features related to (1) style, (2) structure, and (3) argumentation to distinguish between ANM and MSM.

For all three domains (style, structure, and argumentation) many linguistic features could be hypothesized to be indicative of the media type (ANM vs. MSM). This study explores the potential of this type of analysis, by analyzing representative features per domain. For style, the use of clickbait in the headlines; for structure, the journalistic content in the first and the last paragraph; and for argumentation, the use of argumentation from authority in quotes.

Even in a short textual element like a headline, style might be indicative of whether an article is published on MSM or on ANM. In this case, we focus on clickbait elements in headlines. Clickbait tries to “capture attention by awakening one’s curiosity and/or arousing emotions, so as to prompt the reader to click on the headline” (Mormol 2019: 3). The use of clickbait has been linked to ANM (Palau-Sampio 2023). However, MSM’s online news has to compete for readers’ attention too, and might also show a “tendency toward interaction-cumulating content … which may reflect a universal appeal of clickbait” (Lischka and Garz 2023: 2088). Nevertheless, we expect ANM headlines to contain more clickbait elements than MSM headlines. A comparison of ANM and MSM headlines gives insight into which clickbait elements are used.

For structure, the journalistic content in the first and last paragraphs was analyzed. In order to do so, we focus on the typical information structure of news items: the inverted pyramid structure. Rafiee et al. (2023) have identified immediacy (starting with the most important information) and rollability (structuring the information in a text from most to least important) as important concepts in news writing, according to Dutch journalism handbooks. This means that the first paragraph should answer as many of the six wh-questions (who, what, when, where, why, and how) as possible. Less important information usually appears towards the bottom of the news item. The last paragraph should, however, still present information, rather than express opinions, because important characteristics of professional journalism are among other things noninvolvement and – according to Western journalists in particular – objectivism, and there should be no active promotion of particular values, ideas, or social change (Hanitzsch et al. 2011). If ANM and MSM both follow these guidelines, this would mean that they both answer a considerable number of wh-questions in the first paragraph and that the last paragraph should be informative, rather than present an opinion. Since ANM distance themselves from MSM, we expect to find that ANM adhere less to the guidelines described here.

For argumentation, we focus on quotations, because they can be used as arguments from authority. Quotes are often seen as adding credibility to news articles by (among other things) adding precision and emphasizing neutrality and truthfulness (Van Krieken 2020). In addition, quotes can be used persuasively as well (Smirnova 2012). By selecting certain quotes, journalists can “imprint their personal views on the events and ultimately serve an ideological function in the text” (Jullian 2011: 766). It is conceivable that ANM use quotes to present or even promote opinions, whereas MSM prefer to cite sources to be objective. This would mean that ANM and MSM differ in the way they emphasize the credibility of their sources.

By analyzing these features related to style, structure, and argumentation, we gain more insight into the question: what linguistic features in news articles can be used to distinguish between ANM and MSM?

2 Materials and methods

For this exploration, a corpus of 60 Dutch news articles was used, translated throughout this article by the authors themselves. Half of these articles appeared on MSM news websites. These articles were selected from the three biggest and most trusted (Commissariaat voor de Media 2023) free news websites in the Netherlands (nu.nl, nos.nl, rtlnieuws.nl). The other 30 articles were selected from three of the most well-known Dutch ANM websites with a focus on conspiracy theory news^[2] (niburu.co, worldunity.me, ninefornews.nl). The latter sites post multiple news articles per day and present themselves as a critical alternative to MSM. Often, readers come across such articles via social media and are unaware of their ANM origin (Klawier et al. 2021).

Ten articles were randomly selected per website, all published between 11 October 2023 and 10 November 2023. The mean length of the articles was 420.8 words (SD = 190.4). Texts and images were copied and placed in Microsoft Word files. For all websites, the same font was used in the Word files, and if the website’s name was mentioned, this was replaced by “[THIS WEBSITE]”, to prevent the coders from recognizing the news website. Articles were presented to the coders in a random order. A list of translated headlines and source information is included in Appendix A. The coding scheme is presented in Table 1. More detailed instructions, as used by the coders, are included in Appendix B.

For style – the analysis of clickbait elements in headlines – a list of seven stylistic elements used to attract attention and encourage internet users to click on a particular headline was compiled (formatting, lists, numbers, exaggeration, extreme content, prediction, and suspense), based on Molina et al. (2021) and Biyani et al. (2016). For each headline presence or absence of these elements was coded.

For structure, the coding scheme was based on the concept of the inverted pyramid structure, mentioned above. The first paragraph was analyzed to see whether three or more wh-questions (out of six) were answered. The final paragraph was analyzed to establish the type of information that was added there (new information, evaluation of previously presented information, or unrelated information). Because the articles differed greatly in length, the focus was solely on the first and last paragraphs.

For argumentation, we focused on two strategies related to the use of quotes as arguments: (i) emphasizing the traceability of the source and (ii) emphasizing the quote’s veracity (or lack thereof). Using a specific and named source, rather than an anonymous one, makes the quote more traceable. This results in anonymous sources being perceived by readers of news stories as less credible than identified sources (Pjesivac and Rui 2014). For that reason coders coded whether or not the source was mentioned in the article itself, rather than in a hyperlinked text or not at all. They also coded whether a specific person was quoted, rather than another news source or an organization. The last feature in the coding scheme was the presence or absence of an evaluation of the quote’s veracity in its presentation. In cases where the writer indicated that the quote was to be believed or disbelieved, this was marked as an evaluation of the quote.

Only the first quote in the body of each article was analyzed in this way. Later quotes in an article often cite the same source as the first quote, but do not necessarily repeat all information about that source. This complicates coding the source information, and can even add ambiguity about who the source is, if several sources are used within one news article.

For all three explorative analyses, all three authors coded all texts individually. The majority verdict was used (a characteristic is considered present when two or three coders coded this characteristic as being present; Lombard et al. 2002). For all three domains (style, structure, and argumentation), the proportions of ANM and MSM headlines in which a specific feature was present or absent were compared, using chi-square analyses. An additional analysis was done for style. Because headlines could contain more than one clickbait element, the number of clickbait elements per headline was calculated. The mean number of clickbait elements per headlines was determined and compared using a Welch independent samples t-test.^[3] For all analyses, IBM SPSS Statistics (version 27) was used.

3 Analysis

4 Conclusion and discussion

Our exploration is promising: there are specific linguistic features that are used to different extents in ANM and in MSM. Some features are more common in ANM than in MSM, in particular clickbait elements in headlines (specifically exaggeration, extreme content and suspense), evaluations (rather than new information) in the last paragraph, and quotes being presented explicitly as (un)true. Other features are more common in MSM: a first paragraph answering more than three out of six wh-questions and a last paragraph that adds relevant news information.

In addition to focusing on the presence or absence of features, the results suggest that a more qualitative focus on how linguistic features are used (e.g., mention of profession of a source) could provide relevant insights as well.

The fact that differences between ANM and MSM can be found in features that nonexpert readers could potentially recognize (style in headlines, journalistic quality in first and last paragraphs, and the use of arguments from authority in quotes) invites us to delve deeper into these and similar features. Extending knowledge of this type of feature is insightful in designing educational interventions to help less skilled readers assess the quality of news articles they are reading. It is important to note that because some ANM articles might resemble MSM articles in one respect (for instance having a traditional structure) but not in another (the use of clickbait features in the headline), a combination of features should be included when describing ANM and MSM and in the development of educational interventions.

Although these results are encouraging, some caveats should be mentioned. Most importantly: some of the features did not result in high intercoder agreement. In a headline such as “Death to all citizens” (text 26) all coders agreed on the extreme content of this headline, but one of the coders did not consider words like “death” or “all” to be exaggerations, whereas the other coders did. Disagreements on linguistic features like these are to be expected, given the interpretive nature of our data (Spooren and Degand 2010). For this exploration we used majority verdicts, but for a single reader this does not work. In future analyses, individual coder patterns will be checked, to see whether these follow the majority patterns.

Our analysis is far from exhaustive: it is very likely that other text features can be identified that could help distinguish between ANM and MSM. However, this exploration has shown that this type of analysis yields useful results and calls for more features to be analyzed in a larger corpus of ANM and MSM texts, to find out which features are best taught in a future educational intervention.