GERMA: a comprehensive corpus of untrustworthy German news

1 Introduction

The latest World Economic Forum has declared that misinformation stands out as the most significant short-term global risk, capable of disrupting elections and exacerbating societal polarization (https://www.weforum.org/publications/global-risks-report-2024/). The reasons are multiple: on one side, the structure of social media rewards users for sharing information that attracts others’ attention, which often translates to misinformation sharing (Ceylan et al. 2023); on the other hand, generative AI minimizes the efforts needed to craft fake information, allowing almost anyone to create an array of falsified content, ranging from articles to deepfakes (Ferrara 2024). The risk is that citizens will be exposed to a variety of conflicting information, causing societal paralysis amid serious issues – COVID-related misinformation cost Canada at least 2,800 lives and 300 million dollars (Himelfarb et al. 2023) – and the erosion of democratic processes based on deliberation (Farrell and Schneier 2018; Lasser et al. 2023).

There are two main ways to detect misinformation online. The first consists of deploying experts to read and judge news articles as trustworthy or fake news. This is often the ideal process but requires time for thorough fact-checking (Hassan et al. 2015). The second way involves using large language models (LLMs) that are specifically trained to scan an article and classify it as fake or genuine based mostly on textual characteristics. These models are faster and cheaper than human classification but usually require a vast amount of data on which they can be trained for precise classification. The issue is that the majority of datasets of fake and genuine news available online are in English (D’Ulizia et al. 2021), despite misinformation being a problem everywhere.

In Germany, for example, fake news has been connected with derogatory content towards immigrants (Humprecht 2019), and a propensity to believe misinformation has been correlated with a tendency to vote for far right parties (Leuker et al. 2022). The availability of an open-access dataset of fake news content in German would substantially encourage research into the role of misinformation in the German context.

This is why we created GERMA (the GErman coRpus of MisinformAtion), a comprehensive corpus of news articles generated by sources classified as untrustworthy by professional fact-checkers. Despite not being the first corpus of its kind, as other scholars have published German fake news corpora (Mattern et al. 2021; Rieger et al. 2023; Vogel and Jiang 2019), there are two aspects that differentiate GERMA from previous efforts. First, we made the choice to consider news reliability based on the source rather than articles per se. While this means that we cannot specifically treat the material in GERMA as fake news but rather as news from unreliable sources, it allows for a greater variety of content and a generous sample size. This is crucial because fact-checking every single article would be rather expensive and time-consuming. GERMA includes over 230,000 news articles, making it the largest dataset currently available for German-language misinformation research. Second, GERMA comes not only with texts but also with a variety of corollary measures that are useful for scholars. For example, it includes keywords, topics, dates, and lexical features, as well as incoming traffic metadata of the source domains. By focusing on source reliability and providing a rich dataset with associated measures, GERMA serves as a valuable resource for researchers aiming to enhance their comprehension of the complex dynamics surrounding misinformation.

2 Methods

In the following, we describe the methodology we employed to build GERMA and to extract the rich set of (meta)data from it. Like other similarly built corpora of misinformation (e.g., Carrella et al. 2023; Miani et al. 2021), GERMA was compiled from sources, instead of articles. This means that we first identified sources that deliver misinformation and then extracted individual articles from these sources. Such a method has been shown to be reliable in extracting conspiracy versus mainstream documents (Hemm et al. 2024; Mompelat et al. 2022). Once the list of sources was created, we downloaded all articles allowed for scraping from each domain, and then cleaned the corpus. Next we then extracted a set of complementary measures capable of defining the semantic content for each document: keywords, topics, and lexical features. We also provide, for each website, a set of metadata about the quality of the information shared, and the behavior of the users reaching such websites.

3 GERMA’s features

GERMA is a large corpus (132,330,548 tokens) containing 237,848 documents, spanning more than 30 years (see Figure 1), with an average word count of 556.37 (SD = 649.94; range: 101–9,953). It is built from 30 domains classified as delivering misinformation by Lin et al. (2023) and NewsGuard (2020). In Table 2, we list the domains used to build GERMA, including their document counts and domain quality as outlined by Lin et al. (2023). Besides the texts themselves, GERMA comes with a rich set of data and metadata. GERMA is freely available at the OSF repository at https://osf.io/3bthj. In the following, we describe the datasets stored in the repository to illustrate the richness of GERMA.

Figure 1:

Distribution of documents by date (for those with dates; N = 130,381) aggregated by year. From 2007, each year contains at least 1,000 documents with dates.

4 How to use GERMA

GERMA represents a rich source of data for better understanding the content of misinformation. Being for the most part composed of texts with additional metadata and lexical features, GERMA is conceptualized as a turnkey resource. Researchers can test hypotheses, replicate results, further extract features, or build classification and predictive models from GERMA.

By using texts, researchers can replicate previous analyses by extracting from GERMA co-occurrence of terms, measures of documents similarity and textual cohesion, and relationships between nominal compound constituents and syntactical elements (Fleckenstein 2024; Miani et al. 2022, 2024]; Samory and Mitra 2018). By leveraging the lexical features provided through LIWC, researchers can explore the psychological dimensions of German misinformation. This allows them to replicate findings from English-language studies, which show that the language of misinformation – especially conspiracy theories – is often rooted in social identification and exclusion, filled with negative emotions, and focused on themes of power, dominance, and aggression, frequently challenging the official narrative (Fong et al. 2021; Klein et al. 2019; Miani et al. 2021).

As illustrated in Figure 2, low-quality documents tend to use, on average, more words (indicated by the LIWC feature WC) and longer sentences (as reflected by WPS). These documents also frequently focus on themes of tribalism and social inclusion or exclusion (as indicated by features such as they, we, pronoun, and social), employ a rhetoric of questioning (evidenced by interrog and QMark), and use causal language (reflected by cause). Different sets of psycholinguistic measures can be further extracted from the texts such as word norms for concreteness or age of acquisition, with their German translations (Birchenough et al. 2016; Brysbaert et al. 2013; Charbonnier and Wartena 2020; Kuperman et al. 2012).

Figure 2:

The 20 highest and 20 lowest β coefficients (on the y-axis; all p < 0.001, Bonferroni corrected) from regressions predicting LIWC lexical features (on the x-axis) by websites credibility scores. The credibility scores have been inverted to improve interpretability. This means that positive values indicate the feature being positively correlated with less credibility. Note that, due to the large sample, the error bars are smaller than the dots representing the coefficient, hence not visible.

Keywords, topics, and lexical features can serve as variables of interest in diachronic analyses. For instance, researchers can examine fluctuations of a specific topic of interest over time. As shown in Figure 3, we averaged the gamma values, which indicate the “presence” of a topic in the articles, for the topic “us_usa_trump_prasident_china” across each month from 2007 to 2024. The figure reveals peaks corresponding to the two most recent presidential elections at time of writing, specifically in late 2016 to early 2017, and late 2020 to early 2021.

Figure 3:

LDA topic gamma values (y-axis) over time (x-axis). Topic k20_004 relates to the United States. The 10 most relevant words for the topic are displayed in decreasing order above the plot (English translation: ‘US, USA, Trump, President, China, Govern, Biden, Chinese, Iran’). The plot reveals peaks corresponding to the US presidential elections in late 2016 and late 2020, as well as a peak in June 2009, which may be associated with former US president Barack Obama’s visit to Germany earlier that month.

Moreover, these same elements – keywords, topics, and lexical features – can be utilized not only as dependent variables but also to further subset GERMA prior to analyses. This would be parallel, for example, to a study focused on specific subsets of LOCO (the Language of Conspiracy corpus; Miani et al. 2021), which revealed distinct conceptualizations of health-related content between conspiracy documents, framed in terms of beliefs, and mainstream documents, focused on scientific evidence (Reiter-Haas 2023), paralleling analyses of political communication on Twitter (Lasser et al. 2023). Another such investigation showed that the group of documents mentioning conspiracy (searching for the pattern “conspir*” in the text) show an increase in conspiratorial language (marked by lexical features associated with crime, terrorism, deception, and stealing) in non-conspiratorial documents as well, and that documents relying on prototypical conspiratorial language are shared more on Facebook (Miani et al. 2021). Using keywords and topics, other works have shown that documents delivering misinformation exhibit greater interconnectedness (Carrella et al. 2023; Miani et al. 2022), providing empirical support for an overarching worldview (Lewandowsky et al. 2018).

Using the URLs associated with each document, researchers can extract HTML data in order to analyze web markup features as previous work has done on fake news (Castelo et al. 2019). Websites’ information could be used to analyze – or compare with other datasets – the behavior of users reaching misinformation websites. Previous work analyzing those patterns suggests that such behavior has elements of confirmation bias and in particular selective exposure to misinformation (Cardenal et al. 2019; Carrella et al. 2023; Miani et al. 2021). Lastly, efforts could also be made to develop annotation schemes and tools capable of automatically identifying misinformation language (Diab et al. 2024; Fort et al. 2023; Mompelat et al. 2022).

5 Conclusions

In this paper, we introduce GERMA, a freely accessible corpus of over 230,000 articles collected from German news websites classified as unreliable by fact-checkers. The corpus joins a group of similar existing corpora (Mattern et al. 2021; Rieger et al. 2023; Vogel and Jiang 2019), but it stands out both for its larger number of articles and for the additional features available, which allow GERMA to be used for linguistic, sociological, and psychological studies.

Features such as topics and keywords associated with individual articles allow researchers to extract specific linguistic characteristics of each topic and make comparisons. Most of the articles also have a publication date, which allows for diachronic studies to examine the linguistic evolution of each topic. Lexical features (specifically, LIWC scores) allow for a variety of sociological and psychological studies. Finally, GERMA also contains domain-specific features such as the type(s) of news typically shared by a specific source, as well as data on the incoming traffic for a domain, which can be used to study digital community behavior.

One limitation of our dataset is that the news it contains is classified as “untrustworthy” based on the reliability of the sources that produced them. This means that the news articles in our dataset cannot be automatically labeled as “fake news” without further verification. Researchers need to conduct a direct check when working with these sources. While an article-based approach would provide more precision, we believe that a domain-based approach – where domains are classified with an aggregate score from various misinformation datasets and fact-checkers – strikes a suitable balance between data quality and quantity.

Another limitation concerns the aggregate score itself. Although this score is based on multiple expert ratings, each expert uses their own criteria for judgment. Additionally, raters like NewsGuard often derive reliability scores from a combination of factors, including credibility, transparency, and error correction. Therefore, we advise users not to compare the reliability of domains in our dataset directly, as a lower or higher score does not necessarily correlate with the likelihood of sharing misinformation.

Despite these limitations, we consider GERMA to be an important resource in misinformation studies, not only for the quantity of the data it offers, but also for the additional measures it includes, allowing for different types of studies. GERMA also represents an additional resource in a language other than English, which is often over-represented in these contexts where there is a need for datasets to train classifiers. We believe GERMA’s unique features make it a valuable tool for researchers working to combat the growing problem that misinformation represents.