Satire, honey and tears: how The Onion and The Babylon Bee do satire

3.1 Methods

4.1 Analysis 1: Reporting from flyover country

The political landscape in the United States is both metaphor and reality – regardless of whether a state is considered red or blue, there is also a political divide between urban (liberal) and rural (conservative) communities. Because The Babylon Bee is overt in its biases, geographic divides may manifest in how The Babylon Bee does satire. One way to test this possibility is to draw from stereotypes related to flyover country. The term flyover country or flyover states refers to a specific region of the United States, the Wikipedia entry explains:

…the interior regions of the country passed over during transcontinental flights, particularly flights between the nation’s two most populous urban agglomerations: the Northeastern Megalopolis and Southern California.^[2]

The term thus indexes stereotypes about cultural and economic differences between those living in two different regions of the United States. The coastal regions such as New England, California, and the Pacific Northwest are home to large cities generally associated with liberal government policies and officials (e.g., Seattle, San Francisco). In contrast, the interior regions of the country, including large swathes of more rural states that lay between the coasts, are those states which coastal denizens “fly over” during travels. Hence, the term flyover states is generally seen as a pejorative towards those living in those regions. This regional divide acquired particular significance after the 2016 US presidential election, with flyover country being described as America’s overlooked heartland (Kendzior 2018) which may have in part fueled the success of Donald Trump and the MAGA republicans.

The flyover effect is more than a perception or a stereotype. This effect manifests in how regular news is reported, in that articles and news stories are written primarily by journalists residing in the coastal regions of the US (Kendzior 2018). The flyover effect is also reflected in online news consumption patterns, with more frequent exposure to online news found among people who live along the coasts (Althaus et al. 2009).

Given the political significance of the flyover distinction, we questioned whether this effect would manifest in a specific bit of information peculiar to satirical news: the reported locations of the satirical news articles. Using locations in this way is a novel approach that, while simple in method, may nonetheless reveal underlying assumptions and biases of the satirical outlets. Most articles from The Onion and The Babylon Bee come with a fictitious dateline, which lists the location in which the story supposedly took place. We therefore set out to determine whether the fictional news sources reflected the flyover states stereotype for articles purported to be published in the United States. If yes, this would locate a point of similarity between the two outlets in relation to the way they mimic the nature of regular news. We were further interested in whether The Onion and The Babylon Bee differ in their coverage of flyover states. Because of The Babylon Bee’s self-described conservative stance, we expected some variation when compared to The Onion, either through a hyper focus on coastal regions (assuming liberal political targets), or more reporting from within flyover states to demonstrate affiliation with readers from those areas of the country.

4.1.1 Results and discussion

To answer these questions, we extracted the location from the dateline of each article in our corpus and tallied them to create a frequency distribution of locations. Only locations which took place in US states were retained. A higher proportion of articles in The Onion included state locations (86.7 %, n = 4,107) when compared to The Babylon Bee (71.3 %, n = 6,409). In both cases, non-US locations included other real locations in the world, as well as fictional or impossible locations such as heaven, hell, outer space, and others. We plotted the percentage of articles published from each state as a heat map of the United States (Figure 1).^[3] The complete data are available in the Supplemental material.

Figure 1:

Heatmap of proportion of US states located in datelines of satirical news articles.

Figure 1 highlights an imbalanced publication frequency with very few articles published from interior states (i.e., flyover country). A logistic regression predicting a binary outcome of whether an article was published from a state in flyover country found that the odds of an article being published from a flyover state were only 44 % of the odds being published from a coastal state, a significant effect (flyover state log odds = −0.819, SE = 0.021, 95 % CI [−0.861, −0.778], odds ratio = 0.440, p < 0.001).^[4] This demonstrates a preference for publishing articles outside of flyover country, with no significant differences in this preference between the two outlets (differences in log odds between outlets: estimate = 0.002, SE = 0.043, 95 % CI [−0.083, 0.087], p = 0.965).

Instead, a few states and areas, such as Washington DC, California, and New York, take up the lion’s share of the articles. However, logistic regressions indicated significant differences between the outlets for these states. For instance, the focus on DC is higher in The Babylon Bee (24.9 %) when compared to The Onion (19.6 %), which translates into a 37 % greater odds for The Babylon Bee to publish articles from DC when compared to The Onion (log odds = 0.313, SE = 0.049, 95 % CI [0.218, 0.409], odds ratio = 1.37, p < 0.001). Articles emerging from DC can focus on many different aspects of the American federal government, such as Presidents Trump and Biden as well as members of their governments. This increased focus suggests a greater proclivity for The Babylon Bee to focus on the federal government when compared to The Onion. There was also a greater focus on the state of California for The Babylon Bee (17.6 %) when compared to The Onion (13.2 %), a difference of 40 % greater odds (log odds = 0.338, SE = 0.057, 95 % CI [0.227, 0.449], odds ratio = 1.40, p < 0.001). Among American conservatives, California is stereotyped not just as a blue state, but indeed an example of extreme liberalism, responsible for destroying economies and cities which sends residents fleeing to other states. The greater focus on California suggests The Babylon Bee taps into these stereotypes, creating more articles with targets located within or associated with California.

In contrast, articles from New York were higher for The Onion (12.5 %) when compared to The Babylon Bee (6.7 %), a statistical result of 99 % greater odds (log odds = 0.688, SE = 0.069, 95 % CI [0.553, 0.823], odds ratio = 1.99, p < 0.001). Most of these articles take place in New York City. As a major metropolitan area and headquarters of major US newspapers (e.g., The New York Times), the choice of New York City as a location may be a safe one, associated more with American news in general (rather than political news from DC). A final noteworthy difference is a 3.8 % higher publication frequency from Illinois for The Onion (5.7 %) when compared to The Babylon Bee (1.9 %). Statistically, this is the strongest contrast among these comparisons, with a 220 % greater odds of an Illinois article being published by The Onion (log odds = 1.160, SE = 0.114, 95 % CI [0.938, 1.380], odds ratio = 3.20, p < 0.001). This difference can be explained knowing The Onion’s headquarters are located in Chicago, Illinois.

These differences do suggest some variation in the targets and topics of the two publications, but on balance both The Onion and The Babylon Bee appear to follow the flyover country trend. Such a finding makes sense, because satirical news, whether conservative or liberal, nonetheless needs to follow the conventions of its parent genre (i.e., non-satirical news) to be effective.

4.2 Analysis 2: Conservative anger and liberal irony?

Another point of potential (dis)similarity between the two satirical outlets may be found in the emotional and affective language used by The Onion and The Babylon Bee. Such differences have been shown to exist in comparisons of non-satirical texts written from conservative and liberal viewpoints. For instance, text analysis on a set of online news articles covering the 2014 police killing of Michael Brown in Ferguson, Missouri, reported that conservative news used more negative language than liberal news (Turetsky and Riddle 2018). Computational analysis of satirical texts also show that satire contains more negative language than non-satire (Mihalcea and Pulman 2007; Skalicky and Crossley 2015), as well as a greater incidence of both negative and positive framing language (Brugman et al. 2023).

Because the presence of negative affect in text distinguishes satire from non-satire, it may also distinguish between conservative and liberal non-satirical news. One method to measure emotion in text is to use dictionary-based sentiment analysis techniques. We followed such an approach and used the Linguistic Inquiry and Word Count software (LIWC; Boyd et al. 2022). LIWC is a dictionary-based text analysis software created for the purpose of understanding how semantic properties are linked to social and psychological states (Tausczik and Pennebaker 2010). Programs like LIWC count the different words in a text, and then compare this tally to predefined dictionaries which sort words into various different linguistic, psychological, and emotional categories. The Brugman et al. (2023) study reviewed earlier compared emotional frames between satirical and non-satirical news using LIWC. While their results suggested no differences between liberal and conservative news satire, this was only in relation to their deviation from non-satire. We take a different approach here and directly compare emotions between liberal and conservative satire. Differences in emotional tone between political satire sources may reflect Young’s claims that the political right is fueled more by outrage, whereas the political left is more prone to irony (Young 2019). As such, we predicted that right-wing satire may project a more negative emotional tone and sentiment compared to left-wing satire.

We focused on words categories related to sentiment and tone, specifically the categories of positive emotion, negative emotion, positive tone, and negative tone. LIWC distinguishes between emotion and tone in that the emotion dictionaries are more narrow, being made up of emotion words (angry) or words that strongly imply emotions (laughter). The positive emotion dictionary consists of words like happy, great, and good, whereas the negative emotion dictionary consists of words like hate, guilt, and worst. In contrast, the positive and negative tone categories are more inclusive, containing the same words in the positive emotion and negative emotion categories and also words and phrases related to emotions, such as birthday (positive tone), terrorist, and too much (negative tone). We generated a single index of emotion and tone by calculating the overall frequency differences between negative and positive words for both emotion and tone categories. Higher resulting scores indicate a more positive emotion or tone in the text.

4.3 Analysis 3: Topics and targets

Having compared the satirical texts along two surface-level features, we now turn towards more abstract measures of latent topics in the text. Prior research using human annotation has found that topics of satire differ between liberal and conservative outlets (Brugman et al. 2022a; Skalicky and Chen 2023). Here we explore the potential to use computational methods to accomplish a similar task, an approach that has not yet been leveraged to understand differences between satirical outlets. In addition to collecting fine-grained linguistic information from texts using lexicon-based approaches such as LIWC, other natural language processing (NLP) algorithms can be leveraged to model or extract the topics of a text. Such topic modeling is achieved by examining the way keywords group to certain topics, and then using unsupervised machine learning approaches to cluster documents or texts into similar categories (i.e., topics). Typically, the nature of the topics or categories is implicit to the groupings of the documents, meaning that researchers need to infer why certain documents were grouped with another. However, the rapid development of large, pre-trained transformer models which make possible generative artificial intelligence technology have also been leveraged to improve existing NLP methods for querying information from text data, including topic modeling.

One such approach is called BERTopic (Grootendorst 2022). This algorithm first converts documents into a set of multi-dimensional sentence embeddings using pre-trained large language transformer models. The high dimensionality of these embeddings is then reduced, and documents are clustered based on similarities of these embeddings. The clustering of these similarities creates clusters of topics, and each document is assigned to a single topic. The resulting topics are then analyzed for specific words and phrases in the clusters using a class-based term frequency inverse document frequency algorithm (tf-idf), which identifies keywords and phrases specific to those topics. Crucially, the resulting keyword representation is performed after the topic clustering, meaning that the word representations are not the basis for determining the topic clusters (the clusters were formed during the sentence embedding stage). Instead, the word representations are a post-hoc analysis of the clusters. The benefit of this approach is that clusters formed by high dimensional yet opaque sentence embeddings are represented as words and phrases which can be interpreted by the human eye. Moreover, this approach requires no traditional preprocessing of the texts (e.g., stemming, stopword removal) before they are entered into the model.

The BERTopic algorithm is highly customizable, with freedom to choose different pre-trained embedding models, vectorizing algorithms, clustering methods, and pre- and post-processing of the topics. When combined with the stochastic approach inherent to most of the BERTopic algorithm, this means slightly different results will be obtained each time a new model is trained. Such an approach can lead to variation in analysis and threaten replicability. Therefore, we followed the best-practices approach suggested by the author of BERTopic (Grootendorst 2022),^[5] which includes saving the initial state of our algorithm so that the same results can be achieved each time the model is run. One of the most important decisions is the number of topics to create – BERTopic allows for a greater number of specific topics, a smaller number of broad topics, or something in between. We opted for a middle ground, forcing topics to include at least twenty texts so that our topics had some degree of specificity. Texts which do not load strongly into the clusters are initially assigned no topics. However, BERTopic will allow those texts to be assigned to existing topics using different strategies. We opted for the embedding strategy, which assigns outlier texts to existing topics based on semantic similarity. Our topic models used a lightweight sentence embedding model with 384 dimensions (all-MiniLM-L6-v2), which is suitable for the relatively small size of our individual documents and corpus when compared to other machine learning tasks. Clearly, other choices could be made during this process. Therefore, we make out complete code and data available on the Open Science Framework.^[6] Interested researchers can view our results, but also tune the representation parameters to obtain different topic sizes, as well as different vocabulary representations of the topics, or try different embedding models should they like.

Using these parameters, our models yielded 69 topics in The Babylon Bee and 43 topics in The Onion data. We provide spreadsheets listing the complete topic information, including number of documents and word representations in the Supplemental materials. Here, we focus on interpreting some of the most frequent topics obtained from each satirical newspaper. The resulting figures show the top twelve topics based on the number of articles assigned to each topic.^[7] For each topic, the top 10 words most representative of that topic are plotted in descending order of importance.

Topics for The Babylon Bee are shown in Figure 2. The largest topic includes 1,840 documents (∼20 %) and is represented by words related to Christianity and religious culture in the United States. Such a topic makes sense in light of The Babylon Bee’s self-labeled identity as a satirical newspaper from a Christian perspective. The second most frequent topic contains words related to family – suggesting an adjacent core value of family tied to religion. The remaining topics are all more overtly political in nature, but also represent political topics relevant to American conservative talking points. Topics are formed around Joe Biden, protests and ANTIFA, abortions, the COVID-19 vaccine, climate change, and government spending. All of these topics are issues the American political right is critical of, suggesting that The Babylon Bee articles are echoing such talking points in its satire.

Figure 2:

Word representations with importance scores for the top twelve most frequent topics in the Babylon Bee articles. Total number of articles per topic shown above each topic.

Now consider the topics for The Onion, displayed in Figure 3. The most frequent topic contains slightly over 8 % of the documents in the corpus, and appears to be related to food. The second most frequent topic, similar in percentage, is political in nature, with a mixture of both Biden and Trump, the two US presidents during the time of these articles. Like The Babylon Bee, some of the remaining clusters index topics of political significance in the United States, such as the COVID-19 pandemic, policing in the USA, and ultra-rich technology CEOs. Yet other topics depict more mundane matters, such as romance, clothing, driving, and the reports of scientific studies. The latter topic makes sense in light when considering The Onion’s proclivity to publish the results of fictional scientific studies (Skalicky 2019).

Figure 3:

Word representations with importance scores for the top twelve most frequent topics in The Onion articles. Total number of articles per topic shown above each topic.

There is thus a clear difference between the nature of the topics when comparing the two outlets. While The Babylon Bee is focused on topics related closely to Christianity and political topics relevant to the American political right, The Onion data include a wider range of less overtly political topics. Politics are still clearly a frequent target for The Onion, but there is a notable contrast in how the second most frequent topic in The Onion contains both Trump and Biden, whereas a similar topic for The Babylon Bee only names Biden. While this may be evidence that The Babylon Bee is more overtly conservative, it does not rule out left-leaning biases for The Onion. Topics related to police and the pandemic in general (rather than a specific focus on the COVID-19 vaccine) reflect perspectives that are more at home on the left versus the right, but nonetheless does not reflect such a clear satirical echoing of strongly identifiable left-wing narratives.

4.4 Analysis 4: Scripts and publishing habits of The Babylon Bee

Our final analysis deviates from the comparison approach to focus solely on The Babylon Bee. We opted for this singular analysis because The Babylon Bee represents a conservative alternative to the assumed liberal bias of most other satirical outlets (Sienkiewicz and Marx 2022). In light of censorship from social media platforms^[8] and long standing assumptions that conservatives mostly do not do irony and satire (Young et al. 2019), it may be the case that The Babylon Bee must retract inwards and focus on retaining a core audience of readership. Such an assumption is supported by The Babylon Bee’s monetization structure, which caters to a smaller audience with a higher volume of output. For one, The Babylon Bee offers a subscription service with several perks to loyal patrons: on top of the ubiquitous premium ad-free experience offered by many subscription services, patrons of The Babylon Bee enjoy such benefits as accessing premium stories, podcasts, limited-run publications, and a subscriber-only forum to contribute future article headlines. Benefit structures such as these resemble the support plans utilized by streamers on platforms like YouTube and Twitch, where authors often seek to deliver a greater quantity of content within specific niches, rather than quality-driven and intermittent output that can be buried beneath algorithmic trends. In contrast, The Onion has no such subscriber model. It instead relies on traditional advertising revenue, as well as income from t-shirts and other items sold on its online store. In 2019, The Onion was combined with the Gizmodo suite of online blogs (formerly known as Gawker) into a single entity. As such, The Onion is now part of a larger suite of websites focused not just on satirical news, but also video games, technology, and more.

A timeline comparison of publication frequency (Figure 4) indicates The Babylon Bee favors a pattern of high frequency publications. When comparing total publications per month from January 2016 to December 2022, The Onion was quickly outpaced by its conservative counterpart. On average, The Babylon Bee produced some 117 articles per month within this time frame versus The Onion’s 58. We argue this prolific publication record is supported through the recycling of highly context-specific scripts, buzzwords, and punchlines in articles, relying on a deeply conservative frame of reference to successfully be understood by the reader. For instance, when riffing on President Biden’s 2022 State of the Union address, The Babylon Bee published eight articles within a single day, all running with comparisons of Biden’s red-on-black aesthetics seen in the speech to Adolf Hitler’s rise to power in the 1930s and subsequent ramp-up to the Second World War.

Figure 4:

Number of articles published per month by The Babylon Bee and The Onion, starting in 2016 when The Babylon Bee first began publishing.

We explored whether there were other examples of these repeated scripts. To do so, we employed a qualitative, emic exploration into the narrative scripts contained within The Babylon Bee. After obtaining the computational topics from Analysis 3, a shortlist of keywords (names, or recurring buzzwords like identity) were used as seeds to locate prevailing topics about specific people such as Representative Alexandra Ocasio-Cortez (AOC) or topics such as gender identity (the 21st most frequent topic in The Babylon Bee corpus was represented by words related to AOC, see the Supplemental material). Articles containing these keywords were identified, and subtopics within the articles were then identified.

Our analysis revealed several repeating scripts echoing American conservative talking points. These basic scripts allow The Babylon Bee to create articles ad hoc through piecing together various references off a key topic. For example, of the ninety articles AOC is mentioned in, twenty categorized her as a cartoonishly inept socialist, framing her as infantile and performative. Articles also portrayed AOC as simply dumb (choking herself on her own shoelaces being a recurring theme in these forty-two articles), the victim to some kind of imagined violence (twenty-one articles), sexualized her (eleven articles), or portrayed her in a childlike way (fifteen articles) through inserts of fairy tale-esque props such as unicorns, using arbitrarily absurd numbers (e.g., an “eleventy-quadrillion dollar” relief package proposal), or with youth adjacent activities (shopping at Forever 21 or performing TikTok dances). In total, only twenty-four articles mention AOC without one of these scripts, and of those the majority characterize her as some kind of cartoonishly performative socialist or social justice warrior. Similar scripts were found within forty-three articles referencing Ihlan Omar, another member of the U.S. House of Representatives notable for support of climate change initiatives, outspoken defense of abortion rights, and a vocal supporter of Palestine. Within these articles, two scripts emerged: twenty-five accused Omar of antisemitism (many of these being fictional quotes attributed to Omar asking to “kill the Jews”), while six follow a trope of Omar being married to her brother.

The repeating script approach persists in ideological topics as well. For example, 282 articles included the word ident, of which 117 were primarily in reference to some form of gender identity (the rest including race politics and articles referring to a president). Of these 117 articles, specific scripts were repeated: eighty-five contained a reductive or obfuscating script (identity jokes such as “I identify as an Apache Attack Helicopter/M1 Abrams/owlbear”), sixty-five conflated social gender identity with biological sex (often focusing on physical traits such as facial hair, genitalia, or athletic performance), thirty-nine included a victimization script for individuals who exhibited some form of bigotry (including references to authoritarian style arrest, re-education, or the like), and thirty-seven included scripts referring to child predation and grooming (of these in particular, fourteen were aimed at educators perpetrating such behavior).

This echoing of conservative ideals could represent salient scripts for readers of The Babylon Bee. Such scripts are more than just talking points, and instead represent fundamental assumptions about the way the world is (and how it should be). For all satire, including The Babylon Bee, scripts are necessary components of the humor process (Attardo 1997; Attardo and Raskin 1991). Members of different humor communities rely on such scripts as an a priori guide to what is acceptable to joke about, as well as satirize, which partially explains why satire has such varied outcomes among different audiences (Pfaff and Gibbs 1997; Simpson 2003). Although we do not analyze them here, The Onion certainly relies on scripts in the same manner as The Babylon Bee, and we thus do not mean to imply that The Babylon Bee is doing something different than other satirical outlets in its use of scripts. Instead, we argue the increased publication frequency from The Babylon Bee amplifies these scripts through repetition – making salient a general tendency of all satire to rely on shared scripts between the satirist and the audience.