Understanding content dissemination on sensemaking: WHO’s social listening strategy on X during the initial phase of COVID-19

2.1 CERC framework, sensemaking, and self-efficacy

Health crises occur in myriad formats, places, etc., and so institutions need effective methods for communicating key information to the public (e.g., Seeger et al. 2020). One such communication model is Crisis and Emergency Risk Communication model, developed by the U.S. Centers for Disease Control and Prevention to “educate and equip public health professionals for expanding public health communication responsibilities in emergency situations” (Veil et al. 2008: 26). The CERC model has been used in prior public health crisis research such as influenza (Vos and Buckner 2016), Ebola (Kieh et al. 2017), Zika outbreak (Lwin et al. 2018), etc. to analyze communication strategies and their impact on public perception and behavior. The findings demonstrated the adaptability of the CERC model for effective communication across various stages of disease outbreak (pre-outbreak, during and post outbreak). However, the researchers (Lwin et al. 2018) assert the need for further testing and refinement of the CERC model to understand health outbreak communication on other social media platforms such as X (formerly Twitter), as well as for other infectious diseases.

CERC views any crisis as an ongoing event that progresses through stages with specific and unique communication needs and strategies (Miller et al. 2021; Reynolds and Seeger 2005; Veil et al. 2008). Pre-crisis, initial event, maintenance, resolution, and evaluation are the CERC’s five stages of crisis progression. This study focuses on the first two stages: pre-crisis and initial event. We focused on the first phase of the pandemic, from December 31, 2019 (when a Chinese doctor alerted WHO to the virus) to April 1, 2020 (a few weeks into the pandemic when the majority of the world was experiencing quarantine).

Pre-crisis is an incubation stage where a threat has emerged but the crisis has not yet unfolded (Seeger et al. 2020). Because crises are unpredictable, messages distributed at this stage include warnings to educate the public about the risks and information to help them monitor and prepare in case the threat becomes a crisis (Seeger et al. 2020).

The initial event stage of the crisis is the triggering event that indicates the crisis is underway. At this stage, the crisis is fast developing, often eliciting a strong sense of threat, panic, and ambiguity among publics because information about the crisis remains scattered, incomplete, or even contradictory (Reynolds and Seeger 2014). Aiming to reduce uncertainty and reassure the public about the various measures taken to improve the situation, CERC advises communicators to provide timely updates. Messages should include statements that are sympathetic and reassuring to the affected groups. The messages should also be simple and provide general information about immediate risks, the crisis, expected outcomes, and appropriate actions including self-efficacy promotion (Reynolds and Seeger 2014; Seeger, Reynolds & Day 2020). Sensemaking and self-efficacy are key processes at this stage of CERC.

2.2 Social media, infodemic, and social listening

During the COVID-19 crisis, social media became a source of infodemic, an epidemic of information that may or may not be accurate (Lu et al. 2021; Tangcharoensathien et al. 2020; Zarocostas 2020). Eysenbach (2020) described the COVID-19 infodemic using the “informational cake model,” according to which the volume of information has four levels: (a) science, (b) policy and practice, (c) news media, and (d) social media. The social media platforms constitute the largest portion of the cake, containing the largest amount of unfiltered information.

An infodemic can hinder sensemaking. In the context of COVID-19, a flood of inconsistent, inconclusive, and contradictory information on social media can confuse the public. Instead of helping people comprehend what is happening, an infodemic can “pose concerns for the public to distinguish fact from fiction and, for government agencies to conduct evidence-based policy making” (Erku et al. 2021: 1955). Previous research has connected various infodemics to misinformation (e.g., Chowdhury et al. 2023; Do Nascimento et al. 2022), and misinformation impacted COVID-19 (Chowdhury et al. 2023). Misinformation in health communication could be fatal to people (Cheng and Nishikawa 2022), especially considering that COVID-19 contributed to the deaths of 6.7 million people as of January 2023 (World Health Organization n.d.). Social media users lacking medical expertise engaged in messaging to delegitimize public health officials, too (Chen et al. 2023). Eysenbach (2020) proposed several infodemic management strategies. One was constant social media monitoring or listening (i.e., “infovigilance”).

Social listening is the practice of monitoring social media data and discussions (Stewart and Young 2018). At one point during the COVID-19 pandemic, tweets about COVID-19 were circulated every 45 ms (Josephson and Lambe 2020). Since social media permits rapid diffusion of pandemic-related health information, the role of social listening warrants further attention. Real-time monitoring of COVID-19 conversations on social media can help cut through the noise and help affected communities respond effectively to threats. Public often uses social media platforms, including X, for sensemaking during crisis, such as COVID-19 (Thygesen et al. 2021; Valiavska and Smith-Frigerio 2023). Incorporating social listening into public health messaging helps people understand the disease and its scope, potentially reducing the effects of an infodemic and providing direct access to accurate information. However, organizations are limited in their effectiveness in combating misinformation (e.g., Bode and Vraga 2018; Liu et al. 2022; Vraga and Bode 2018). Liu et al. (2022) found that people with close connections can help stem the flow of misinformation, but sometimes efforts to fact check can actually have the reverse effect and strengthen some people’s belief in inaccurate information. Thus, experts do have limitations in their ability to correct already formed narratives.

Because of the volume and complexity of data collected from social media and UGC platforms, machine learning is an appropriate tool for social listening. Machine learning encompasses a wide range of computer-based data mining techniques designed to uncover complex patterns in large datasets and potentially inform decision-making. Previous research suggests that machine learning can highlight conversations during epidemics like the Zika virus (e.g., Miller et al. 2017). WHO emphasized/described the usage of AI and machine learning tools for social listening and infodemic management during the COVID-19 pandemic (World Health Organization 2021). Understanding social media conversations is important, but so is understanding how WHO used that vast amount of data to identify emerging trends, and address those trends swiftly. Therefore, the second goal of the current study was to address the following research question:

3.1 Study design

To analyze the large dataset and answer the research questions, we used three methods. First, we used unsupervised machine learning to analyze content called Latent Dirichlet Allocations (LDA). LDA is a well-known unsupervised machine learning technique that uses Bayesian statistics to identify themes or patterns in unstructured text (Blei 2012; Kabir 2022). Scholars previously used this method to analyze COVID-19-related X data (Abd-Alrazaq et al. 2020; Xue et al. 2020). Emergent topics are not labeled by the LDA and need human interpretation. So, once the topics emerged, we used thematic analysis (Braun and Clarke 2006; Kumble et al. 2022) to interpret the tweets. After identifying themes, we used content analysis to see if WHO used X to address dominant conversations.

3.2 Data collection

We scraped relevant data using “Tweepy” and “GetOldTweets” (Henrique, n.d) libraries on Python for dates ranging (December 31, 2019–March 31, 2020) while the situation was unfolding and on-going. Some of the notable events (Centers for Disease Control and Prevention 2023a) during this time period were as follows:

1. China’s WHO office was “informed of several cases of a pneumonia of unknown etiology” on December 31, 2019.

2. WHO declared the virus a “Public Health Emergency of International Concern” on January 31, 2020.

3. WHO named the virus COVID-19 on February 11, 2020.

4. WHO labeled “COVID-19 a pandemic” on March 11, 2020.

Additionally, this data was collected before the academic license for X existed. Some of the previous studies (e.g., Bacsu et al. 2021) have also used the same library to get X data. Search queries included “#2019nCov,” “#coronavirus,” “#Wuhan,” (for the first month in the date range), and “#COVID-19.” After cleaning the data for duplicates and removing WHO tweets from the main dataset, we analyzed 6,168,274 user-generated tweets in English. We conducted a second search using “Tweepy” to collect tweets sent by WHO (n = 2086).

3.3 Latent Dirichlet allocation

LDA uses probabilistic calculations to infer topics from a document (e.g., tweets) using a bag of words approach (Blei 2012; Kabir 2022; Wheeler et al. 2024). Given that the initial stages of the pandemic were unprecedented, and information was unfolding (the study authors did a thorough read of the WHO’s tweets during that timeframe to notice the change of information), the dataset was divided into smaller time ranges based on the timeline of events that unfolded and the volume of tweets generated. January and February 2020 had relatively fewer tweets (i.e., less than one million). Therefore, the authors did bi-weekly LDA models during those months. Since the number of tweets increased to over one million per week in March 2020, the authors conducted LDA analysis weekly, running a total of eight (8) LDA analyses using the following steps.

3.4 Thematic analysis

After the machine learning algorithm identified the topics, it was time to identify the labels. Topic labels were created using thematic analysis (e.g., Braun and Clarke 2006; Kumble et al. 2022). Data-driven thematic analysis can be inductive (data-driven) or deductive (theory-driven), where the coding is done with the study’s main theory in mind (Braun and Clarke 2006; Lee and Barnett 2020). This study employed deductive coding to code the tweets for themes of sensemaking, efficacy, and misinformation. The authors looked at only the top two or three topics generated by the LDA model for each date range.

Braun and Clarke (2006) recommend six steps to generate themes from the dataset. The first step involves becoming familiar with the data. For this step, the authors became familiar with the top words in each topic. Next, based on the data, we generated initial codes. For this step, we randomly selected 1–4% of the tweets from each topic generated by the LDA model, depending on the sample size for the week and each author read the tweets to come up with the basic codes and also took down notes. The third step involves collating the codes and searching for themes within the data. For this step, the authors discussed the data and compared their notes, tweets, and co-occurring words to help determine topic labels. The fourth step involves reviewing potential themes. During this step, the authors resolved conflicts with the themes by discussing them further. For some topics, the authors went back to the previous step re-read the tweets and the collated codes to ensure both the authors agreed on the theme. The fifth step involves naming the themes from the codes (i.e., topics). In this step, once we identified the codes, we assigned them to 4 themes: sensemaking, efficacy, misinformation, and other. Sensemaking, and efficacy messages codes were adapted from previous research by Vos and Buckner’s (2016) coding scheme. Sensemaking codes involved themes of placing the emerging crisis into frameworks, enabling the public to comprehend what is going on (e.g., information about the spread of the virus and the number of cases), accommodating the unexpected, and identifying patterns in the crisis. Codes that indicated tweets containing non-verifiable messages were put under misinformation theme. The final step involves final analysis to ensure the themes are coherent and guided by the CERC model and picking out vivid and compelling tweets that best describe the codes, and the themes.

3.5 Content analysis of WHO’s tweets

In order to examine whether WHO addressed the dominant conversations that emerged on X during the specified time frame, content analysis was used. The authors manually checked the dominant topics for the computational models for the target weeks against the tweets that WHO posted during the corresponding period. Three coders independently reviewed tweets posted by WHO to see whether the dominant themes were present or absent and calculated the number of tweets that corresponded to the topic. The three coders reached a 98 % agreement.

4.1 LDA topics by period

To address RQ1 and understand how publics on X made sense of the health crisis and shared self-efficacy messages, eight LDA analyses were conducted, one for each time period. Figure 1 presents the word cloud generated for the top topics during each period. The sections below detail the analysis of the top two or three topics for each period.

1. January 1–January 24. The authors analyzed 56,760 tweets from the first three weeks of January. The highest coherence score (Cv) was 0.35, corresponding to k = 6 discrete topics on LDAvis. A thematic analysis revealed that the top two topics for the time period were “number of cases of people suffering from nCOV-19” and “origins of the virus.”

2. January 25–January 31. Next 262,818 tweets from the last week of January were analyzed. The highest Cv was 0.36, corresponding to k = 8 discrete topics on LDAvis. After completing the thematic analysis, the top two topics for the time period were “outbreak and spread of the virus including blaming China” and “cases emerging outside of China and around the world.”

3. February 1–February 14. The authors analyzed 341,312 tweets from the first two weeks of February. The highest Cv was 0.34, corresponding to k = 6 discrete topics on LDAvis. After completing the thematic analysis, it was found that during this time period, the Twitterverse continued blaming China and the Chinese government and continued discussing the cases. Another dominant topic from this week was sharing scientific evidence about the novel virus and its spread.

4. February 15–February 29. A total of 477,628 tweets from the last two weeks of February were examined. The highest Cv was 0.37, corresponding to k = 6 discrete topics on LDAvis. Based on the thematic analysis, dominant conversations revolved around certain leaked Chinese videos (mostly misinformation videos) that fueled negative sentiment towards the Chinese government (20.42 % of the tweets). According to the thematic analysis, several tweets also flagged the content and corrected the misinformation. The next dominant theme was messages about quarantining cruise ship passengers. The third most popular topic was COVID-19 protection. Wearing masks, protecting eyes, and using natural remedies like lemon, garlic, and ginger to boost immunity were mentioned.

5. March 1–March 7. The authors analyzed 473,013 tweets from the first week of March. The highest Cv was 0.37, corresponding to k = 8 discrete topics on LDAvis. After completing the thematic analysis one of the dominant topics for this week was the impact of COVID-19, including cancellation of events and travel restrictions. The next most dominant topic was questioning the severity of COVID-19 and whataboutism vis-a-vis death rates and fatalities from other diseases like the flu.

6. March 8–March 13. Next 960,566 tweets from the second week of March were analyzed. The highest coherence score was 0.37, corresponding to k = 10 discrete topics on LDAvis. During this period, WHO declared COVID-19 a global pandemic. The results from the thematic analysis indicated that one of the dominant topics for this week was this declaration and the shift to online delivery (e.g., work and education). The next most dominant topic was new cases and fatalities related to COVID-19.

7. March 14–March 21. There was a large increase in tweets as 1,860,130 tweets from the third week of March were examined. The highest coherence score was 0.37, corresponding to k = 8 discrete topics on LDAvis. With the declaration of the pandemic and stay-at-home orders issued in several countries around the world, one of the dominant topics that emerged during this week was life indoors. Another dominant topic was government response (or lack thereof), particularly in the United States and the United Kingdom. The next most dominant topic was COVID-19 mitigation behaviors.

8. March 22–March 31. A total of 1,694,047 tweets from the final 10 days of March were analyzed. The highest coherence score was 0.34, corresponding to k = 6 discrete topics on LDAvis. One of the dominant topics that emerged during this week was the politicization of the virus, primarily in the United States, and the upcoming stimulus bill. Another dominant topic was government response (or lack thereof), particularly in the United States and the United Kingdom. Another dominant topic was the lack of testing. The next most dominant topic was the call to support essential workers, including healthcare workers.

Figure 1:

Topic word clouds with dominant co-occurring words for each topic mentioned in Table 2.

4.2 WHO tweets and social listening

A content analysis was conducted to address RQ2 as to whether WHO sent tweets addressing the dominant topics on X, potentially helping make sense of the health crisis. Three coders (including the two of the three study authors) independently looked at the dominant topics and compared them to determine whether WHO had addressed the topics during the respective periods. Results indicated that WHO disseminated tweets to address most of the topics in a timely manner. Table 2 indicates examples of tweets sent by WHO that addressed dominant topics, along with the date of the first tweet sent by WHO on that topic. Some topics (e.g., lack of effective leadership by the U.S. and U.K. governments) did not fall under the purview of WHO. Figure 2 indicates the number of tweets that WHO sent during each of the periods to address the dominant topics on X (Figure 2).

Figure 2:

The number of tweets sent by WHO to address the dominant topics.

5.1 X and sensemaking

The majority of dominant topics in COVID-19 tweets from January to March 2020 related to understanding the virus, and the crisis it caused. Twitter users tried to make sense of their surroundings and re-create their familiar world by framing events. Content analysis revealed that WHO engaged in effective social listening and responded quickly to dominant X conversations to help people make sense of the situation.

During January 2020, most X conversations focused on the number of reported cases, the virus’s origins and transmission, and blaming China. Attempts to frame the virus indicate an effort to reduce uncertainty and fill knowledge gaps about the unfolding health crisis. Throughout January, WHO sent out several tweets containing information that helped people understand what was going on, including the number of cases that arose both within and outside China. WHO also helped people prepare for the unexpected by providing timely information about the virus’s origins. For example, in mid-January the organization reported that an animal may have spread pneumonia-like symptoms in Wuhan, China, helping reduce ambiguity. WHO also tweeted messages expressing confidence in China, stating that China was investigating the virus, sequencing its genome, and sharing information with WHO. This type of information helped people understand the crisis.

As more information about the novel coronavirus emerged in February 2020, X users continued with the sensemaking process. During the first two weeks of February, one of the main topics of discussion was China’s role and transparency. Despite WHO’s efforts to reduce hostility, suspicion remained. Despite WHO’s efforts to promote scientific terminology, influential people and politicians kept calling COVID-19 the “Chinese Virus.” Because social media platforms thrive on message redistribution, more influential accounts than WHO could reach a larger audience and diffuse more stigma-laden tweets within the network. Towards the end of February, another dominant conversation focused on a video with misinformation about Chinese authorities mistreating COVID-19 patients. WHO began sharing information to encourage people to reduce stigma, hate, and to support those affected. WHO also enlisted the help of social media and technology companies (e.g., Google, X, Facebook) to track down misinformation.

By March 2020, the virus had gone global. By mid-March, WHO had declared COVID-19 a pandemic, and people were grappling with the new normal and how to manage quarantine. WHO addressed the issue by providing messages to promote physical and mental health. It even planned live online concerts on its X handle. Other major themes were fatality and case count. WHO assured the public that they were tracking and disseminating data promptly. The US and UK governments’ perceived inaction and the virus’s politicization were other themes (predominantly in the United States). Such topics would be difficult for any organization, let alone WHO. However, WHO did urge governments to keep citizens informed and assist those in need.

5.2 X and self-efficacy

While the X conversations were heavy on sensemaking, the results show that self-efficacy was mentioned frequently during a few of the periods. These messages helped people respond appropriately to the crisis (e.g., protective behaviors and preventive measures). Self-efficacy emerged as a dominant theme in late February. Several mentions of eye protection and masks were also noted. However, terms like “handwashing,” “six feet distance,” and “staying homesick” were not popular until March. CERC stresses the importance of self-efficacy messages early on. When a virus is new, non-vaccine self-efficacy is critical. Lack of self-efficacy messages on X was not a good sign, as previous research suggests the general public is unaware of their role in disease spread (Vos and Buckner 2016). However, the virus did not become truly global until mid-March, which could explain the delay. Additionally, this study may have missed self-efficacy tweets in other languages from people living near China because it examined English-language tweets only.

WHO began tweeting self-efficacy messages to educate the public about preventive behaviors on January 9, 2020. But this important topic fully emerge until late February. WHO’s self-efficacy messages may not have received enough retweets because social media users were not sufficiently engaged with the topic. WHO should consider partnering with social media influencers and celebrities to help spread the message. Actors and businessmen from all over the world are among the UN’s goodwill ambassadors. These influential people could help WHO spread messages by sharing those messages and setting agendas for social media users.

5.3 Misinformation in messages

This study also found messages that promoted sensemaking and self-efficacy simultaneously. Many of the tweets containing self-efficacy messages were not backed by scientific evidence. For example, by mid-February, some messages were promoting the use of garlic and lemon as preventive measures to boost immunity. Other messages suggested the use of ginger to boost immunity. Other messages suggested avoiding “foreign food” and “packages from China” in order to curtail the spread of the virus.

It is to be noted that WHO did tweet information debunking the misinformation and the myths. However, an infodemic was inevitable given the global scope of the health crisis. Concerns about the infodemic had been raised by WHO online by February 15, 2020, and subsequently, WHO organized an online conference to crowdsource on infodemic management (Tangcharoensathien et al. 2020). However, recent epidemics (Ebola, Zika, H7N9) show the prevalence of health misinformation and conspiracy theories (Chen et al. 2018; Oyeyemi et al. 2014; Vijaykumar et al. 2018).

5.4 Theoretical and practical implications

The study offers important theoretical implications. It expands the utility and adaptability of the CERC model by examining user generated content on social media platforms such as X. While bulk of research on CERC centers on the organization’s perspective (Tomasi et al. 2023) and communication strategies and tactics appropriate for the distinct phase of the evolving crisis, this study additionally demonstrates how the CERC framework examines how public processes and evaluates crisis information during the early stages of the outbreak in the digital age. Building on prior research (Lwin et al. 2018; Vos and Buckner 2016) consistent with CERC, the current study highlights the crucial role of social media platforms in disseminating public sensemaking and efficacy messages during the initial phases of a public health crisis. The findings revealed the presence of the two core CERC tenets-sensemaking and efficacy in the social media messages shared by users during the early stages of COVID-19 pandemic. These messages reflected the change in people’s awareness, understanding and sensemaking of the crisis, as well as their efforts to build self-efficacy in navigating the crisis. Such sensemaking and efficacy efforts play a key role in helping the public manage the uncertainty that arises from the crisis and assuage the anxiety that is triggered by unprecedented long-term, evolving public health crises like the pandemic.

The findings have practical implications. The study provides a case study on the understanding of the implementation of WHO’s social listening strategies to effectively address public concerns via sensemaking process and promoted protective behaviors during a global pandemic. While social listening can help experts understand dominant conversations, it can also help them infer missing ones. Health organizations can draw on these insights to refine their communication strategies, ensuring that they are not only reactive but also proactive in managing public discourse for future pandemics. Akin to previous findings on health crises (Lu et al. 2021; Tangcharoensathien et al. 2020; Zarocostas 2020), this study found people utilized social media to engage in sensemaking regarding COVID-19. However, WHO’s reactive approach to dominant conversations could be a suboptimal strategy if misinformation is prominent because fake news gets disseminated faster than information correcting the misinformation (Vosoughi et al. 2018). Additionally, health experts and organizations should explore amplifying the prosocial and accurate health messages from opinion leaders from various communities and constituencies (Liu et al. 2022). In some cases, these opinion leaders could also be more effective in correcting misinformation (Liu et al. 2022) than WHO itself.

5.5 Limitations and future direction

While the current study was robust, it has several limitations. First, we only analyzed English-language tweets. The results might not be replicable in other languages. Second, our search terms contained only certain relevant keywords. We certainly expect that our search results did not include all possible tweets about COVID-19. Finally, machine learning strategies have limitations. For example, LDA uses a “bag of words” approach, estimating the probability of certain words occurring together; furthermore, topic interpretation largely depends on human coding. Another challenge associated with machine learning is that online content can fit into multiple categories (Salminen et al. 2019). In addition, the algorithms for machine learning are not necessarily interchangeable across channels, potentially raising issues with labeling. Finally, we used only one social media platform, potentially limiting the generalizability of the results. While the results showed that WHO made every effort.

Future research should look into how health organizations use social media as part of their strategy and look at understanding the benefits and the drawback of implementation of use of latest AI tools and chatbots in combating misinformation. Additionally, scholars should focus on the diffusion speed/rate of health organizations’ messages to better understand the social listening strategies these organizations employ for future health crisis. While the current study employed a content analysis, future studies could use computational methods to calculate diffusion size and speed, in order to understand how quickly sensemaking messages are diffused within the network and better examine health organization’s real-time engagement with the public. This could be done by monitoring and mining analytic data hourly to obtain relative values (see Zhu et al. 2020 for mathematical calculations).