Emergency Manager Perceptions of the Effectiveness and Limitations of Mass Notification Systems: A Mixed Method Study

1 Introduction

In a time of crisis or disaster, information is a critical commodity. Decision-making is driven by information. Shelter in place or evacuate, direction of evacuation, urgency of action; all represent pre-crisis decisions predicated on quality information. Post-crisis information can similarly mean the difference between life and death—acquiring water, fuel, and food, access to shelter, health-care locations, recovery facilities, and similar, are all outcomes which require information. Even in non-crisis situations, information helps to drive decision-making (Haddow and Haddow 2008; Kapucu 2008).

There are many modalities for crisis period information distribution. Traditional media such as radio and television can provide detailed perspectives but presuppose electricity (at both distribution and reception points), attention of the public, and timely information. Newspapers can provide the highest level of detail but are often obviated due to the time lag between publication and acquisition, in conjunction with delivery requirements. Internet news sources are a tremendous source of timely information but require both power and signal supply (at both distribution and reception points), and the reliability of information may be questionable.

Mass notification systems (MNS) utilize cell phones and Short Message Service (SMS) or text messaging. This approach represents an important step in crisis communications (Leiva 2014) since many people have cell phones with them at all times and there is a high level of redundancy in signal distribution. A brief piece of information, texted to a large portion of the population, can provide preliminary alerts or key pieces of information (Madden 2015). Cell phones are not the perfect information distribution mechanism—they are vulnerable to spam, hacking, and false attribution.

Bristow (2004) suggested that minimizing the consequences of a disaster is a direct artifact of effective communication between first responders and the community. While there is limited scholarly literature on the effectiveness of mass notification systems, the available studies indicate that an MNS can offer significant benefits to a community. Case studies testing an MNS demonstrate their importance and potential for mitigating the consequences of disasters (Edwards, Cuthbertson, and Peterson 2011; Kapucu 2008; Martin and Rice 2012).

Anecdotally, utilizing an emergency alert system can make a major difference in disasters. The recent California wildfires represent a key example. When wildfires struck the Sonoma and Napa county region in December of 2017, some local emergency managers decided not to use the emergency alert systems available to them (St. John 2018). They referenced a concern about inducing panic behavior and a lack of connectivity. Lake County officials made a different decision—they issued an emergency alert, broadcast through cell phones, ordering a mandatory evacuation. There were zero fatalities in that county and 33 deaths in Sonoma County (Davis and Somashekar 2017). Although Sonoma County is more populated and was hit harder by the fire, the lack of evacuation notices placed residents in a position where decisions were made with limited information.

Unfortunately the lessons of the Sonoma and Napa wildfires were not enacted by the following year. During November of 2018, the Camp Fire became the world’s costliest disaster of the year (Rice 2019). A fast moving fire, exacerbated by dry conditions and 50 mile-per-hour winds, the Camp Fire quickly set upon Paradise, California. Within just a few hours, the city of Paradise was destroyed and the fire quickly moved on to other cities. Some emergency alerts were sent but distribution was inconsistent. Approximately 25% of Paradise residents had opted in to the emergency alert (CodeRed) system, however, many received no alerts. Some citizens received a warning, some received a more serious evacuation notice, and a large portion of residents received no contact at all. Only half of the cell phones in the area received any kind of alert at any time. Frustratingly, a number of citizens received alerts after the fire had destroyed their homes. Text messaging on mass notification systems are often first-in last-out, resulting in impracticable delays in text delivery. Seventeen cell phone towers were destroyed by the fire and 66 were damaged, impeding alert distribution.

The MNS is not a panacea of information distribution. There are important limitations and regulations, dictated by Federal Emergency Management Agency (FEMA) and the Federal Communications Commission (FCC), which define how and when notifications can be sent (Levia 2014). Additionally, satellites and towers must be in working order, phone carriers must have agreed to communicate messages (company participation is voluntary), and municipalities must have an agreement in place with FEMA. Limitations associated with communicating across cell phones via text messages also exist (brevity of messages, presumption of power, and recipient responsiveness). Effective decision-making is predicated on knowledge management (von Lubitz, Beakley, and Patricelli 2008). Decisions made under rapidly changing conditions are especially challenging. Boyd’s (1976) Observe/Orient/Decide/Act (OODA) Loop—Observe, Orient, Decide, Act—provides a framework through which decision-making can be better understood. The intersection of knowledge management and decision-making is especially important as emergency managers make communication choices in alerts.

An emergency alert system can be used mistakenly. This failure has the short term effect of causing undue alarm, and potentially panic, coupled with the long term effect of reducing trust and responsiveness to emergency alerts. The Hawaii ballistic missile crisis of 2018 is an example of an emergency alert failure (Fisher 2018; Wang 2018). During late 2017, North Korea tested several ballistic missiles and claimed their devices could reach Hawaii. During an internal drill, an employee misunderstood the nature of the drill, believed that the hypothetical missile attack was a real attack, and sent an emergency alert stating: “Ballistic missile threat inbound to Hawaii. Seek immediate shelter. This is not a drill.” The message briefly caused panic as news media interrupted broadcasts to distribute the message. Local and state leaders quickly informed media of the mistake, however, 38 min elapsed before a correction emergency alert was sent. The Federal Communications Commission had to approve a revisionist alert as federal guidelines precluded a state agency from issuing a correction.

Given the potential role that mass notification systems can play in disasters and periods of crises, a greater understanding of their role, efficacy, and utility is needed.

The following sections review the available literature on MNS, describe a set of research methods and analysis on MNS enrollment, review qualitative perspectives of emergency managers on MNS, and posit conclusions and policy recommendations.

2 Literature Review

Effective communication between officials and citizens prior to, during, and after crisis situations can increase public safety and expedite response and recovery. MNS are intended to connect the public at large with emergency management personnel, providing a vehicle for relaying critical information. The effectiveness of an MNS is predicated on a number of variables, including contact information possessed by emergency management systems, speed with which information is provided, utility of the information, and the urgency of the communications (Haddow and Haddow 2008).

The proliferation of social media has produced an important network for reaching persons in threat of a crisis. Jurisdictions typically rely on cell phone text systems and traditional media (i.e. local television and radio) to communicate alerts. The omnipresence of social media platforms (such as Facebook, Instagram, Twitter, and others) provide a communication vehicle that may surpass traditional alert networks. Carvin et al. (2010) and Martin and Rice (2012) stress the role of social media. They contend that the sheer scope of social media and the pervasive, always-on modality of social media platforms, can reach far more people than crisis communications via traditional media (such as radio, television, and newspapers). Social media is not risk-free. The rapid dissemination of news must be contrasted with the possible dissemination of false-news. Malware and hacking also represent risks. Incorporating change-authentication and attribution systems on mass notification systems, with tagging such as block chain accounting, would likely further enhance the security of messaging systems.

Pechta, Brandenburg, and Seeger (2010) support the utility of social media as a communication device in crisis situations. They advocate a multi-channel modality of crisis communications where there is a feedback system. This approach changes constituents from recipients of information to participants in the information distribution process. There is a critical flaw, however, in this approach. Citizens may not exercise the same judgment and discretion as emergency management professionals. Although there may be scenarios, such as a tornado, where eye witnesses can provide better information than a remote expert, in pre-disaster communication casting the widest possible net has many benefits but also risks. A panicky citizen Tweet (e.g. “Evacuate Now or Die”), if re-Tweeted on high profile platforms, can produce a dangerous and unnecessary flight.

There have been several studies of emergency notifications systems at colleges and universities. The findings of these studies generally note the importance of complete contact databases and the rapidity with which alerts are broadcast (Elsass, McKenna, and Schildkraut 2016; Madden 2015). Colleges and university campuses are not comparable to broader communities for a number of reasons. Most importantly, a college or university system has contact information for all members of their community by necessity. Students, faculty, and staff are beholden to a registration and/or payroll system which collects contact information as a matter of course. While a small percentage of the contact information is inaccurate or outdated, the vast majority of college and university community members are reachable. That is not necessarily true for a city, town, county, or even state where only a fraction of residents have provided their contact information to emergency authorities. Those local government institutions which do have contact information (i.e. city, county, or state tax collectors, schools, utilities) may be reluctant to share that information absent a judicial directive.

In his study of Florida’s response to a prolific Hurricane season (in 2004 Florida was hit with four successive hurricanes), Kapucu (2008) found that effective disaster planning, communication between government officials and first responders, and community alert strategies were critical. Kapucu notes that citizens often do not listen to alerts. There are many possible explanations for this behavior: citizens may assume the alerts apply to others, that their government is mistaken, that the risk is exaggerated, or they experience alert fatigue. As noted in the federal by the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina (2006) thousands of people stayed in New Orleans despite dire warnings regarding Hurricane Katrina. Sometimes the decision to ignore warnings is purposive (plans to “ride out the storm”). For some citizens, as evinced in the studies that followed Hurricane Katrina, many citizens lack the capacity or means to leave their homes without government agencies providing, and effectively advertising, transportation and temporary residence modalities (Bipartisan Committee Report 2006). Crisis fatigue in a place frequently wracked with weather related situations may also impede exhortations from government officials.

As with many technology choices, it is entirely possible that there is no single best approach. Malizia et al. (2010) reviewed emergency notification system protocols and attempted to define an ontology for mass notification, revolving around technology and need. They noted a high degree of variability in the various technologies available, with some concentrating on landline phones, some on cell phones, some SMS/text centric, some fax, and some based on Internet communiques (such as emails or alerts). The authors contend that matching community needs with technology access and sources is an important step in selecting the right notification system. They conclude that there is no single “right” system for citizen oriented disaster communications. Instead, jurisdictions must go through a complex calculus evaluating cost, effectiveness, range, technology access, and communication needs. The mode of communication represents one element influencing effectiveness. Another important element in the communication process is the decision-making of the communicators. Emergency Managers are frequently responsible for coordinating alerts and emergency communications. Key decision points in the mass notification process include: pre-sensitization warnings, word choices, urgency evident in the message, frequency and timing of communiques, and integration with other media sources. This decision-making process is complex and influenced by an array of factors.

The deficits in the existing literature on MNS indicate several important areas of research and raise key research questions. The utility and effectiveness of MNS, the variables influencing MNS implementation, and emergency manager perceptions of MNS warrant study. One of the key gaps in the literature is the perceptions of emergency managers. This is important as those persons play a critical role in the utilization of these systems and they should have high knowledge levels regarding their potential. Research questions addressing these issues include:

How are persons enrolled in a Mass Notification System?

How does a jurisdiction select a Mass Notification Systems?

How are messages sent via Mass Notification Systems?

What is the role of the Emergency Manager in defining and distributing notifications?

The following description of methods and population addresses these topics.

3 Research Methodology

The role and efficacy of MNS warrants research, particularly concerning enrollment and MNS utilization. The following paragraphs describe a research methodology for collecting and analyzing relevant data. A mixed methods approach was employed to facilitate triangulation of findings (Yin 2014). In this approach, alignment between quantitative and qualitative findings enhances outcome validity. Additionally, each data set provides insight into the other and can facilitate interpretation.

4 Results

To address research questions on Mass Notification System enrollment, utility, and usage, survey data were collected from emergency coordinators across Virginia. Forty coordinators (out of 129 in the entire population) returned completed surveys (for a response rate of 31%). Additionally, a series of interviews were conducted with emergency managers to provide insight into survey findings and key project constructs.

Survey results indicated a wide range of participants in terms of populations served. Seventeen respondents represented populations between 5000 and 50,000, ten served populations between 50,000 and 100,000, seven represented populations between 100,000 and 250,000, five served populations between 250,000 and 500,000, and one served a population over one million. The median household income for responding jurisdictions was approximately $62,000 with a standard deviation of $24,000.

The survey posed several questions about the nature of the mass notification system utilized in the responding district. A significant majority of responding districts (33 of 40, or 82.5%) are opt-in. The remaining seven (17.5%) are not opt-in. The brand of mass notification system utilized was requested. Half of participating districts (N = 20, 50%) use the trademarked brand Everbridge. CodeRed was the second most commonly utilized brand (N = 10, 25%). The remaining districts reported using a different brand (N = 10, 25%).

Participants were asked how often they send notifications. Most jurisdictions (80%) have sent less than 10 alerts in the past year. Two jurisdictions sent between 100 and 500 and three sent between 1000 and 3000. One jurisdiction reported sending 5600 notifications in one year, which averages to over 15 notifications sent per day. The jurisdictions sending hundreds of notifications each year reported that the vast majority of notifications were traffic related. Traffic notifications informed users of accidents, construction, weather issues, or other delays. The granularity of data for certain types of communication are quite high resulting in difficulties of non-homogeneous characterization.

A series of correlations were conducted between key variables. Correlations (bivariate) were run between Population, Income per Capita, Poverty Rate, and Registered Accounts in MNS. As demonstrated in Table 2, Population, Income per Capita, and Poverty Rate are all significantly correlated. Registered Accounts in MNS is not significantly correlated, suggesting that large jurisdictions do not necessarily have more registered MNS accounts.

Analysis of Variance (ANOVA) tests were conducted between Population and Registered MNS Accounts. The F score was 5.232 (at 33 degrees of freedom), with a significance score of 0.096. The outcome of the ANOVA was not statistically significant indicating that jurisdiction population does not predict registered account numbers.

The lack of statistical significance in the ANOVA is likely an artifact of several small jurisdictions which claim large numbers of MNS accounts. The finding that population size does not predict MNS accounts is interesting and potentially important. This indicates that some jurisdictions have been successful in garnering account registrations while others, including several large jurisdictions, have not been successful. There are large jurisdictions, encompassing several hundred thousand residents, with low numbers of accounts registered for the MNS. If a significant event were to occur, the population of these jurisdictions would not be well informed via the MNS.

Semi-structured interviews were conducted with a non-systematic subsample of survey participants. These interviews drew on findings from the survey analysis and addressed issues of MNS utilization, selection, effectiveness, obstacles, and needs. Several themes emerged from the interviews. The emergency managers that were interviewed overwhelmingly felt that MNS represented an important and potentially critical modality of communication with the public. During a time of crisis, particularly a rapidly evolving scenario (such as a weather emergency), MNS could significantly enhance public safety. Participants also indicated that MNS should be used infrequently to avoid development of notification fatigue. When told that some jurisdictions used MNS hundreds, even thousands of times per year, participants were surprised and noted that overuse could lead to ignoring important messages.

The emergency managers interviewed consistently expressed concern about enrollment issues. While they were disappointed that a small portion of their constituency voluntarily enrolled for MNS communications, they felt greater recruitment and education would enhance enrollment. The obstacle to these recruiting methods revolved around funding. Unfortunately, none of the interview participants played a significant role in the selection of the MNS for their jurisdiction, noting this was a jurisdictional bidding and contract process. Several expressed a desire for more flexibility in issuing notifications. Some participants suggested that collaborating with public safety agencies, particularly fire, police, and emergency medical services, could extend the capacity of the MNS.

When asked about the decision-making process of which issues rise to the level of warranting MNS consideration, variation existed among interview participants. Some used pre-defined protocols that list some of the issues that can be considered for MNS. Other participants described a subjective process which weighed potential risk and consequences with concerns about unnecessary communication and alarm. No participant expressed regret or concern about the decision-making process utilized by their jurisdiction.

5 Conclusions

The rapidity and urgency of crisis situations necessitates effective communication modalities. A MNS uses SMS text messages sent to cell phones to provide critical information to constituents who have enrolled in the MNS. While federal agencies such as the National Weather Service can send weather alerts to all cell-phones in a region, an MNS requires enrollment. While the literature on crisis communication is extensive, the research on the effectiveness and limitations of MNS is nascent. Studies on discrete communities such as college and university campuses have demonstrated that an MNS is a viable communication modality (Elsass et al. 2016; Madden 2015). Research conducted in jurisdictions like cities and towns is less defined—while an MNS can be a highly effective communication tool, enrollment and messaging utilization play an important role (Bristow 2004; Leiva 2014). Critical points in the MNS process are emergency manager decision-making, citizen recruitment and enrollment, and messaging utilization.

Recent examples of California wildfires underscore the importance of timing and decision-making. In some communities, MNS warnings provided important information to residents who were able to evacuate prior to extremely dangerous wildfires (Davis and Somascheker 2017). In other communities, warnings were either not issued or came too late to matter (St. John 2018). Although a raging and rapidly moving wildfire is an extreme example, the urgency and involuntariness of crisis demonstrates the importance of communication between decision makers and their constituents.

Multiple gaps exist in the literature regarding MNS. The role and decision-making process of emergency managers, the persons generally responsible for utilization of the MNS, is poorly understood. The variables influencing MNS type and selection are also understudied. Exogenous variables such as population and demographics have not been considered relative to MNS usage.

The extant study attempted to address some of these deficits using a mixed method approach. A survey of all emergency managers (relying on a census rather than a sample) was conducted. Questions were posed addressing MNS brand, usage, and enrollment issues. Subsequent to the survey, a convenience sample of follow-up interviews were conducted to provide depth on the survey results.