A Capabilities-Based Framework for Disaster Response Exercise Design and Evaluation: Findings from Oil Spill Response Exercises
-
Brandon Greenberg
Abstract
The responder community must be ready to respond quickly and effectively in the event of a disaster. In order to maintain readiness, many disaster response communities exercise their response capabilities on a regular basis. The critical challenge is to design, conduct, and evaluate exercises in a manner that effectively tests responders’ readiness and generates lessons that can improve readiness. This paper describes a framework to enable assessment of response readiness through evaluation of critical capabilities in exercises. It was developed for oil spill response based on the observation and analysis of four response exercises. The framework (1) identifies critical capabilities that lead to readiness for spill response, and maps them to (2) exercise design components that test each capability and (3) evaluation measures to evaluate each capability within an exercise. The framework enables continuous improvement by linking the evaluation of exercises to the critical capabilities required of an oil spill response organization; by evaluating the performance of specific capabilities, areas for improvement are clearly identified and can be re-tested in a future exercise. While the findings are necessarily specific to oil spill response, the principles apply to any disaster response context.
Acknowledgments
This work was sponsored by the Department of the Interior Bureau of Safety and Environmental Enforcement (BSEE) contract E14PC00045. The authors gratefully acknowledge the support of many individuals at BSEE, including Paul Meyer, Jason Langteau, Eric Miller, David Moore and several others in Washington, DC; and Alton Bates, John Calvin, Gary Petrae, Deserie Soliz, Thomas Tregle, and the rest of the team in New Orleans, LA. We are also extremely grateful to the participants in the four exercises we observed and to the operators, contractors, and government agencies involved; they are not named to protect their confidentiality. We thank Samuel Babbitt for his support in the literature review portion of this work. Finally, we thank John Harrald and Gregory Shaw for lending their valuable expertise throughout the project.
Andersson, Annika, Eric D. Carlstrom, Bengt Ahgren, and Johan M. Berlin. 2014. “Managing Boundaries at the Accident Scene – A Qualitative Study of Collaboration Exercises.” International Journal of Emergency Services 3 (1): 77–94.10.1108/IJES-02-2013-0003Search in Google Scholar
Arnold, Jacqueline J., LeAnn M. Johnson, Sharon J. Tucker, James F. Malec, Sarah E. Henrickson, and William F. Dunn. 2009. “Evaluation Tools in Simulation Learning: Performance and Self-Efficacy in Emergency Response.” Clinical Simulation in Nursing 5 (1): e35–e43.10.1016/j.ecns.2008.10.003Search in Google Scholar
Babus, Sylvia, Kathryn Hodges, and Erik Kjonnerod. 1997. “Simulations and Institutional Change: Training US Government Professionals for Improved Management of Complex Emergencies Abroad.” Journal of Contingencies and Crisis Management 5 (4): 231–240.10.1111/1468-5973.00061Search in Google Scholar
Bakacsi, Gyula 2010. “Managing Crisis: Single-Loop or Double-Loop Learning?” Strategic Management 15 (3): 003–009.Search in Google Scholar
Beamon, Benita M., and Burcu Balcik. 2008. “Performance Measurement in Humanitarian Relief Chains.” International Journal of Public Sector Management 21 (1): 4–25.10.1108/09513550810846087Search in Google Scholar
Berlin, Johan M., and Eric D. Carlström. 2013. “The Dominance of Mechanistic Behaviour: A Critical Study of Emergency Exercises.” International Journal of Emergency Management 9 (4): 327–350.10.1504/IJEM.2013.059878Search in Google Scholar
Berlin, Johan M., and Eric D. Carlström. 2015. “The Three-Level Collaboration Exercise – Impact of Learning and Usefulness.” Journal of Contingencies and Crisis Management 23 (4): 257–265.10.1111/1468-5973.12070Search in Google Scholar
Biddinger, Paul D., Rebecca O. Cadigan, Bruce S. Auerbach, Jonathan L. Burstein, Elena Savoia, Michael A. Stoto, and Howard K. Koh. 2008. “On Linkages: Using Exercises to Identify Systems-Level Preparedness Challenges.” Public Health Reports 123 (1): 96–101.10.1177/003335490812300116Search in Google Scholar
Boin, Arjen, Celesta Kofman-Bos, and Werner Overdijk. 2004. “Crisis Simulations: Exploring Tomorrow’s Vulnerabilities and Threats.” Simulation & Gaming 35 (3): 378–393.10.1177/1046878104266220Search in Google Scholar
Brady, Thomas F (2003)."Emergency Management: Capability Analysis of Critical Incident Response," Winter Simulation Conference. 1863–1867 http://dl.acm.org/citation.cfm?id=1031075.Search in Google Scholar
Cashman, Dennis 2011. “Designing Operations-Based Exercises Based on Artificialities and Realism.” In 2011 International Oil Spill Conference. American Petroleum Institute http://www.ioscproceedings.org/doi/abs/10.7901/2169-3358-2011-1-42.10.7901/2169-3358-2011-1-42Search in Google Scholar
Cashman, Dennis, Jackie Stephens, and LT. Thomas Boyles. 2003. “Designing, Executing, and Evaluating a US Government-Led National Preparedness for Response Exercise Program (PREP) Drill.” In 2003 International Oil Spill Conference., 597–602. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-2003-1-597.10.7901/2169-3358-2003-1-597Search in Google Scholar
Chi, Chih-Hsien, Wen-Hsin Chao, Chia-Chang Chuang, Ming-Che Tsai, and Liang-Miin Tsai. 2001. “Emergency Medical Technicians’ Disaster Training by Tabletop Exercise.” The American Journal of Emergency Medicine 19 (5): 433–36.10.1053/ajem.2001.24467Search in Google Scholar
Cooper, Simon, Robyn Cant, Joanne Porter, Ken Sellick, George Somers, Leigh Kinsman, and Debra Nestel. 2010. “Rating Medical Emergency Teamwork Performance: Development of the Team Emergency Assessment Measure (TEAM).” Resuscitation 81 (4): 446–52.10.1016/j.resuscitation.2009.11.027Search in Google Scholar
Dausey, David J., James W. Buehler, and Nicole Lurie. 2007. “Designing and Conducting Tabletop Exercises to Assess Public Health Preparedness for Manmade and Naturally Occurring Biological Threats.” BMC Public Health 7: 92.10.1186/1471-2458-7-92Search in Google Scholar
Descatha, Alexis, Thomas Loeb, François Dolveck, Nathalie-Sybille Goddet, Valerie Poirier, and Michel Baer. 2009. “Use of Tabletop Exercise in Industrial Training Disaster.” Journal of Occupational and Environmental Medicine 51 (9): 990–991.10.1097/JOM.0b013e3181b3265fSearch in Google Scholar
Donahue, Amy K., and Robert V. Tuohy. 2006. “Lessons We Don’t Learn: A Study of the Lessons of Disasters, Why We Repeat Them, and How We Can Learn Them.” Homeland Security Affairs II (2): 1–28.Search in Google Scholar
Draft National Preparedness for Response Exercise Program Guidelines. 2015. USCG, EPA & BSEEhttp://www.cleangulfassoc.com/uploads/files/PREP Guidelines MASTER_March 2015.pdf.Search in Google Scholar
Eisenhardt, Kathleen M 1989. “Building Theories from Case Study Research.” Academy of Management Review 14 (4): 532–550.10.4135/9781473915480.n52Search in Google Scholar
Fautua, David, Sae Schatz, Emilie Reitz, and Patricia Bockelman. (2014)."Institutionalizing Blended Learning into Joint Training: A Case Study and Ten Recommendations," Interservice/Industry Training, Simulation, and Education Conference 2014 http://www.iitsec.org/about/PublicationsProceedings/Documents/BP_ED_14208_Paper.pdf.Search in Google Scholar
Franks, Nicole, Scott Knutson, Heather A. Parker, and Fred LeJeune. (2011)."Plan-Holder Exercises – How to Recapture a Lost Opportunity for Area Plan Improvement," 2011 International Oil Spill Conference. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-2011-1-155.10.7901/2169-3358-2011-1-155Search in Google Scholar
Gagné, Robert Mills, Walter W. Wager, Katharine C. Golas, and John M. Keller. 2005. Principles of Instructional Design., 5th ed. Wiley Online Library. DOI:10.1002/pfi.4140440211.Search in Google Scholar
Gebbie, Kristine M., Joan Valas, Jacqueline Merrill, and Stephen Morse. 2006. “Role of Exercises and Drills in the Evaluation of Public Health in Emergency Response.” Prehospital and Disaster Medicine 21 (3): 173–182.10.1017/S1049023X00003642Search in Google Scholar
Gralla, Erica, Brandon Greenberg, Paule Voevodsky, John Harrald, Gregory Shaw, and Samuel Babbitt. (2016).A Capabilities-Based Framework for Designing and Evaluating Oil Spill Response Exercises. Technical ReportU.S. Bureau of Safety and Environmental Enforcement.Search in Google Scholar
Harrald, John R 2006. Testimony for the Senate Homeland Security Government Affairs Committee on Hurricane Katrina: Recommendations for Reform. March.Search in Google Scholar
Homeland Security Exercise and Evaluation Program. 2013. Policy. U.S. Department of Homeland Securityhttps://www.preptoolkit.org/web/hseep-resources/home.Search in Google Scholar
Hovden, Jan, Fred Storseth, and Ranveig Kviseth Tinmannsvik. 2011. “Multilevel Learning from Accidents - Case Studies in Transport.” Safety Science 49 (1): 98–105.10.1016/j.ssci.2010.02.023Search in Google Scholar
Jackson, Brian A., and Shawn McKay. 2011. “Preparedness Exercises 2.0: Alternative Approaches to Exercise Design That Could Make Them More Useful for Evaluating and Strengthening Preparedness.” Homeland Security Affairs 7. http://calhoun.nps.edu/handle/10945/24976.Search in Google Scholar
Jönsson, Henrik, Marcus Abrahamsson, and Henrik Johansson. 2007. “An Operational Definition of Emergency Response Capabilities.” In Proceedings of Disaster Recovery and Relief: Current & Future Approaches (TIEMS 2007). The International Emergency Management Society http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:An+Operational+Definition+of+Emergency+Response+Capabilities#0.Search in Google Scholar
Langley, Ann 1999. “Strategies for Theorizing from Process Data.” Academy of Management Review 24 (4): 691–710.10.5465/amr.1999.2553248Search in Google Scholar
Mendonça, David 2007. “Decision Support for Improvisation in Response to Extreme Events: Learning from the Response to the 2001 World Trade Center Attack.” Decision Support Systems 43 (3): 952–967.10.1016/j.dss.2005.05.025Search in Google Scholar
Miles, Matthew B., and A. Michael Huberman. 1984. Qualitative Data Analysis: A Sourcebook of New Methods. Sage Publications https://lccn.loc.gov/84002140.Search in Google Scholar
National Incident Management System. 2008. U.S. Department of Homeland Securityhttps://www.fema.gov/pdf/emergency/nims/NIMS_core.pdf.Search in Google Scholar
National Preparedness for Response Exercise Program Guidelines. 2002.USCG, EPA, & DOI.Search in Google Scholar
National Response Framework. 2013. U.S. Department of Homeland Security May. https://www.fema.gov/media-library-data/20130726-1914-25045-1246/final_national_response_framework_20130501.pdf.Search in Google Scholar
Ott, Gary L., David C. Haynes, and Robert G. Pond. 1999. “Considering ‘Best Response’ Capabilities in Area Contingency Plans.” In 1999 International Oil Spill Conference., 7–13. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-1999-1-7.10.7901/2169-3358-1999-1-7Search in Google Scholar
Rosen, Michael A., Sallie J. Weaver, Elizabeth H. Lazzara, Eduardo Salas, Teresa Wu, Salvatore Silvestri, Nicola Schiebel, Sandra Almeida, and Heidi B. King. 2010. “Tools for Evaluating Team Performance in Simulation-Based Training.” Journal of Emergencies, Trauma, and Shock 3 (4): 353.10.4103/0974-2700.70746Search in Google Scholar
Savoia, Elena, Marcia A. Testa, Paul D. Biddinger, Rebecca O. Cadigan, Howard Koh, Paul Campbell, and Michael A. Stoto. 2009. “Assessing Public Health Capabilities during Emergency Preparedness Tabletop Exercises: Reliability and Validity of a Measurement Tool.” Public Health Reports 124: 138–148.10.1177/003335490912400117Search in Google Scholar
Silenas, Rasa, Ralitsa Akins, Alan R. Parrish, and Janine C. Edwards. 2008. “Developing Disaster Preparedness Competence: An Experiential Learning Exercise for Multiprofessional Education.” Teaching and Learning in Medicine: An International Journal 20 (1): 62–68.10.1080/10401330701798311Search in Google Scholar
Smith, Eleanor M., Ford J. Kevin, and Steve W. J. Kozlowski. 1997. “Building Adaptive Expertise: Implications for Training Design Strategies.” In Training for a Rapidly Changing Workplace Applications of Psychological Research., edited by Meguel A. Quinones, and Ebrenstrin Addie, 89–118. Washington, DC, US: American Pschological Association.10.1037/10260-004Search in Google Scholar
Spradley, James P. 1980. “Participant Observation.” Qualitative Research 3: 1–16.Search in Google Scholar
Target Capabilities List. 2007. A Companion to the National Preparedness Guidelines U.S. Department of Homeland Security papers2://publication/uuid/B07AFDA1-93F0-40AC-BE67-5B1700D68031.Search in Google Scholar
U.S. Coast Guard Incident Management Handbook. 2001. US Coast Guard.Search in Google Scholar
Yin, Robert K. 2014. Case Study Research: Design and Methods., 5th ed. SAGE: Los Angeles.Search in Google Scholar
Zhou, Bojia, Gang Sun, Xiaoyong Zhang, Jianhua Xu, Junyan Lai, Xiaoxia Du, Masafumi Hosokawa, Haruo Hayashi, and Reo Kimura. 2015. “Development of Web-Based Tabletop Emergency Earthquake Exercise System.” Journal of Disaster Research 10 (2): 217–224.10.20965/jdr.2015.p0217Search in Google Scholar
©2016 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Opinion
- Young Adults’ Security Perceptions: Troubling, but an Opportunity for the Response Field
- Research Articles
- A Capabilities-Based Framework for Disaster Response Exercise Design and Evaluation: Findings from Oil Spill Response Exercises
- Pets and Evacuation: An Ongoing Challenge in Disasters
- The Civil Air Patrol: The Struggle for Identity and Relevance
Articles in the same Issue
- Opinion
- Young Adults’ Security Perceptions: Troubling, but an Opportunity for the Response Field
- Research Articles
- A Capabilities-Based Framework for Disaster Response Exercise Design and Evaluation: Findings from Oil Spill Response Exercises
- Pets and Evacuation: An Ongoing Challenge in Disasters
- The Civil Air Patrol: The Struggle for Identity and Relevance
