Development of a rubric for assessing delayed diagnosis of appendicitis, diabetic ketoacidosis and sepsis

Kenneth A. Michelson; David N. Williams; Arianna H. Dart; Prashant Mahajan; Emily L. Aaronson; Richard G. Bachur; Jonathan A. Finkelstein

doi:10.1515/dx-2020-0035

Article

Development of a rubric for assessing delayed diagnosis of appendicitis, diabetic ketoacidosis and sepsis

Kenneth A. Michelson , David N. Williams , Arianna H. Dart , Prashant Mahajan , Emily L. Aaronson , Richard G. Bachur and Jonathan A. Finkelstein

Published/Copyright: June 26, 2020

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Diagnosis Volume 8 Issue 2

Abstract

Objectives

Using case review to determine whether a patient experienced a delayed diagnosis is challenging. Measurement would be more accurate if case reviewers had access to multi-expert consensus on grading the likelihood of delayed diagnosis. Our objective was to use expert consensus to create a guide for objectively grading the likelihood of delayed diagnosis of appendicitis, new-onset diabetic ketoacidosis (DKA), and sepsis.

Methods

Case vignettes were constructed for each condition. In each vignette, a patient has the condition and had a previous emergency department (ED) visit within 7 days. Condition-specific multi-specialty expert Delphi panels reviewed the case vignettes and graded the likelihood of a delayed diagnosis on a five-point scale. Delayed diagnosis was defined as the condition being present during the previous ED visit. Consensus was defined as ≥75% agreement. In each Delphi round, panelists were given the scores from the previous round and asked to rescore. A case scoring guide was created from the consensus scores.

Results

Eighteen expert panelists participated. Consensus was achieved within three Delphi rounds for all appendicitis and sepsis vignettes. We reached consensus on 23/30 (77%) DKA vignettes. A case review guide was created from the consensus scores.

Conclusions

Multi-specialty expert reviewers can agree on the likelihood of a delayed diagnosis for cases of appendicitis and sepsis, and for most cases of DKA. We created a guide that can be used by researchers and quality improvement specialists to allow for objective case review to determine when delayed diagnoses have occurred for appendicitis, DKA, and sepsis.

Keywords: diagnostic error; diagnostic safety; error measurement

Corresponding author: Kenneth A. Michelson, MD MPH, Division of Emergency Medicine, Boston Children’s Hospital, 300 Longwood Avenue, BCH 3066, Boston, MA 02115, USA, Phone: +1 6173556624, Fax: +1 6177301024, E-mail: kenneth.michelson@childrens.harvard.edu

Funding source: Agency for Healthcare Research and Quality

Award Identifier / Grant number: 1K08HS026503

Funding source: Agency for Healthcare Research and Quality

Award Identifier / Grant number: 1R01HS024953

Funding source: Agency for Healthcare Research and Quality

Award Identifier / Grant number: 1R18HS026622

Acknowledgments

We deeply appreciate the contributions of our pilot testers and Delphi panelists for volunteering their time and energy to this endeavor. Pilot: Todd Lyons, MD, MPH (Boston Children’s Hospital); Christopher Rees, MD, MPH (Boston Children’s Hospital); Catherine Perron, MD (Boston Children’s Hospital); Joel Hudgins, MD, MPH (Boston Children’s Hospital); Susan Lipsett, MD (Boston Children’s Hospital); Anna Cushing, MD (Boston Children’s Hospital); Jeffrey Neal, MD (Boston Children’s Hospital). Appendicitis panel: Anupam Kharbanda, MD, MS (Children’s Minnesota); Rakesh Mistry, MD, MS (Children’s Hospital Colorado); Andrew Olson, MD (University of Minnesota Medical School); James Moses, MD, MPH (Boston Medical Center); Peter Smulowitz, MD, MPH (Beth Israel Deaconess Medical Center); Shawn Rangel, MD, MSCE (Boston Children’s Hospital); Biren Modi, MD, MPH (Boston Children’s Hospital). DKA panel: Rachel Rempell, MD (Children’s Hospital of Philadelphia); Paul Aronson, MD, MHS (Yale School of Medicine); Nicole Nadeau, MD (Massachusetts General Hospital); Erinn Rhodes, MD, MPH (Boston Children’s Hospital); Michael Agus, MD (Boston Children’s Hospital). Sepsis panel: Fran Balamuth, MD, PhD (Children’s Hospital of Philadelphia); Halden Scott, MD (Children’s Hospital Colorado); Timothy Dribin, MD (Cincinnati Children’s Hospital Medical Center). Drs. Aaronson and Michelson participated in all three panels.

Research funding: Dr. Michelson was funded by award 1K08HS026503 from the Agency for Healthcare Research and Quality, with project support from the Boston Children’s Hospital Office of Faculty Development. Dr. Mahajan was funded by awards 1R01HS024953 and 1R18HS026622 from the Agency for Healthcare Research and Quality.
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Competing interests: Authors state no conflict of interest.
Informed consent: Informed consent was obtained from all individuals included in this study.
Ethical approval: The Institutional Review Board deemed the study not human subjects research and thus it was exempt from review.

References

1. Balogh, EP, Miller, BT, Ball, JR, editors. Improving diagnosis in health care. Washington, D.C.: National Academies Press; 2015.10.17226/21794Search in Google Scholar PubMed

2. Berthelot, S, Lang, ES, Quan, H, Stelfox, HT. Identifying emergency-sensitive conditions for the calculation of an emergency care inhospital standardized mortality ratio. Ann Emerg Med 2014;63:418–24. https://doi.org/10.1016/j.annemergmed.2013.09.016.Search in Google Scholar PubMed

3. Nafsi, T, Russell, R, Reid, CM, Rizvi, SMM. Audit of deaths less than a week after admission through an emergency department: how accurate was the ED diagnosis and were any deaths preventable?. Emerg Med J 2007;24:691–5. https://doi.org/10.1136/emj.2006.044867.Search in Google Scholar PubMed PubMed Central

4. Singh, H, Meyer, AND, Thomas, EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf 2014 Sep;23:727–31. https://doi.org/10.1136/bmjqs-2013-002627.10.1136/bmjqs-2013-002627Search in Google Scholar PubMed PubMed Central

5. Croskerry, P, Sinclair, D. Emergency medicine: a practice prone to error?. CJEM 2001 Oct;3:271–6. https://doi.org/10.1017/S1481803500005765.Search in Google Scholar

6. Kovacs, G, Croskerry, P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med 1999;6:947–52. https://doi.org/10.1111/j.1553-2712.1999.tb01246.x.Search in Google Scholar PubMed

7. Hall, MK, Burns, K, Carius, M, Erickson, M, Hall, J, Venkatesh, A. State of the National Emergency Department Workforce: who provides care where?. Ann Emerg Med 2018;72:302–7. https://doi.org/10.1016/j.annemergmed.2018.03.032.Search in Google Scholar PubMed

8. Mahajan, P, Mollen, C, Alpern, ER, Baird-Cox, K, Boothman, RC, Chamberlain, JM, et al. An operational framework to study diagnostic errors in emergency departments: findings from a consensus panel. J Patient Saf 2019 Nov 25. https://doi.org/10.1097/PTS.0000000000000624.Search in Google Scholar PubMed

9. Warrick, C, Patel, P, Hyer, W, Neale, G, Sevdalis, N, Inwald, D. Diagnostic error in children presenting with acute medical illness to a community hospital. Int J Qual Heal Care 2014;26:538–46. https://doi.org/10.1093/intqhc/mzu066.Search in Google Scholar PubMed

10. Hussain, F, Cooper, A, Carson-Stevens, A, Donaldson, L, Hibbert, P, Hughes, T, et al. Diagnostic error in the emergency department: learning from national patient safety incident report analysis. BMC Emerg Med 2019;19:77. https://doi.org/10.1186/s12873-019-0289-3.Search in Google Scholar PubMed PubMed Central

11. Grubenhoff, JA, Ziniel, SI, Cifra, CL, Singhal, G, McClead, RE, Singh, H. Pediatric clinician comfort discussing diagnostic errors for improving patient safety. Pediatr Qual Saf 2020;5:e259. https://doi.org/10.1097/pq9.0000000000000259.Search in Google Scholar PubMed PubMed Central

12. Schiff, GD, Hasan, O, Kim, S, Abrams, R, Cosby, K, Lambert, BL, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881–7. https://doi.org/10.1001/archinternmed.2009.333.Search in Google Scholar PubMed

13. Singh, H, Khanna, A, Spitzmueller, C, Meyer, AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl) 2019;6:315–23. https://doi.org/10.1515/dx-2019-0012.Search in Google Scholar PubMed

14. Olson, APJ, Graber, ML, Singh, H. Tracking progress in improving diagnosis: a framework for defining undesirable diagnostic events. J Gen Intern Med 2018;33:1187–91. https://doi.org/10.1007/s11606-018-4304-2.Search in Google Scholar PubMed PubMed Central

15. Singh, H, Giardina, TD, Forjuoh, SN, Reis, MD, Kosmach, S, Khan, MM, et al. Electronic health record-based surveillance of diagnostic errors in primary care. BMJ Qual Saf 2012;21:93–100. https://doi.org/10.1136/bmjqs-2011-000304.10.1136/bmjqs-2011-000304Search in Google Scholar PubMed PubMed Central

16. Singh, H, Schiff, GD, Graber, ML, Onakpoya, I, Thompson, MJ. The global burden of diagnostic errors in primary care. BMJ Qual Saf 2017;26:484–94. https://doi.org/10.1136/bmjqs-2016-005401.Search in Google Scholar PubMed PubMed Central

17. Singh, H. Helping health care organizations to define diagnostic errors as missed opportunities in diagnosis. Jt Comm J Qual Patient Saf 2014;40:99–101. https://doi.org/10.1016/S1553-7250(14)40012-6.Search in Google Scholar PubMed

18. Singh, H, Sittig, DF. Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf 2015;24:103–10. https://doi.org/10.1136/bmjqs-2014-003675.Search in Google Scholar PubMed PubMed Central

19. Fink, A, Kosecoff, J, Chassin, M, Brook, RH. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74:979–83. https://doi.org/10.2105/AJPH.74.9.979.Search in Google Scholar PubMed PubMed Central

20. Michelson, KA, Buchhalter, LC, Bachur, RG, Mahajan, P, Monuteaux, MC, Finkelstein, JA. Accuracy of automated identification of delayed diagnosis of pediatric appendicitis and sepsis in the ED. Emerg Med J 2019;36:736–40. https://doi.org/10.1136/emermed-2019-208841.Search in Google Scholar PubMed

21. Sills, MR, Macy, ML, Kocher, KE, Sabbatini, AK. Return visit admissions may not indicate quality of emergency department care for children. Acad Emerg Med 2018 Mar 2;25:283–92. https://doi.org/10.1111/acem.13324.Search in Google Scholar PubMed PubMed Central

22. Lokulo-Sodipe, K, Moon, RJ, Edge, JA, Davies, JH. Identifying targets to reduce the incidence of diabetic ketoacidosis at diagnosis of type 1 diabetes in the UK. Arch Dis Child 2014;99:438–42. https://doi.org/10.1136/archdischild-2013-304818.Search in Google Scholar PubMed

23. Harris, PA, Taylor, R, Thielke, R, Payne, J, Gonzalez, N, Conde, JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. https://doi.org/10.1016/j.jbi.2008.08.010.Search in Google Scholar PubMed PubMed Central

24. Diamond, IR, Grant, RC, Feldman, BM, Pencharz, PB, Ling, SC, Moore, AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014;67:401–9. https://doi.org/10.1016/j.jclinepi.2013.12.002.Search in Google Scholar PubMed

25. Singh, H, Giardina, TD, Meyer, AND, Forjuoh, SN, Reis, MD, Thomas, EJ. Types and origins of diagnostic errors in primary care settings. JAMA Intern Med 2013;173:418–25. https://doi.org/10.1001/jamainternmed.2013.2777.Search in Google Scholar PubMed PubMed Central

26. Singh, H, Thomas, EJ, Wilson, L, Kelly, PA, Pietz, K, Elkeeb, D, et al. Errors of diagnosis in pediatric practice: a multisite survey. Pediatrics 2010;126:70–9. https://doi.org/10.1542/peds.2009-3218.Search in Google Scholar PubMed PubMed Central

27. Cifra, CL, Ten Eyck, P, Dawson, JD, Reisinger, HS, Singh, H, Herwaldt, LA. Factors associated with diagnostic error on admission to a PICU: a pilot study. Pediatr Crit Care Med 2020 May;21:e311-5. https://doi.org/10.1097/PCC.0000000000002257.Search in Google Scholar PubMed PubMed Central

28. Bhise, V, Meyer, AN, Singh, H, Wei, L, Russo, E, Al-Mutairi, A, et al. Errors in diagnosis of spinal epidural abscesses in the era of electronic health records. Am J Med 2017;130:975–81. https://doi.org/10.1016/j.amjmed.2017.03.009.Search in Google Scholar PubMed

29. Michelson, KA, Lyons, TW, Bachur, RG, Monuteaux, MC, Finkelstein, JA. Timing and location of emergency department revisits. Pediatrics 2018;141:e20174087. https://doi.org/10.1542/peds.2017-4087.Search in Google Scholar PubMed

30. Bui, H, To, T, Stein, R, Fung, K, Daneman, D. Is diabetic ketoacidosis at disease onset a result of missed diagnosis?. J Pediatr 2010;156:472–7. https://doi.org/10.1016/j.jpeds.2009.10.001.Search in Google Scholar PubMed

Supplementary Material

The online version of this article offers supplementary material (DOI:https://doi.org/10.1515/dx-2020-0035).

Received: 2020-03-16

Accepted: 2020-05-14

Published Online: 2020-06-26

Published in Print: 2021-05-26

You are currently not able to access this content.

Supplementary Material Details

Articles in the same Issue

https://doi.org/10.1515/dx-2020-0035

Keywords for this article

diagnostic error; diagnostic safety; error measurement