Identifying error types in visual diagnostic skill assessment
-
Cécile J. Ravesloot
,
Anouk van der Gijp
Abstract
Background:
Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests.
Methods:
Medical students in a radiology elective (n=109) took a radiology test including 11–13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated.
Results:
The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen’s κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases.
Conclusions:
A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: SURF Foundation (Grant Number: TTL 11.0269).
Employment or leadership: None declared.
Honorarium: None declared.
Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.
References
1. Bhargavan M, Sunshine JH. Workload of radiologists in the United States in 2002–2003 and trends since 1991–1992. Radiology 2005;236:920–31.10.1148/radiol.2363041316Suche in Google Scholar
2. Bhargavan M, Kaye AH, Forman HP, Sunshine JH. Workload of radiologists in United States in 2006–2007 and trends since 1991–1992. Radiology 2009;252:458–67.10.1148/radiol.2522081895Suche in Google Scholar
3. van der Gijp A, van der Schaaf MF, van der Schaaf IC, Huige JC, Ravesloot CJ, van Schaik JP, et al. Interpretation of radiological images: towards a framework of knowledge and skills. Adv Health Sci Educ Theory Pract 2014;19:565–80.10.1007/s10459-013-9488-ySuche in Google Scholar
4. Nodine CF, Kundel HL, Mello-Thoms C, Weinstein SP, Orel SG, Sullivan DC, et al. How experience and training influence mammography expertise. Acad Radiol 1999;6:575–85.10.1016/S1076-6332(99)80252-9Suche in Google Scholar
5. Norman GR, Coblentz CL, Brooks LR, Babcook CJ. Expertise in visual diagnosis: a review of the literature. Acad Med 1992;67:S78–83.10.1097/00001888-199210000-00045Suche in Google Scholar PubMed
6. Taylor PM. A review of research into the development of radiologic expertise: implications for computer-based training. Acad Radiol 2007;14:1252–63.10.1016/j.acra.2007.06.016Suche in Google Scholar PubMed
7. Gruen RL, Jurkovich GJ, McIntyre LK, Foy HM, Maier RV. Patterns of errors contributing to trauma mortality: lessons learned from 2594 deaths. Ann Surg 2006;244:371–80.10.1097/01.sla.0000234655.83517.56Suche in Google Scholar PubMed PubMed Central
8. Wechsler RJ, Spettell CM, Kurtz AB, Lev-Toaff AS, Halpern EJ, Nazarian LN, et al. Effects of training and experience in interpretation of emergency body CT scans. Radiology 1996;199:717–20.10.1148/radiology.199.3.8637994Suche in Google Scholar PubMed
9. Guly HR. Diagnostic errors in an accident and emergency department. Emerg Med J 2001;18:263–9.10.1136/emj.18.4.263Suche in Google Scholar PubMed PubMed Central
10. Rhea JT, Potsaid MS, DeLuca SA. Errors of interpretation as elicited by a quality audit of an emergency radiology facility. Radiology 1979;132:277–80.10.1148/132.2.277Suche in Google Scholar PubMed
11. Gwynne A, Barber P, Tavener F. A review of 105 negligence claims against accident and emergency departments. J Accid Emerg Med 1997;14:243–5.10.1136/emj.14.4.243Suche in Google Scholar
12. Hu CH, Kundel HL, Nodine CF, Krupinski EA, Toto LC. Searching for bone fractures: a comparison with pulmonary nodule search. Acad Radiol 1994;1:25–32.10.1016/S1076-6332(05)80780-9Suche in Google Scholar
13. Krupinski EA. Visual scanning patterns of radiologists searching mammograms. Acad Radiol 1996;3:137–44.10.1016/S1076-6332(05)80381-2Suche in Google Scholar
14. Drew T, Le-Hoa Vo M, Olwal A, Jacobson F, Seltzer SE, Wolfe JM. Scanners and drillers: characterizing expert visual search through volumetric images. J Vis 2013;13:pii: 3.10.1167/13.10.3Suche in Google Scholar PubMed PubMed Central
15. Renfrew DL, Franken Jr EA, Berbaum KS, Weigelt FH, Abu-Yousef MM. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992;183:145–50.10.1148/radiology.183.1.1549661Suche in Google Scholar PubMed
16. Pinto A, Acampora C, Pinto F, Kourdioukova E, Romano L, Verstraete K. Learning from diagnostic errors: a good way to improve education in radiology. Eur J Radiol 2011;78:372–6.10.1016/j.ejrad.2010.12.028Suche in Google Scholar PubMed
17. Donald JJ, Barnard SA. Common patterns in 558 diagnostic radiology errors. J Med Imaging Radiat Oncol 2012;56:173–8.10.1111/j.1754-9485.2012.02348.xSuche in Google Scholar PubMed
18. Donovan T, Litchfield D. Looking for cancer: expertise related differences in searching and decision making. Appl Cognit Psychol 2013;27:43–9.10.1002/acp.2869Suche in Google Scholar
19. Rubin GD, Roos JE, Tall M, Harrawood B, Bag S, Ly DL, et al. Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking. Radiology 2015;274:276–86.10.1148/radiol.14132918Suche in Google Scholar PubMed
20. Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investigat Radiol 1978;13:175–81.10.1097/00004424-197805000-00001Suche in Google Scholar PubMed
21. Pecaric M, Boutis K, Beckstead J, Pusic M. A big data and learning analytics approach to process-level feedback in cognitive simulations. Acad Med 2017;92:175–84.10.1097/ACM.0000000000001234Suche in Google Scholar PubMed
22. Ravesloot CJ, van der Schaaf MF, van Schaik JP, ten Cate OT, van der Gijp A, Mol CP, et al. Volumetric CT-images improve testing of radiological image interpretation skills. Eur J Radiol 2015;84:856–61.10.1016/j.ejrad.2014.12.015Suche in Google Scholar PubMed
23. Ravesloot CJ, van der Gijp A, van der Schaaf MF, Huige JC, Vincken KL, Mol CP, et al. Support for external validity of radiological anatomy tests using volumetric images. Acad Radiol 2015;22:640–5.10.1016/j.acra.2014.12.013Suche in Google Scholar PubMed
24. Morita J, Miwa K, Kitasaka T, Mori K, Suenaga Y, Iwano S, et al. Interactions of perceptual and conceptual processing: expertise in medical image diagnosis. Int J Hum Comp Stud 2008;66:370–90.10.1016/j.ijhcs.2007.11.004Suche in Google Scholar
25. Sadler DR. Beyond feedback: developing student capability in complex appraisal. Assess Eval High Educ 2010;35:535–50.10.1080/02602930903541015Suche in Google Scholar
26. Black P, William D. Assessment and classroom learning. Assess Educ Princ Pol Pract 1998;5:7–74.10.4135/9781446250808.n2Suche in Google Scholar
©2017 Walter de Gruyter GmbH, Berlin/Boston
Artikel in diesem Heft
- Frontmatter
- Editorial
- The imperative to address diagnostic safety
- Review
- Challenges and opportunities from the Agency for Healthcare Research and Quality (AHRQ) research summit on improving diagnosis: a proceedings review
- Opinion Paper
- Identifying and analyzing diagnostic paths: a new approach for studying diagnostic practices
- Original Articles
- Are health care provider organizations ready to tackle diagnostic error? A survey of Leapfrog-participating hospitals
- Medical laboratory associated errors: the 33-month experience of an on-line volunteer Canadian province wide error reporting system
- “Dr. Google” and his predecessors
- Identifying error types in visual diagnostic skill assessment
- Letters to the Editor
- Response to paper on probabilistic diagnosis
- There is no escape from using probabilities in diagnosis-making
- IgA plasmablastic large B-cell lymphoma
- Diagnostic accuracy for hybrid oncocytic/chromophobe renal cell tumors by exploiting an immunohistochemical and histochemical combined panel
Artikel in diesem Heft
- Frontmatter
- Editorial
- The imperative to address diagnostic safety
- Review
- Challenges and opportunities from the Agency for Healthcare Research and Quality (AHRQ) research summit on improving diagnosis: a proceedings review
- Opinion Paper
- Identifying and analyzing diagnostic paths: a new approach for studying diagnostic practices
- Original Articles
- Are health care provider organizations ready to tackle diagnostic error? A survey of Leapfrog-participating hospitals
- Medical laboratory associated errors: the 33-month experience of an on-line volunteer Canadian province wide error reporting system
- “Dr. Google” and his predecessors
- Identifying error types in visual diagnostic skill assessment
- Letters to the Editor
- Response to paper on probabilistic diagnosis
- There is no escape from using probabilities in diagnosis-making
- IgA plasmablastic large B-cell lymphoma
- Diagnostic accuracy for hybrid oncocytic/chromophobe renal cell tumors by exploiting an immunohistochemical and histochemical combined panel
