Pediatric faculty knowledge and comfort discussing diagnostic errors: a pilot survey to understand barriers to an educational program

Introduction

The 2015 National Academy of Medicine’s report, Improving Diagnosis in Healthcare [1], called attention to diagnostic errors and their contribution to patient harm. Among the major themes highlighted in the report is the need to improve training in clinical reasoning among diagnosticians. Theoretically, diagnosticians who are aware of potential flaws in human decision-making might be better prepared to recognize and avoid pitfalls in their diagnostic reasoning, thereby reducing patient harm [2].

Diagnostic reasoning can be understood using the dual process theory of decision-making, a general model describing how humans make decisions [3]. Dual process theory suggests two modes of reasoning – System 1 and System 2 – are used to make decisions. Deliberate, conscious, time-consuming and effortful judgments characterize System 2 reasoning. System 1 reasoning, in contrast, is primarily intuitive, subconscious, fast and relatively effortless [4]. System 1 reasoning relies on heuristics (often called cognitive biases). These mental shortcuts allow experienced clinicians to arrive at decisions quickly with a reasonable degree of certainty in familiar situations. Using heuristics to make decisions is neither avoidable nor an inherently inappropriate method for clinical reasoning [5]. While experienced clinicians may recognize their reliance on System 1 reasoning in clinical practice, the theories underlying decision-making are not formally taught in most medical school or residency curricula [6].

In clinical vignettes, cognitive errors contribute to overall diagnostic errors in as many as 37–77% of cases [7] leading some authors to suggest that understanding the etiology of these errors is paramount to reducing them [2]. Others have argued that knowing about heuristics may not necessarily allow clinicians to avoid diagnostic errors [8]. Consequently, some authors have suggested novel interventions (e.g. diagnostic checklists, differential diagnosis generators) to mitigate the risks posed by heuristic-driven reasoning [9], [10]. However, the deliberate use of such tools requires awareness of situations at high risk for cognitive errors. It is not clear that diagnosticians currently possess this awareness, but some understanding of the role of heuristics in diagnostic reasoning may be helpful in reducing harm from diagnostic errors.

Discussion of diagnostic errors during morbidity and mortality (M&M) conferences represents an approach to reducing harm through structured analysis of the errors including identification of cognitive factors [11]. Katz and Detsky suggest that M&M is an ideal venue to introduce discussion of cognitive errors [12]. Fruitful discussions of diagnostic error require an environment that is open to honest discussion focused on patient safety rather than harmful criticism of individuals [13]. Yet, discussions of harm arising from flawed diagnostic reasoning are uncomfortable for clinicians [14], [15]. For example, loss of reputation and fear of incurring blame are two significant components that influence clinicians’ comfort discussing their own errors with colleagues [16]. Recently, the promotion of both psychological safety (including a non-blaming atmosphere) and transparency when reporting and reviewing medical errors in public venues has received significant attention [17]. While some institutions have deliberately revamped their M&M conferences to promote psychological safety [18], there remains an undercurrent of blame culture in medicine [19].

We aimed to better understand two obstacles to improving training in clinical reasoning for clinicians. First, little is known about how familiar clinicians are with heuristics operating during the diagnostic process. Second, it is not clear whether current culture promotes psychological safety when discussing diagnostic errors in public venues such as M&M. In anticipation of developing a curriculum on diagnostic errors at our institution, we wanted to better characterize these obstacles among a subset of pediatric faculty. We hypothesized that participants’ awareness of heuristic definitions would positively correlate with an ability to recognize heuristics operating in clinical vignettes. We also sought to describe participants’ comfort discussing their own diagnostic errors in public venues and the perceived barriers to such discussions.

Materials and methods

Results

There were 229 eligible faculty from the three divisions surveyed. Overall, 88 (38.4%) faculty members accessed the survey and provided demographic information. Eight surveys were excluded from all further analyses because participants only provided demographic information; an additional 10 were excluded because they were only partially complete. There were no significant differences with respect to age, gender, years in practice or provider type between those that provided complete information compared to those with truncated responses. The overall response rate was 30.6% [22]. The response rate was significantly lower for APPs (9.0%) compared to physicians (44.3%) (p<0.001). Table 1 shows the demographics of participants. Only 15.9% endorsed receiving formal training in clinical reasoning beyond training received in medical school or, for APPs, during graduate training. All participants endorsing formal training were less than 50 years old. Most participants (84.3%) reported that they had been involved in a diagnostic error resulting in patient harm during their career.

Knowledge of heuristic definitions

No participant answered all five heuristic definition questions correctly; the mean number of correctly selected heuristic definitions was 1.60/5 (SD=1.13) (Figure 1). Reporting formal training in clinical reasoning did not significantly increase the likelihood of correctly selecting the heuristic term for any definition (p-values range from p=0.05 to 1.00). Age, gender, years in practice and provider type were not significantly associated with answer accuracy.

Figure 1:

Knowledge of heuristic definitions.

Solid bars indicate the proportion of participants answering correctly. Stippled bars indicate the proportion of participants answering incorrectly. Hashed bars indicate the proportion of participants responding “I don’t know”.

Identifying heuristics in vignettes

Only two participants selected the correct heuristic for all four vignettes; the mean number of correctly selected heuristics among the vignettes was 1.01/4 (SD=1.06) (Figure 2). Reporting formal training in clinical reasoning did not significantly increase the likelihood of correctly identifying the heuristic for any vignette (p=0.72–1.00). Age, gender, years in practice and provider type were not significantly associated with answer accuracy for any of the vignettes.

Figure 2:

Identifying heuristics in clinical vignettes.

Solid bars indicate the proportion of participants answering correctly. Stippled bars indicate the proportion of participants answering incorrectly. Hashed bars indicate the proportion of participants responding “I don’t know”.

There was a low but significant correlation between scores on the definition and vignette portions of the survey (Spearman’s ρ=0.27, p=0.02).

Discussion

Pediatric faculty in this study demonstrated limited ability to identify heuristic definitions. Similarly, they were frequently unable to correctly name heuristics employed in clinical vignettes. We did find evidence to support our hypothesis that knowledge of heuristic definitions correlates with being able to identify heuristics in clinical vignettes, although the correlation is low. Pediatric faculty expressed significant discomfort discussing their own errors publicly in venues such as M&M despite being generally comfortable discussing diagnostic errors in private and acknowledging diagnostic uncertainty to themselves and their peers. This discomfort appears to arise from concerns about negative peer perceptions regarding clinical reputation and the diagnosticians’ clinical reasoning skills and knowledge.

Clinicians are central to efforts to reduce patient harm resulting from diagnostic errors. Success in these efforts may be dependent on two related conditions: (1) the diagnostician’s ability to recognize how diagnostic reasoning might be subconsciously influenced through reliance on heuristics; and (2) the willingness to discuss cognitive errors in public in order to identify solutions to avoid them in the future.

Pediatricians can appreciate their own diagnostic errors in hindsight. Singh and colleagues demonstrated that 54% of surveyed pediatricians admit to making 1–2 diagnostic errors per month, and 45% reported making 1–2 diagnostic errors per year that result in patient harm [23]. However, reducing patient harm due to diagnostic error requires that diagnosticians recognize and avoid heuristic failures prior to committing them.

There are few studies investigating strategies to avoid, or mitigate the effects of, heuristics and results are mixed [24]. Most studies have included trainees and have focused on clinical vignettes rather than real patient encounters. While some studies have demonstrated small but significant improvements in diagnostic accuracy when residents were instructed to rely on slow deliberate System 2 reasoning (through structured reflection) [25], [26], one study did not demonstrate a benefit with this approach [27]. Sherbino and colleagues provided instruction on two common heuristics (search satisfying and availability) to senior medical students followed by training on cognitive forcing strategies to avoid heuristic errors. Despite immediate testing after the educational intervention, fewer than 50% of students successfully “debiased” themselves [28]. Little work has been done to evaluate practicing clinicians’ familiarity with and ability to recognize heuristics. We conducted this study to begin to address that gap.

Participants in this study were unable to correctly name several defined heuristics or identify heuristics operating in several clinical vignettes. We found that formal training in diagnostic reasoning did not improve participants’ ability to identify heuristics. We did not collect information regarding the timing or content of this training so it is difficult to know whether more robust or more proximate education would have yielded improved accuracy. However, our results align with another study in which experts in diagnostic error were asked to identify heuristics (cognitive biases) perceived to be operating in eight clinical vignettes [29]. Participants demonstrated no agreement when naming heuristics represented by the vignettes (κ=0.0–0.6).

The goal of introducing diagnosticians to concepts including heuristics and dual process theory is not to teach new vocabulary. Rather, improving diagnosticians’ awareness of these concepts aims to improve recognition of the risk of diagnostic errors arising from System 1 reasoning. Nemes-Walsh and Lee evaluated the effect of an educational intervention introducing topics including dual process theory and cognitive bias pertaining to diagnostic errors [30]. This study demonstrated a slight improvement in responses on the National Patient Safety Foundation Reducing Diagnostic Errors: Strategies for Solutions Quiz and increased willingness to take a “diagnostic timeout” following the intervention. There was no indication, however, that fewer diagnostic errors occurred after the training. In a more rigorous study, residents completing a yearlong curriculum on cognitive biases and diagnostic errors demonstrated improvement in correctly naming heuristics [31]. Additionally, they were frequently able to identify heuristics in standardized clinical vignettes and describe potential debiasing strategies. The modest yet significant association between correctly naming defined heuristics and identifying heuristics in clinical vignettes in the present study aligns with those findings. While mastery of vocabulary alone is unlikely to reduce diagnostic errors arising from use of heuristics, conceptual familiarity with this type of reasoning may improve recognition of high-risk clinical scenarios and warrants further study to determine whether these interventions reduce diagnostic errors in clinical encounters.

Diagnosticians in this study were generally comfortable speaking privately to peers about their diagnostic errors. That comfort did not persist when participants considered discussing diagnostic errors in public settings. This discomfort appears to stem from a general concern over how diagnosticians’ knowledge, decision-making and reputation are judged rather than the error simply being made public. Diagnosticians also reported a poorer self-perception of themselves as clinicians as a barrier. These findings suggest significant cultural changes are necessary to advance patient safety. We did not inquire about the fear of malpractice litigation which may also be a substantial barrier to openly discussing diagnostic errors.

We were encouraged by the finding that more experienced clinicians were significantly more comfortable with public discussions of their errors than their less experienced peers. These data imply that senior clinicians may serve as influential role models in the cultural shift required to share experiences publicly in efforts to reduce harm from diagnostic errors.

Interpretation of our study must be considered in light of some limitations. We experienced a low response rate from APPs and, as a pilot study, sampled a relatively narrow segment of our hospital’s pediatric faculty. Our findings may not be representative of faculty at other hospitals or providers in non-academic and outpatient settings. We studied a small number of heuristics. Croskerry lists over 30 heuristics encountered in clinical reasoning but data describing which heuristics are most common or significant in diagnostic reasoning are limited [20]. Thus, we chose heuristics that our own clinical experience indicates are employed relatively frequently but acknowledge that they are not proven to be the most common culprits in diagnostic errors. We also acknowledge that our survey attempted to assess familiarity with heuristic concepts using definitions as a surrogate. It would be more useful to assess familiarity using more qualitative methods discussing vignettes with diagnosticians; this is an opportunity for future study.

Conclusions

Clinicians in our study were unable to match heuristic definitions or clinical vignettes exemplifying heuristics to the proper heuristic term. Attempts to foster a culture supportive of discussing diagnostic errors will require both improved awareness of the role heuristics play in the diagnostic process and better methods to promote psychological safety when sharing errors.