Impact of diagnostic checklists on the interpretation of normal and abnormal electrocardiograms

Introduction

In recent years, checklists have received increasing attention as a promising tool to reduce medical errors [1], [2], [3]. This started with the successful implementation of checklists in reducing hospital-acquired infections [4] and preventing errors during surgeries [5]. These checklists aimed to reduce clinician’s cognitive load and reliance on memory [6] by documenting the steps of a specific task (e.g., a surgical procedure). Following these successes, the use of checklists has also been advocated as a tool to reduce diagnostic errors [7], [8], [9], [10], [11], a long understudied type of medical errors [12] that occur when diagnoses are wrong, missed, or delayed [13]. Diagnostic errors are a large burden on patient safety and it is estimated that a majority of people will experience a diagnostic error during their lifetime [13, 14]. Therefore, developing successful interventions to reduce diagnostic errors is crucial [15].

Flaws in the cognitive processes underlying reasoning are seen as a primary cause of diagnostic errors [16], [17], [18], [19], [20], [21] and consequently, diagnostic checklists aim to reduce errors by supporting clinicians’ reasoning processes. These checklists can generally be divided into two types [22, 23]. The first type aim to have clinicians examine and improve their reasoning processes. These process checklists generally give broad instructions to carefully reconsider your diagnosis or to check your reasoning for cognitive biases (i.e., predispositions to think in a way that leads to systematic failures in judgement [24]) [22, 25, 26]. The second type includes content-specific checklists, which aim to compensate for possible knowledge deficits or mistakes [21, 27] by having clinicians examine the content of their reasoning. Content checklists can give possible diagnoses for certain symptoms [23, 26, 28] or ensure the clinician considers all relevant information for a diagnosis, as even those who were trained to follow the steps of a specific protocol will not always adhere to this protocol [29], [30], [31], [32], [33], [34], [35], [36], [37]. Furthermore, content checklists might have the potential to reduce clinicians’ cognitive load by facilitating information integration [7, 38].

Empirical evidence that checklists reduce diagnostic errors is scarce and inconsistent [10, 22, 23]. Reviews on error interventions generally report small to medium improvements in diagnostic accuracy [6, 22, 39], but the practical significance of this improvement is unclear. Overall, existing studies hint that checklist effectiveness might depend on the type of checklist, the relative difficulty of the clinical cases that have to be diagnosed, and the participants’ level of expertise. For example, process checklists [22] were shown to be ineffective in increasing diagnostic accuracy [28, 33], with exception of one study by Sibbald et al. [31] that showed an improvement. Content checklists often led to small reductions in diagnostic errors [29], [30], [31, 40] – except in one study where no benefit was seen [33]. Furthermore, checklists were more effective in improving diagnostic accuracy for novices than for experts in two studies examining ECG interpretation and dermatological images, respectively [30, 36]. Finally, in some studies checklists only benefited the diagnosis of complex clinical cases [28, 31, 40]. Unfortunately, the factors impacting checklist effectiveness are still poorly understood and more research is necessary to determine if, and when, checklists are effective [10, 22, 23].

Our understanding of checklist effectiveness is especially limited for settings such as general practice. For general practitioners (GPs), it is more important to recognize normal cases and to exclude certain diagnoses than it is to arrive at the precise correct diagnosis. Existing studies, however, mostly test checklists on abnormal cases that were designed to be complex. This approach is intended to create a situation where the potential for making and subsequently correcting mistakes is high, so that benefits from an intervention can be observed [22]. Furthermore, GPs are also expected to correctly manage patients, even before knowing the exact diagnosis. Existing studies primarily measure diagnostic accuracy, leaving out other such aspects of diagnostic performance. A task for which these issues are relevant is the interpretation of ECGs. At least one-third of ECGs seen in Dutch general practice are normal [41] and the most important decision GPs make is on whether or not to refer the patient to a specialist. In the Netherlands, ECG interpretation has recently shifted more and more from secondary to primary care, even though most GPs are not specialized in this task and have had limited training [41]. GP education now often implements checklists to teach this skill [34]. It is therefore crucial to understand how checklist use will impact ECG interpretation, as checklists could lead to overtesting and overdiagnosis, or unnecessary consumption of resources such as time and personnel [22].

In this randomized experiment, we examined the impact of checklist use on the performance of GP residents when interpreting normal, simple abnormal, and complex abnormal ECGs. Performance was measured as residents’ diagnostic accuracy, confidence, patient management, and time to diagnose. Additionally, residents’ confidence-accuracy calibration was assessed. We studied two types of checklists – a debiasing checklist focused on detecting and correcting cognitive biases [28, 33, 38] and a content checklist focused on ensuring all ECG elements important for interpretation are checked [34]. We expected that neither checklist would benefit performance for normal cases. For simple and complex abnormal cases, we expected that only the content checklist would be beneficial. Furthermore, we expected that residents’ confidence-accuracy calibration would increase for the content checklist, but decrease for the debiasing checklist.

Materials and methods

Results

In total, 55 first year GP residents participated in at least one phase. Five residents did not give permission to use their data for research and an additional eight only completed one phase or did not interpret all ECGs. 42 residents completed both phases. 21 residents were allocated to the debiasing checklist and 21 to the content-specific checklist. Participant demographics are shown in Table 3 and Supplementary Material 2.

Residents’ prior experience (specifically, the number of ECGs diagnosed) was used to test whether experience moderated the dependent variables (Table 4). Only confidence systematically varied with experience and post-hoc tests indicated that only residents who diagnosed fewer than 10 ECGs differed from the other experience groups. Therefore, these three residents were excluded to correct for experience as a covariate, leaving 18 participants in the content-specific condition. Age, sex, and time in residency did not moderate diagnostic performance.

Diagnostic performance

Phase 1 vs. phase 2

When interpreting ECGs in phase 2 (M=0.63, SD=0.2) compared to phase 1 (M=0.55, SD=0.2), there was a trend for overall accuracy to improve (Z=−1.81, p=0.070, g=0.25). Checklist use did not affect residents’ confidence (phase 1: M=5.5, SD=1.7), (phase 2: M=5.5, SD=1.9, Z=−0.23, p=0.817) and patient management (phase 1: M=0.61, SD=0.2, phase 2: M=0.63, SD=0.2, Z=−0.92, p=0.358) in phase 1 compared to phase 2. Lastly, residents took less time to interpret all ECGs in phase 2 (phase 1: M=189, SD=80, phase 2: M=144, SD=76, Z=−3.10, p=0.002, g=0.54). Resident’s performance on each outcome measure is summarized in Table 4.

Checklist type

Using either the debiasing or content-specific checklist did not differentially affect accuracy (U=158, p=0.707), confidence (U=134, p=0.270), patient management (U=137, p=0.311), or time spent to diagnose (U=162, p=0.821).

ECG type

ECG type did not affect checklist use for accuracy (χ²(2, 43)=2.54, p=0.281), confidence (χ²(2, 43)=2.74, p=0.254), patient management (χ²(2, 43)=2.10, p=0.350) and time to diagnose (χ²(2, 43)=1.16, p=0.559). For patient management, residents descriptively scored the best for complex abnormal cases, where the patient should be referred to the cardiologist. For normal and simple abnormal cases, more than 90% of the incorrect answers constituted referral to the cardiologist.

Confidence-accuracy calibration

In both phases, confidence increased when accuracy increased (phase 1: r_s=0.42, p=0.004; phase 2: r_s=0.67, p<0.001). Moreover, residents’ confidence-accuracy calibration was lower when they interpreted ECGs without specific instructions (R²=0.14, Figure 1) compared to when they used a checklist (R²=0.40, Figure 2), although calibration remained moderate. Further analysis showed that their absolute accuracy did not differ between phases (Z=−0.59, p=0.554). Bias showed a trend to decrease, indicating that residents became less overconfident when using a checklist (Z=−3.10, p=0.055). Residents improved using either checklist compared to interpretation without a checklist, but seemed to benefit more from using the debiasing checklist (R²=0.59) than from the content-specific checklist (R²=0.32).

Figure 1:

Scatterplot of residents’ confidence-accuracy calibration in phase 1, R²=0.14.

Figure 2:

Scatterplot of residents’ confidence-accuracy calibration in phase 2, R²=0.40.

Discussion

This study examined the impact of checklist use on the interpretation of normal, simple abnormal, and complex abnormal ECGs. There was a trend for improvement in residents’ accuracy when they used a checklist, whereas their confidence did not change. This resulted in an overall improved confidence-accuracy calibration: participants were less overconfident after using a checklist compared to when they first interpreted the ECGs. Furthermore, residents’ patient management was very conservative as they consistently referred patients to the cardiologist. This was not affected by checklist use. Finally, residents took less time to interpret ECGs in phase 2. Contrary to our expectations, these findings were similar for all ECG types and both the debiasing checklist and the content-specific checklist, although the debiasing checklist seemed to improve to residents’ calibration the most.

For our interpretations regarding diagnostic accuracy, we should consider the possibility of a learning effect on diagnostic performance. There was no independent control group and therefore, residents saw the ECGs twice. Furthermore, the effects were similar across all ECGs and both checklist types, which indicates that the trend for improvement in accuracy and the decrease in time to diagnose are likely due to a small learning effect and do not fully reflect the effects of checklist use. These findings contradict previous studies that found increases in accuracy when a checklist was used [23, 28], [29], [30], [31], [32], specifically for the content-specific checklist. In most of these studies, participants also examined cases once before verifying their diagnosis using a checklist, although this verification took place immediately after the initial diagnosis [28], [29], [30, 32]. Similar studies often did not find an improvement when using a debiasing checklist, in line with our current findings [23, 28, 33]. Despite this limitation, our study design is reflective of how checklists would be used in practice: to verify a working diagnosis or to check someone’s reasoning process. Alternatively, the current lack of improvement in diagnostic accuracy could be explained by the use of singular reasoning approaches. Our participants were asked only to reason analytically, following either a feature list (given by the content-specific checklist) or a debiasing approach. This contrasts work by Eva et al. [46] and Ark et al. [47, 48], who showed in several experiments with naïve students that combining analytical and non-analytical approaches is more effective than applying singular reasoning approaches. Future studies might benefit from not only comparing singular methods but also combining reasoning strategies in error intervention studies.

Interestingly, the trend for improved accuracy did not coincide with an increase in confidence. One would expect that if a previously incorrect diagnosis was changed or if a previously correct diagnosis was confirmed with the help of a checklist, this would boost confidence, especially if residents were simply re-examining a case. Our data showed that in each phase, if residents were more accurate, they were also more confident. However, this did not translate to an increase in confidence between phases, indicating that residents became less overconfident and potentially that their insight in their own skills improved. Despite the increased confidence, the majority of residents still chose to refer the patient to a cardiologist. This is likely related to the relative inexperience of our participants. Future research should also measure participants’ referral behavior, as overreferral leads to large economic costs.

The increase in residents’ confidence might have been extra pronounced for the debiasing checklist because GP residents in the Netherlands are already taught to interpret ECGs using the content-specific checklist, whereas the debiasing checklist was completely new to them. The fact that the GP residents were already familiar with the content-specific checklist, and because novices often use a more analytical step-by-step approach than more experienced clinicians [49], might also have diluted potential effects of the content-specific checklist. Although our study showed no immediate benefits of improved calibration, there could be value in using checklists to reduce overconfidence. Overconfidence has previously been indicated as a cause of diagnostic errors [50] and fostering proper calibration could improve residents’ diagnostic process and potentially improve their diagnostic performance in the long term. Future research should confirm whether checklists can be used to reduce overconfidence and what the long-term effects of checklist use are.

This study had several strengths and limitations. Strengths include that this was a randomized experiment that used ECGs of an appropriate level for first year GP residents. Furthermore, the ECGs were verified teaching materials from real patients with a confirmed diagnosis. A final strength was that participants performed the experiment online, from their home, and participated in multiple experiments and lectures. This greatly reduced the chances of participants discussing the study’s ECGs amongst themselves.

The study is limited because of the design without an independent control group in which participants interpreted each ECG twice, which left the possibility for a learning effect to influence our results. This primarily influenced the interpretation of diagnostic accuracy and time on task, but even with a possible learning effect participants did not improve on immediate accuracy. Furthermore, we chose to have participants diagnose the same ECGs twice so we could directly compare changes in confidence, calibration, and patient management. This allowed for reliable assessment of residents’ confidence, as there was no room for between-case variability. The remaining variables could be inflated by a possible learning effect and should be interpreted with caution. A second limitation is the relative inexperience of our participants. Considering that most residents had interpreted few ECGs during their studies, suddenly seeing 9 ECGs in one day was a significant increase in practice. This might have contributed to the trend for improved accuracy. Lastly, a limitation is that the overall sample size and the sample size for the separate checklist analyses were relatively small and might be underpowered, meaning these results should be interpreted with caution. The study might, additionally, have benefited from including more than 9 EKGs. The a priori power calculation was performed assuming 9 measurements but the true effect might have been smaller than the medium effect size we estimated. Future research should examine the impact of checklist use on accuracy and calibration in more experienced GP residents, as the issue remains crucial to GPs, with a control group and a larger sample of EKGs.

In summary, checklist use did not differentially affect GP residents’ diagnostic process for normal cases compared to simple abnormal or complex abnormal cases. Surprising was that residents’ confidence did not increase over repeated viewing of the ECGs and that checklists improved residents’ confidence-accuracy calibration, which translated in reduced overconfidence. Although more research is needed to evaluate how checklists impact residents’ confidence in the long term, checklists could be promising. Reducing overconfidence, an important cause of diagnostic errors, could improve residents’ insight into their own skill level, and in the long term has the potential to improve their diagnostic performance.