Game-based learning to improve diagnostic accuracy: a pilot randomized-controlled trial

Participants

We enrolled medical students, internal medicine residents and practicing clinicians at two hospitals. Potential participants were contacted through a group email to medical students, another email to internal medicine residents and emails and direct recruitment of practicing clinicians (physicians and nurse practitioners (NP)).

Participants were randomized to intervention (GBL; 30 participants), intervention with cases to remind participants of game-based learning (30 participants) or control (30 participants). In total, participants were assigned to three groups in a 1:1:1 fashion. These groups were GBL intervention, GBL intervention plus emails, and control (see Figure 1, study overview). Randomization was stratified by type of participant (student, resident, clinician). After enrollment, participants were given a link to a Qualtrics survey that contained links to individual interventions and follow up questionnaires. We provided gift cards at three points of engagement: a $100 gift card after completing the initial assessment and $50 gift cards after the one- and three-month post-intervention assessments.

Figure 1:

Overview of participant assignment and frequency of assessments during the study.

Intervention

The game was developed iteratively with clinician feedback and is publicly available (https://bird.testingwisely.com/). It requires users to estimate probability of diagnosis based on 10 simplified case presentations. Playing one game takes 5 min or less. Immediate feedback on each question is delivered using natural frequency methods with a score, tips, and comparisons to other users. The game has a short tutorial and different play options.

The entire GBL intervention instructed participants to (1) visit the testingwisely.com website and review materials, (2) watch a 4 min video explaining natural frequencies to determine post-test probability (Episode 4, https://www.testingwisely.com/educational-videos); (3) play the game tutorial and two games. The initial GBL intervention was expected to take 45 min or less. Participants attested to completing each step of the intervention.

After assessing the primary outcome, half of the GBL intervention group was randomized to receive weekly emails containing a clinical case emphasizing points from GBL.

Control exposure

The control condition required a similar amount of time, and standard materials taught diagnosis using traditional calculations and 2×2 table methods. Materials were provided in the control Qualtrics survey and included reading the UpToDate online medical textbook testing chapter [17] and two highly-viewed YouTube videos on diagnostic testing (https://youtu.be/Z5TtopYX1Gc and https://youtu.be/dHj7ygeqelw). Participants attested to completing each step of the control.

Measures

Statistical methods, analyses and expected outcomes

The primary outcome compared the continuous accuracy score in GBL intervention vs. control subjects. The distribution of this score was compared between the GBL intervention or control at the initial assessment, and all three groups at the one-month and three-month assessments. Point estimates and p-values were based on a longitudinal regression model fit by maximum likelihood.

Sample size considerations

This was a pilot study with the goal of making estimates around effect size, optimal intervention and feasibility of enrolling different groups.

This study was deemed exempt by the Institutional Review Board (IRB) of note. Because this was a pilot study, it was not registered in clinicaltrials.gov consistent with guidance for pilot studies [18].

Demographics

Among the 90 participants, 30 were medical students, 30 residents and 30 practicing clinicians. Demographics are provided in Table 1.

Follow-up

All participants completed the initial evaluation. At one month, 71/90 completed follow-up (79 %) and at three months, 66/90 completed follow-up (73 %). Notably, follow-up was worse in those receiving intervention plus emails than intervention alone (19/30 vs. 24/30), likely as a participant noted, because of routinely ignoring study emails in the group that received additional emails.

Primary outcome

The game-based learning intervention improved mean diagnostic accuracy; mean scores were 59.4 (of 100) in those who underwent game-based learning vs. 37.6 in controls. The estimated initial difference in score based on the longitudinal regression model was found to be 21.8 points (95 % confidence interval 9.8 to 33.9 points, p=0.0005). This is based on a longitudinal regression model accounting for the correlation between repeated measures on the same participant (see Figure 2).

Figure 2:

Impact of game-based learning with or without follow-up emails (intervention) vs. a standard control education immediately after game-based learning (primary outcome) and at one- and three-month follow-up. Horizontal bars represent group means.

Secondary outcomes

At one-month and three-month follow up, there were three groups: GBL intervention, GBL intervention plus emails and control. At one-month follow up, GBL (mean score 59.2) and GBL plus emails (mean score 54.2) performed better than control (33.9), p=0.024. There was no difference between GBL with or without emails (p=0.99). At three-month follow up, game-based learning (mean score 56.2) and GBL plus emails (mean score 42.9) were not significantly different than control (35.1), p=0.077. There was no significant difference between GBL with and GBL without emails (p=0.091) (see Figure 2).

Medical students statistically performed no differently than residents or clinicians in practice, although the effect of GBL appeared potentially less in practicing clinicians than other groups. Initial performance after GBL intervention (mean 68.0 residents, vs. 62.1 students, vs. 48.2 practicing clinicians) (p=0.067).

Experience of game-based learning

Participants were asked if they would recommend GBL or use GBL again (possible answers yes/maybe/no) and their degree of support for statements about GBL (possible answers not at all/slightly/moderately/very/extremely). Participants were generally positive about GBL being enjoyable and improving test interpretation and most would recommend or use again (see Figure 3).

Figure 3:

Participants perceptions of game-based learning (GBL). Recommending or using GBL could be answered yes, maybe or no. Opinion of enjoyability and improving test interpretation could be extremely, very, moderately, slightly, not at all/no.

Qualitative responses to game-based learning

Intervention group participants were asked to give optional feedback on GBL. All participants who completed the intervention provided feedback (60/60). Themes and examples are described in Table 2.

Limitations

Limitations to this pilot study include that the sample size was moderate and within a single medical system and may not be generalizable. The outcome scale of diagnostic accuracy has not been validated. Although GBL greatly improved mean accuracy score, there was variability in clinician scores in all groups. We do not know how lack of follow-up may have impacted results at one and three months. Finally, participants were reimbursed for completion of the study and attention may not be replicated with standard voluntary participation.