Effects of a focused training on first-year osteopathic medical students’ ability to incorporate point-of-care ultrasound in assessment of the anterior knee

Participants

This cross-sectional, single-center design study was conducted from March through May 2018 at A.T. Still University’s School of Osteopathic Medicine in Mesa, Arizona and was considered exempt by the A.T. Still University-Arizona Institutional Review Board. First-year osteopathic medical students, with little to no point-of-care ultrasound experience, were recruited to participate on a voluntary basis. Each student provided informed and written consent before participation. All learning activities occurred in the osteopathic manipulative medicine (OMM) laboratory, and additional materials were posted online. All students were offered the same educational experiences whether they participated in the study or not, and their grades were unaffected. Various steps related to training and assessment for the current study are illustrated in Figure 1. The overall duration of the study was approximately 10 weeks from students receiving the online study materials to the follow-up written test.

Figure 1:

Study flowchart for effectiveness of a focused training on first-year osteopathic medical students’ ability to incorporate point-of-care ultrasound in assessment of the anterior knee. ^aOrder of test questions modified from previous test.

Pretraining: written pretest and pretraining questionnaire

One week before the focused ultrasound training, students were given the pretraining questionnaire (Supplementary Material). The eight-item questionnaire asked for information about previous ultrasound experience and assessed self-reported confidence levels regarding their ability to identify normal anatomy and common pathologies of the anterior knee. Confidence levels were measured on a 5-point Likert scale, where 1 was strongly agree and 5 was strongly disagree.

Next, students completed a 12-question, multiple-choice pretest to assess baseline knowledge, including ultrasound findings for joint effusion, prepatellar bursitis, cellulitis, and the normal anterior knee. Questions also focused on ultrasound knobs, probes, and orientation, as well as echogenicity, patient diagnosis, and next best step in management (e.g., antibiotics, arthrocentesis, OMT and/or rest, ice, compression) based on sonographic images and clinical vignettes for a patient with a painful swollen knee. The paper pretest and questionnaire were administered during OMM laboratory and took approximately 15 min to complete.

Focused ultrasound training

The next day, a 20 min instructional video was posted online for students to view. They were informed about the video after completing the pretest and again by email once it was posted. The instructional video was developed specifically for the current study and consisted of the following: a brief discussion on ultrasound fundamentals, transducers, scanning technique, and echogenicity; normal sonographic anatomy of the anterior knee; and examples of the three primary pathologies for joint effusion, prepatellar bursitis, and cellulitis of the knee. The presentation also briefly covered clinical presentation, physical examination findings, and appropriate next-step management of each condition. Students were asked to view the video before the following week’s OMM laboratory, which also included the study’s hands-on evaluation.

Faculty and sonographers involved with the in-person focused training and hands-on evaluation also received training for the current study. They were given access to a PowerPoint (Microsoft Corporation, Redmond, WA) and 5 min video created for review of ultrasound techniques of the anterior knee and specific evaluation criteria to ensure consistency during the hands-on student assessment. Proctors included: one ultrasound fellowship-trained emergency physician, one physician and two registered diagnostic medical sonographers credentialed by the American Registry for Diagnostic Medical Sonography, one PGY-4 radiology resident, one biomedical engineer, and two predoctoral OMM fellows.

For the in-person focused ultrasound training, students were divided into their usual morning and afternoon OMM laboratory groups (approximately 50 students each). A 20 min didactic session was presented and included an overview of the online materials and learning objectives (Figure 2). Any questions that the students had were answered in real time. After the didactic presentation, groups of five or six students practiced scanning the anterior knee on each other for approximately 40 min. Students located and identified anatomical landmarks and tissue types while describing features of echogenicity on ultrasound. Each group was supervised by an instructor.

Figure 2:

Learning objectives for the focused training session in the current study.

Calibration of standardized patients

Before the hands-on assessment, each standardized patient was examined and scanned by the study’s primary investigator (CW). Height, weight, body mass index, and knee circumference directly over the patella and 2 cm proximal were recorded for each standardized patient. The calibration was intended to verify that students could identify the required anatomical landmarks and determine the absence or presence of joint effusion, prepatellar bursitis, and cellulitis. Results of the calibration were utilized as the gold standard for the hands-on assessment. Although they represented a variety of shapes and sizes, none of the standardized patients had clinically relevant pathology of the knee.

Posttraining: hands-on assessment, written posttest, and posttraining questionnaire

For the hands-on assessment, each student was evaluated at one of six similar testing stations and given 12 min to complete the process. While the student interacted with the standardized patient, a trained instructor asked scripted questions and documented student responses (Figure 3). Students utilized either a GE LOGIQ e (GE Ultrasound, Milwaukee, WI), Sonosite Edge II (FUJIFILM SonoSite, Inc., Bothell, WA), or Sonosite M-Turbo ultrasound machine with a high-frequency (12–5 MHz) linear probe for scanning. To begin, each student was asked to identify several anatomical landmarks of the anterior knee: skin/subcutaneous tissues, muscle, quadriceps tendon, patella, femur, suprapatellar recess, distal patellar tendon, and tibial plateau.

Figure 3:

Combined script and checklist utilized by proctors in the current study during the hands-on assessment. Tx, treatment; WNL, within normal limits.

Next, and unique to this study, students were instructed to “place the probe in longitudinal view, begin scanning through the knee, proximal to distal. Now stop and hold the probe in place” (Figure 4). A digital tablet with a prerecorded sonographic video clip matching the same anatomical location of the student’s probe placement on the standardized patient was then superimposed in front of the ultrasound screen. Depicting either joint effusion, prepatellar bursitis, cellulitis, or the normal knee, students were then asked to combine their knowledge of probe placement, anatomy, and sonographic findings from the superimposed video clip to make a point-of-care diagnosis (Figure 5).

Figure 4:

Point-of-care ultrasound of the anterior knee. Students described the anatomy seen on ultrasound as they scanned the standardized patient’s knee.

Figure 5:

A novel approach to providing reliable sonographic pathology when scanning a standardized patient. Once students have the correct probe placement and image in view, a digital tablet with the same anatomic view plus one of our three focused pathologies is manually superimposed over the ultrasound screen. Students were asked to consider their probe placement, underlying anatomy, and the image on the “new” screen to make a diagnosis.

Immediately after the hands-on assessment, students spent approximately 15 min completing the written posttest and posttraining questionnaire. The posttest had the same 12 questions as the pretest, but questions were presented in a different order to limit memorization of answers. The posttraining questionnaire repeated four questions from the pretraining questionnaire and included an additional question on whether students felt the ultrasound training would change how they diagnosed and managed a patient with a swollen, painful knee in future practice (Supplementary Material).

Learning retention: follow-up written test

Nine weeks later, students were asked to complete the follow-up written test, which was identical to the posttest. Like previous parts of the study, completion of the follow-up written test was voluntary. It was administered during OMM laboratory and took approximately 10 min to complete.

Data analysis

Identification of the three common pathologies—joint effusion, prepatellar bursitis, and cellulitis—were reported as proportions for the written pretest to posttest and for the pretest to 9-week follow-up written test. These data were compared utilizing the Fisher’s exact test. An independent sample t test was utilized to compare questionnaire results. A preliminary analysis indicated that the data met normality assumption requirements. Data were summarized utilizing frequency and percentage and mean and standard deviation (SD). The questionnaire and written test responses were anonymous and pooled for analysis. Paired tests were not utilized because results could not be matched by student. Instead, we assumed that the groups were independent for analysis. A p value of <0.05 was considered statistically significant. All analyses were conducted utilizing Stata 15 (College Station, TX).

Hands-on assessment

Overall, 78.3 % (595 correct of 760 aggregated responses) of the time students successfully identified sonographic landmarks of the anterior knee. Specifically, they correctly identified the following structures: skin (93.7 %, 89/95), muscle (72.6 %, 69/95), quadriceps tendon (90.5 %, 85/95), patella (94.7 %, 90/95), femur (89.5 %, 85/95), suprapatellar recess (73.7 %, 70/95), distal patellar tendon (89.5 %, 85/95), and tibial plateau (23.2 %, 22/95).

When asked to diagnose a prerecorded ultrasound image while scanning the corresponding location on the standardized patient’s knee, 71.4 % (20/28) of students accurately identified joint effusion, 60.9 % (14/23) correctly diagnosed prepatellar bursitis, and 93.3 % (28/30) recognized cellulitis.

Written tests

On the written tests, students accurately identified joint effusion (22.8 % [23/101] pretest, 65.3 % [62/95] posttest, 33.3 % [28/84] follow-up test), prepatellar bursitis (14.9 % [15/101] pretest, 47.4 % [44/95] posttest, 36.9 % [31/84] follow-up test), and cellulitis (38.6 % [39/101] pretest, 90.5 % [86/95] posttest, 73.8 % [62/84] follow-up test) (Figure 6). Differences were found between pretest and posttest for student identification of the three pathologies (all p<0.001) and between pretest and 9-week follow-up test for identification of prepatellar bursitis and cellulitis (both p≤0.001).

Figure 6:

The percent of correct responses of first-year osteopathic medical students on written tests for identification of joint effusion, prepatellar bursitis, and cellulitis from ultrasound images of the anterior knee.

Higher-order test questions pertaining to the treatment of these conditions utilized sonographic images and clinical vignette to determine the next best steps in management; students identified correct treatment for the joint effusion case (38.6 % [39/101] pretest, 48.4 % [46/95] posttest, 44.0 % [37/84] follow-up test), cellulitis case (17.8 % [18/101] pretest, 58.9 % [56/95] posttest, 28.6 % [24/84] follow-up test), and the sonographic normal knee with overuse case (81.2 % [82/101] pretest, 94.7 % [90/95] posttest, 88.1 % [74/84] follow-up test). Significant differences were found between pretest and posttest for treatment utilizing antibiotics in the cellulitis case (p<0.001) but not pretest to 9-week follow-up test (p=0.11). A significant difference in pretest to posttest was found for treatment with OMT and rest, ice, and compression in the overuse case (p=0.004) but not on the 9-week follow-up test (p=0.23). No difference was found pretest to posttest for learning the treatment of joint effusion (p=0.195), and a question regarding treatment of prepatellar bursitis was not asked on the written tests.

Questionnaires

Pretraining and posttraining questionnaires included four common questions. Utilizing a 5-point Likert scale (1 strongly agree to 5 strongly disagree), the mean (SD) self-reported student confidence for correctly identifying normal sonographic anatomy of the anterior knee was 3.50 (1.01) at pretraining and 1.59 (0.72) at posttraining (Table 1). The mean (SD) score for perceived ability to differentiate joint effusion, prepatellar bursitis, and cellulitis utilizing point-of-care ultrasound was 4.33 (0.78) at pretraining and 1.99 (0.78) at posttraining. Student confidence for differentiating these three pathologies utilizing palpation/visual inspection alone went from 3.96 (1.09) at pretraining to 2.08 (0.83) at posttraining. The mean (SD) for self-reported student comfort in handling the ultrasound probe was 2.23 (0.73) at pretraining and 1.77 (0.68) at posttraining (all p<0.001).

Unique to the posttraining questionnaire, the majority of students (93.6 %, 89/95) agreed or strongly agreed that this ultrasound training would change how they diagnosed and managed patients who presented with a painful, swollen knee (mean [SD]=1.49 [0.68]).

In terms of previous ultrasound knowledge, the pretraining questionnaire showed that prior to our study, 89.1 % (90/101) of students had performed six or fewer ultrasound examinations. On the posttraining questionnaire, students reported that the majority of their ultrasound training was received in the medical school curriculum through synchronous lectures (69.5 %, 66/95), hands-on learning laboratories (90.5 %, 86/95), or asynchronous online materials (29.5 %, 28/95). Only 5.3 % (5/95) had pursued additional training through online videos, podcasts, or tutorials.

Replication

Our study found a significant increase in students’ level of confidence for recognizing normal sonographic landmarks and differentiating between joint effusion, prepatellar bursitis, and cellulitis of the anterior knee utilizing ultrasound. A similar protocol could be utilized at other undergraduate medical institutions, whereby the focused ultrasound training could be based on a specific clinical complaint or limited set of possible diagnoses. In this model, we recommend that the teaching intervention be focused, clinically relevant, and as concise as possible. This could be followed by a hands-on assessment incorporating prerecorded video clips of specific sonographic pathology placed in front of the ultrasound screen, which corresponds with probe placement on the standardized patient. As in the current study, written pretests and posttests when combined with pretraining and posttraining questionnaires are useful tools to assess student learning because they are relatively easy to produce and yield valuable insight. Care should also be taken to minimize variability between proctors through the use of specialized instructor training, scripts, and/or checklists. See Figure 3 for an example of the combined script and checklist utilized in the current study.

Limitations

The current study had several limitations. Our sample size was relatively small, and results may have been affected by errors related to self-reported data. One challenge for studies utilizing multiple proctors during assessment is to reduce evaluator bias. We tried to minimize bias in our study by utilizing a large number of proctors with multidisciplinary ultrasound experience, by providing specific proctor training, and through the use of a standardized script and checklist during the hands-on assessment of students. Although these measures likely helped, we still noticed small variations in teaching styles and individual assessment of students. Given the number of outcomes analyzed in the study, we did not perform a separate analysis evaluating uniformity among proctors.

It is difficult to guarantee that specific pathology will be observed on any given day in the clinical setting, and it is even more difficult to ensure in the preclinical years. Therefore, to provide specific findings for our three designated pathologies, we relied on sonographic video clips prerecorded on a digital tablet and placed in front of the ultrasound screen at the appropriate time. Although this protocol is a novel approach, ideally, standardized patients would possess varying degrees of the chosen pathology or, alternatively, the video clips could be loaded directly on the ultrasound system itself, which would help to improve fidelity.

Our protocol deliberately involved a single, focused teaching intervention. Although we did see an immediate increase in student knowledge and self-reported confidence, much of this was lost on the follow-up written test 9 weeks later, and a follow-up hands-on skills assessment was not performed to assess for the retention of technical skills. It should be noted that students were not given additional exposure to the ultrasound content during the 9 weeks leading up to the follow-up written test, which may have affected these scores. In an article about the benefits of spaced repetition, Kang [12] reported that spaced repetition has been shown to benefit memory retention and improve the ability to make what has been learned generalizable to new situations and problem-solving. A recent study by Walsh et al. [13] investigated how repetition of items should be allocated across a fixed number of predetermined sessions to maximize retention. In that study involving 38 participants [13], enhanced learning retention primarily depended on the total number of sessions in which the learner engaged with the content rather than the way in which repetitions occurred. Therefore, it seems reasonable that regular review of the ultrasound content and more frequent scanning opportunities would likely reinforce learning and support retention.

Lastly, the current study took place during OMM laboratory time. Although the initial cohort of students was required to participate in the focused ultrasound training as part of the curriculum, participation in our study was fully voluntary. Therefore, students self-selected to continue participation by completing the written posttest, posttraining questionnaire, and hands-on assessment components of the study and then completing the follow-up written test administered during a subsequent OMM laboratory 9 weeks later. Normal fluctuations in student attendance at laboratory sessions and the choice to continue participating highlight the inherent challenges related to dropout rates that are commonly found in spaced follow-up data collection. Because of these factors, we did not make within-subjects comparisons and all data before and after training were pooled for comparison. To improve participation in future studies, this teaching and testing exercise could be repeated later in the students’ curriculum, when the students are more experienced with ultrasound-based medical education. A different clinical case, centered around a limited number of specific diagnoses, could be utilized.