Understanding COMLEX-USA Level-1 as a Pass/Fail examination: impact and opportunities

Methods

The Institutional Review Board (IRB) at ATSU-Kirksville deemed the study exempt (IRB #: PS20220223-001). The Arkansas College of Health Education IRB provided a memo stating that the study was exempt per A.T. Still University-Kirksville.

With funding awarded through an AACOM research grant, survey data were collected to investigate the perceived impact of the COMLEX-USA Level 1 transition to Pass/Fail scoring. All study participants agreed through informed consent that was sent before the completion of the surveys. All participation was voluntary. No compensation was provided for participation.

The study was completed with the assistance of the Indiana University Center for Survey Research (IUCSR). The survey field period (n=1,304) was December 8, 2022, through February 2, 2023. On December 8, 2022, the IUCSR sent invitations to participate via email. Subsequently, the IUCSR sent four email reminders on December 16, 2022, January 9, 2023, January 18, 2023, and January 25, 2023. Surveys were distributed to all current OMS of both ATSU-KCOM and ARCOM (n=607), recent graduates (Classes of 2021 and 2022; n=623) of both ATSU-KCOM and ARCOM, all full-time faculty at ATSU-KCOM and ARCOM who act as academic advisors (n=61), and ATSU-KCOM and ARCOM curriculum committee members (n=13). Those faculty who are part-time and do not act as student advisors were excluded from the survey, as were ex officio members of the curriculum committee.

Additionally, the follow-up survey consisted of sliding-scale questions, open-ended responses, and additional Likert scale questions to gauge student perceptions of: the pressure to take the USMLE; the stress level in taking and preparing for COMLEX-USA Level 2CE; the stress incurred in the residency placement process; the emphasis on audition rotations; and the competition for residency placement. The results demonstrate findings from both the initial survey and the follow-up survey.

Both the initial survey and the follow-up survey utilized a 4-point Likert scale. The 4-point Likert scale was chosen to avoid the ambiguity of neutral participants. A selection of neutral would not reflect an accurate representation of participant perception.

Results

Among the survey respondents (n=302; response rate=23.2 %), the change was not perceived as favorable (mean scores<2.5), especially among the students. Based on written responses, the primary reasons that the change was not favorable include a lack of motivation to perform on COMLEX-USA Level 1 and the added stress to perform on COMLEX-USA Level 2CE. Responses also reported that the change would adversely affect audition rotation opportunities and placement into residency programs.

Hypothesis 1 was disproven because there were no statistically significant differences in perception between the two COMs surveyed, with the means less than 3.0, respectively.

When compared to the students in Quartile 1 (76th–99th percentile), the students in Quartile 4 (1st–25th percentile) reported that the change in COMLEX-USA Level 1 to Pass/Fail:

1. is a positive change.

2. will increase their chance of matching into a competitive residency.

3. will place more emphasis on their clinical skills when being considered for a residency position.

Students in Quartile 4 viewed the change to Pass/Fail as positive when compared to Quartile 1 students. Likewise, Quartile 4 students were statistically more likely to believe that the change would increase their residency match competitiveness and that their clinical skills would be a more important factor than test scores. The remaining questions regarding this hypothesis did not result in significant differences, therefore the data regarding hypothesis 2 is mixed.

The data regarding recent graduates’ perception of the helpfulness of a Pass/Fail COMLEX-USA Level 1 examination in the NRMP match was equivocal. This hypothesis was not supported.

Data were only achieved from one COM. However, there was not a strong perception that the Pass/Fail change would be helpful for student advising. This hypothesis was not supported.

Faculty on the curriculum committees of both COMs were neutral regarding Pass/Fail on COMLEX-USA Level 1 as an opportunity. This hypothesis was disproven.

Discussion

Historically, COMLEX-USA Level 1 scores have been a benchmark for residency application review and selection. Regarding the rationale given for the move to Pass/Fail, reducing student stress and improving well-being has been noted as a consideration. Studies have shown that high-stakes examinations contribute to considerable psychological stress among medical students, impacting their overall well-being and academic performance [8]. However, a recent study demonstrated that while there is some evidence that Pass/Fail curricula may improve medical students’ mental well-being and satisfaction, these effects did not translate to board examinations [9]. Likewise, Baniadam et al. [10] reported that when the USMLE Step 1 examination was scored, students felt more stress throughout the second year, whereas the move to Pass/Fail simply moved the high-stress period to closer to the examination. Ultimately, the stress levels experienced were the same in both the cohort with scored examinations and the Pass/Fail cohort.

Osteopathic medical education aims to produce physicians competent in holistic patient care. Among COM faculty and administrators, there has been some discussion that the move to Pass/Fail for COMLEX-USA Level 1 may allow less focus on “teaching to the boards” and allow increased allocation of time on clinical medicine and skills development in the early undergraduate curriculum. While the literature has not addressed this idea, this study indicated that faculty serving on curriculum committees have not identified this as an opportunity. In fact, there are concerns about the ability to guide struggling students.

The study results indicate the majority of students and faculty perceive the change to Pass/Fail on the COMLEX-USA Level 1 examination as generally unfavorable. The sole outlier is the student cohort in the lowest quartile of their class, who believe it will be advantageous to them. Consistent with the findings of Girard et al. [4], those in the top quartile were least likely to consider the change as favorably impacting their residency competitiveness. Additionally, faculty do not consider the change to be an opportunity to focus more of the curriculum on clinical knowledge and skills training. Limitations include the “initial” impressions of the change to Pass/Fail. Perceptions certainly may change over time. Further, only two COMs were included, both in the Midwest United States. More geographically diverse studies may be warranted. Additionally, among the two participating institutions in this study, one was the original osteopathic college, and the other was relatively new beginning in 2017. This limits the population sample by another factor other than geographic location.

On the other hand, the similarities between the perceptions at the oldest COM (ATSU-KCOM) and a newer school (ARCOM) may indicate that these findings can be generalizable to the osteopathic medical education landscape at large.

Conclusions

Among the initial survey respondents (n=302; response rate=23.2 %), the change in COMLEX-USA Level 1 scoring to Pass/Fail is not perceived as favorable, especially among the students. Based on written responses, the primary reasons include:

1. A lack of motivation to perform on COMLEX-USA Level 1.

2. Added stress to perform on COMLEX-USA Level 2CE.

3. Adverse effects on audition rotation opportunities.

4. Adverse effects on placement into residency programs.

When compared to the students in Quartile 1 (76th–99th percentile), the students in Quartile 4 (1st–25th percentile) reported that the change in COMLEX-USA Level 1 to Pass/Fail will be positive, will increase their chance of matching into a competitive residency, and will place more emphasis on their clinical skills when being considered for a residency position.

Based on the student responses to the survey, the results of this study add to the limited literature published demonstrating that student stress has not lessened because of the transition to Pass/Fail performance reporting. Rather, the stress on osteopathic medical students appears to have increased because they perceive the COMLEX-USA Level 2CE scores are even more important to residency program directors as they screen candidates for interviews.

Continuing consideration of how best to advise students preparing for COMLEX-USA Level 1, a high-stakes Pass/Fail examination is necessary. Further, the impact of the change to Pass/Fail on student study behavior during their coursework and in preparation for the COMLEX-USA Level 1 examination should be a consideration. Understanding how residency program directors will modify the resident applicant screening processes is important and yet unknown. Finally, the perceived increased importance of COMLEX-USA Level 2CE examination scores may impact student stress and well-being. All of these are avenues for future research.