Effect of COMLEX-USA Level 1 pass/fail score reporting on student stress, test preparation, and performance

Methods

Results

Changes in stress level during Level 1 examination preparation

Among 9,222 first-time test takers in 2023–2024 cycle, 7,309 (79 %) responded to both new questions in the end-of-examination survey. Table 1 shows the survey findings on perceived stress reduction. Among the respondents, 60 % agreed that pass/fail reporting reduced their stress in Level 1 preparation, 25 % were neutral, and 15 % disagreed.

Table 1:

Level 1 performance summary by survey response on stress reduction.

Response	n	Prop	Pass rate	Z-score
				Mean	SD	Min	Max
Disagree	1,121	0.15	91.5 %	−0.19	0.86	−2.89	2.67
Neutral	1,824	0.25	92.2 %	−0.17	0.84	−3.02	2.79
Agree	4,364	0.60	94.4 %	−0.04	0.85	−3.99	3.83
Total	7,309	1.00	93.4 %	−0.09	0.85	−3.99	3.83

1. SD, standard deviation.

Candidates who agreed with stress reduction demonstrated the highest pass rate and mean z-score, followed by those who held neutral and then those who disagreed. The Spearman correlation between perceived stress changes and z-scores was statistically significant (p<0.0001) but negligible in magnitude (r=0.08). Pairwise comparisons showed that the agree group had a significantly higher mean z-score than the neutral and disagree groups (p<0.0001), with small effect sizes (Cohen’s d=0.16 and 0.18, respectively). No significant difference in mean z-scores was observed between the neutral and disagree groups (p=0.50).

As illustrated in Figure 1, each response category included students across a wide range of z-scores, spanning low to high competency levels. However, the agree group tended to have a slightly higher mean z-score compared to the neutral and disagree groups. This helps explain why the overall correlation between perceived stress reduction and z-score was minimal, yet meaningful differences in mean z-scores were observed between the agree group and the other two groups.

Figure 1:

A box plot of z-scores by survey response on stress reduction.

Changes in Level 1 examination preparation

Table 2 presents a summary of the survey findings regarding changes in Level 1 preparation. A majority of respondents (75 %) reported no change following the transition to pass/fail score reporting, 16 % reported more preparation time, and 10 % reported less preparation time.

Table 2:

Level 1 performance summary by survey response on examination preparation.

Response	n	Prop	Pass rate	Z-score
				Mean	SD	Min	Max
More time	1,138	0.16	88.8 %	−0.38	0.79	−3.99	2.50
No change	5,460	0.75	94.0 %	−0.07	0.85	−2.95	3.83
Less time	711	0.10	96.6 %	0.22	0.86	−3.02	3.38
Total	7,309	1.00	93.4 %	−0.09	0.85	−3.99	3.83

1. SD, standard deviation. The sum of proportions exceeds 1.00 as a result of rounding of individual proportions.

Candidates reporting less preparation time had the highest pass rate and mean z-score, followed by those reporting no change and then those reporting more preparation time. Perceived changes in examination preparation were significantly but weakly correlated with z-scores (r=0.18, p<0.0001). Pairwise comparisons revealed clear differences in mean z-scores between the groups. The largest difference was observed between the less-preparation and more-preparation groups (Cohen’s d=0.72), whereas differences between the less-preparation and no-change groups and between the no-change and more-preparation groups were of small to medium magnitude (d=0.34 and 0.37, respectively).

Figure 2 shows the z-score distributions across the three groups. While each group included students from a wide range of competency levels, noticeable shifts in mean z-scores were observed. This may explain why the individual-level correlation between preparation change and z-score was weak, while aggregated group-level comparisons revealed significant differences in mean z-scores.

Figure 2:

A box plot of z-scores by survey response on examination preparation.

Discussion

There is limited research exploring the impact that transitioning from numeric scoring to a pass/fail score report has had on Level 1. This study addresses this gap by evaluating the effects of the transition on student stress, examination preparation, and performance outcomes. Our findings indicate that the transition alleviated stress for a substantial proportion of candidates, with 60 % of the survey respondents reporting reduced stress. However, the transition’s influence on Level 1 preparation was small, as 75 % reported no change in preparation time. This pattern suggests that while removing numeric scores reduced psychological pressure, students continue to invest similar effort, likely reflecting the high-stakes nature of the examination and the desire to ensure a passing outcome. Aggregated analyses further indicate that perceptions of stress reduction and changes in examination preparation may vary across competency levels, although the correlations between these perceptions and competency were negligible to weak at the individual level. The transition was also associated with declines in national performance indicators during the first pass/fail cycle, including the pass rate, mean score, score variability, and maximum score. In the subsequent cycle, the pass rate rebounded and the mean score stabilized, while score variability and maximum score continued to decline. This pattern is consistent with expectations: with only a “pass” required, the incentive for candidates to pursue higher scores have diminished, leading to a lower mean, fewer extreme high scores, and a more compressed overall score distribution.

Consistent with the previous research [4], [7], [8], [9], this study observed an improvement in students’ well-being following the transition to pass/fail score reporting. However, our findings regarding student stress level and examination preparation differ from those of Twardowski et al., [3] which reported no reduction in student stress and noted fewer studying weeks and practice examinations after the transition to pass/fail scoring for Level 1. This may be related to the differences in sample composition. This study analyzed data from 7,309 first-time Level 1 takers in the 2023–2024 cycle (primarily from the Class of 2025) from all COMs, with some students not taking the Step 1 examination. In contrast, Twardowski et al. [3] focused on Classes of 2023 and 2024 from a single COM, where all students were required to take both Level 1 and Step 1.

Furthermore, trends in first-time pass rates for Level 1 differ from those observed for Step 1 following the transition. For Level 1, the first-time pass rate declined slightly in the first testing cycle after the change but rebounded in the second cycle. In contrast, the Step 1 first-time pass rates showed a slight decline in the first cycle, followed by a continued slight decrease in the second cycle [10]. These differences can be attributed to the unique characteristics of the two examinations and the differences in their examinee populations.

This study has several limitations. First, participation in the end-of-examination survey was voluntary, and 1,913 (21 %) first-time test takers in the 2023–2024 cycle chose not to respond. As a result, the findings may be influenced by nonresponse bias. However, the response rate was consistent with historical patterns observed in prior Level 1 end-of-examination surveys. Compared to all respondents, nonrespondents demonstrated lower performance, with a pass rate of 91.7 % and a mean z-score of −0.20. Their z-scores also covered a wide range of competency levels, spanning from −3.40 to 3.32. Second, the study did not include a pretransition cohort of students who took the Level 1 examination under numeric score reporting. Instead, the posttransition cohort was asked to compare their stress and preparation time under pass/fail reporting with how they believed they would have experienced the examination had numeric score reporting been retained. As numeric score reporting was discontinued starting from the 2022–2023 cycle and the posttransition cohort consisted of the first-time test takers in 2023–2024 cycle, such comparisons were necessarily hypothetical, and responses therefore reflect perceived rather than actual experiences, introducing potential bias into the findings. Third, the study lacked objective measures of stress and preparation time. Stress was self-reported rather than assessed with standardized instruments, and preparation time was estimated rather than directly recorded. Practical considerations limited additional data collection: the end-of-exam survey already included 18 items on test administration, format, and content, and it was administered immediately after a demanding 8-h examination when students were likely fatigued. To minimize respondent burden, only two brief questions with simple response options (e.g., agree, strongly agree, less time, significantly less time) were included to capture students’ general perceptions of the transition.

Conclusions

The COMLEX-USA series is designed as a licensure examination to assess competencies essential for the practice of osteopathic medicine. The transition in score reporting for Level 1 preserves that purpose. This study provides preliminary insights into the differences in student stress levels, examination preparation, and examination results following the change of Level 1 score reporting. Further evaluation of these areas is ongoing to ensure that the scoring transition continues to provide valid, reliable information regarding student progress through osteopathic medical education and preparation for licensure to practice osteopathic medicine.