DO seniors and IMGs have lower match probabilities than MD seniors after adjusting for specialty choice and USMLE Step 1 score

Study design and population

We performed a retrospective, secondary analysis of published data reported in the NRMP charting outcomes reports for MD seniors, DO seniors, US IMGs, and non-US IMGs from 2016, 2018, 2020, and 2022 [4], [5], [6, 9], [10], [11], [12], [13], [14], [15], [16], [17]. These were the most recent charting outcomes reports available for each applicant type [21]. We abstracted the number of applicants matching and not matching to their preferred specialty by applicant type, specialty choice, Step 1 score range, and match year. Matching to their preferred specialty was defined by the NRMP as the specialty of the applicant’s first rank. We limited data collection to the 10 specialties with the most available spots in 2022 to limit any bias from specialties that have fewer non-MD applicants [1]. We included internal medicine, family medicine, pediatrics, emergency medicine, psychiatry, surgery, anesthesiology, obstetrics-gynecology, orthopedic surgery, and neurology [1]. Only DO seniors who reported Step 1 scores were included in the analyses allowing for direct comparisons between applicant types. Similarly, we did not collect match data on DO seniors in the 2016 match year because Step 1 scores were not reported in the charting outcomes report [9]. We present additional methods pertaining to the data collection and dataset creation processes in the Supplementary Data.

Intervention and outcome measures

The study’s independent variable was the applicant type (MD seniors, DO seniors, US IMGs, and non-US IMGs). Match year (2016, 2018, 2020, and 2022) and Step 1 score range (≤180, 181–190, 191–200, 201–210, 211–220, 221–230, 231–240, 241–250, 251–260, and >260) were also collected. The primary outcome was matched to the preferred specialty.

Analysis of outcomes

We reported descriptive statistics utilizing counts with percentages. All variables were analyzed as factors because this was the format in which they were abstracted. Unadjusted differences in match proportions were reported with 95 % confidence intervals (CI) utilizing the 2-sample test for equality of proportions with continuity correction. For the main analysis, a multiple variable logistic regression model was performed examining the association between applicant type (independent variable) and matching to the applicant’s preferred specialty (dependent variable), adjusting for specialty choice, Step 1 score range, and match year. Average marginal effects were calculated to determine the effects on the probability of matching and were multiplied by 100 to be reported as adjusted percent differences. Cluster robust standard errors were utilized to calculate 95 % CIs to account for clustering by specialty. We checked for multicollinearity with generalized variance-inflation factors and model fit with the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

We performed sensitivity analyses repeating the primary adjusted analysis with Step 1 score as continuous and discrete 10-point increments and after random forest multiple imputation of missing Step 1 scores from non-DO seniors. We did not impute missing Step 1 scores for DO seniors because Step 1 is not required for DO students to obtain medical licensure or enter a postgraduate training program [22, 23]. However, DO seniors who complete Step 1 may have greater odds of matching [24]; therefore, they are likely to take the examination based on their predicted performance [24, 25]. With the available data, this missingness in Step 1 score for DO applicants is likely missing not at random and would be inappropriate for imputation.

Secondary analyses were performed exploring the interactions of applicant type by specialty, Step 1 score range, and match year. For the analysis by specialty, multiple variable logistic regression models were performed by specialty, examining the association between applicant type (independent variable) and matching to the applicant’s preferred specialty (dependent variable), adjusting for Step 1 score range and match year. For the analysis by Step 1 score, the model adjusted for match year and specialty. Also, for the analysis by match year, the model adjusted for Step 1 score range and specialty. Robust standard errors were utilized for the analysis by specialty, while cluster robust standard errors at the specialty level were utilized for the match year and Step 1 score secondary analyses. Although we performed several different analyses with different subgroups, we only examined a single outcome and this study was not intended to be confirmatory; therefore, we did not adjust for multiplicity [26].

Analyses were performed with R (Version 4.2.1 2022-06-23, R Foundation for Statistical Computing, Vienna, Austria); packages and package versions are listed in the supplement (Supplemental Figure 1). Differences in the probability of matching were considered statistically significant if their 95 % CIs did not cross zero.

Sample size calculation

Across the study population, the rate of matching to the applicant’s preferred specialty is correlated with specialty and Step 1 score [4], [5], [6]. This heterogeneity makes sample size calculation challenging. Powering the study for the main analysis would leave subgroups underpowered. Although the main analysis may be of more interest to premedical students choosing an undergraduate medical education pathway, the subgroup analyses by specialty and Step 1 score would be of more interest to medical students choosing a specialty knowing their Step 1 score. For example, the overall match rate between MD and DO seniors in 2022 was 92.9 vs. 91.3 %, respectively [1]. For this difference, we would need 1,026 observations per group to achieve 90 % power at a significance level of 95 %. However, random or stratified random sampling of the data down to this sample size would leave the specialty and Step 1 score range subgroup analyses extremely underpowered. Therefore, given how important these subgroup analyses likely would be to medical students, we chose to include all of the most recent data from the published charting outcomes reports since 2016 [4], [5], [6, 9], [10], [11], [12], [13], [14], [15], [16], [17], acknowledging that this would make the main analysis overpowered. Nevertheless, some subgroup comparisons would still be underpowered with respect to this sample size calculation.

IRB statement

The study protocol was reviewed by the Allegheny Health Network Institutional Review Board (IRB) staff and considered not to be human subjects research.

Applicant characteristics

In 2016, 2018, 2020, and 2022, 52,902 US MD seniors, 13,846 DO seniors, 15,400 US IMGs, and 22,241 non-US IMGs met our inclusion criteria. The counts from each applicant group matching to their preferred specialty were 49,174 (93.0 %) for MD seniors, 12,075 (87.2 %) for DO seniors, 8,350 (54.2 %) for US IMGs, and 11,874 (53.4 %) for non-US IMGs (Table 1).

Main and sensitivity analyses

After adjusting for specialty choice, Step 1 score, and match year, all non-US MD applicant types had an overall lower percent difference in matching to their preferred specialty than MD seniors, −7.1 % (95 % CI, −11.3 to −2.9) for DO seniors, −45.6 % (−50.6 to −40.5) for US IMGs, and −56.6 % (−61.5 to −51.6) for non-US IMGs (Table 2). These estimates were similar in sensitivity analyses with Step 1 scores modeled as continuous 10-point increments and after multiple imputation of missing Step 1 scores (Supplemental Table 1).

Secondary analyses

The adjusted percent differences in matching between applicant types varied by specialty. For example, there was no significant difference between MD and DO seniors in the adjusted percent difference in matching into Internal Medicine (−1.4 % [95 % CI, −4.3 to 1.4]), Family Medicine (−3.0 % [−7.4 to 1.5]), and Pediatrics (−1.4 % [−3.4 to 0.6]) (Table 3 and Figure 1). The adjusted percent differences for US and non-US IMGs were all much lower than US MD seniors across all included specialties (Table 3 and Figure 1).

Figure 1:

The figure displays the probability of matching into applicants’ preferred specialties between US MD and non-US MD applicants stratified by specialty.

Similarly, DO seniors, US IMGs, and non-US IMGs had lower adjusted percent differences in matching than MD seniors across almost all Step 1 score ranges. The exception was DO seniors with Step 1 scores <200 whose adjusted percent difference compared to MD seniors was −2.0 % (−9.5 to 5.5) (Figure 2 and Supplemental Table 2). Nevertheless, all non-US MD applicant types followed a similar pattern with smaller differences compared to MD seniors with increasing Step 1 scores (Figure 2). Lastly, adjusted percent differences in matching maintained a similar order to the main analysis between DO seniors, US IMGs, and non-US IMGs compared to MD seniors by match year (Supplemental Figure 2 and Supplemental Table 3).

Figure 2:

The figure displays the probability of matching into applicants’ preferred specialties between US MD and non-US MD applicants stratified by USMLE Step 1 score range. Very low and very high score ranges were merged (≤200 and >250) due to few applicants in those categories.

Limitations

Our study was a retrospective, secondary analysis of NRMP surveys, which potentially exposes the results to self-report bias. Nevertheless, most Step 1 scores were verified by the NRMP with the US allopathic and osteopathic medical schools for MD and DO seniors [4, 6, 11, 12, 14, 15, 17]. Furthermore, the results are observational and statistical adjustments were limited to a few variables (specialty choice, Step 1 score, and match year); therefore, unmeasured confounders, such as letters of recommendations as well as program familiarity with non-MD senior applicants, may have impacted the results. Additionally, the proportion of DO programs directors by specialty has been associated with DO applicant match success [31]. Therefore, although we adjusted for specialty choice, the training credentials of program directors among each applicant’s list of programs they applied to may have confounded the results. Also, applicants are likely to apply to medical school and residency programs based on their self-perceived competitiveness, which might bias our observational data. Similarly, our adjusted match probabilities for DO seniors may be greater than observed, because we excluded DO seniors who did not report taking Step 1. In addition, this subgroup may have worse match success than DO seniors who report taking Step 1 [24]. Also, our analysis was limited to the 10 specialties in 2022 with the most spots to limit bias from specialties with fewer non-MD senior applicants. However, this exclusion also limits the generalizability of our results to other specialties. Our results are also not generalizable to applicants who apply to residency outside the US or NRMP match (e.g., military match, San Francisco match, and Urology Residency Matching Program). Finally, our main analysis was overpowered, so there was possibly an elevated risk of type 1 error. In contrast, some subgroups were underpowered, which may have elevated the risk of type 2 error. Nevertheless, our results were largely consistent across the main, sensitivity, and secondary analyses.