The predictive validity of MCAT scores and undergraduate GPA for COMLEX-USA licensure exam performance of students enrolled in osteopathic medical schools

Data sources and participants

This study was part of a collaborative effort between the AAMC, the American Association of Colleges of Osteopathic Medicine (AACOM) and the NBOME. Each participant’s most recent MCAT record was provided by the AAMC; participant background characteristics and matriculation information were provided by AACOM; and the NBOME provided COMLEX-USA examination results for students who consented for their results to be utilized for research purposes.

The authors analyzed data from individuals who matriculated to 1 of 39 accredited DO-granting medical schools in the United States during the 2017 application cycle. To qualify for inclusion in the study, participants must have reported UGPA and scores from the current MCAT examination on their application. In the sample, a total of 17,470 individuals applied to at least one DO-granting medical school, and 5,919 (33.9 %) individuals matriculated.

Based on the 5,919 individuals who matriculated, 52.4 % (n=3,100) were male and 47.5 % (n=2,811) were female. A total of 683 (12.4 %) matriculants self-reported a race or ethnicity that was categorized as underrepresented in medicine (UIM). This group of students consisted of individuals who identified as Black or African American; Hispanic, Latino or Spanish Origin; American Indian or Alaska Native; or Native Hawaiian or other Pacific Islander. Further, 1,295 (23.4 %) matriculants’ parents did not attain a bachelor’s degree.

For the validity analyses, matriculants must have completed either the COMLEX-USA (Level 1 or Level 2-CE) examination, or both, during their current program of study. The final sample frame consisted of 5,208 students for the COMLEX-USA Level 1 examination and 4,795 students for the COMLEX-USA Level 2 CE examination. All students with COMLEX-USA Level 2-CE examination results had COMLEX-USA Level 1 examination results. Of the 5,919 matriculant records provided by AACOM, 711 (12 %) could not be matched to a COMLEX-USA Level 1 examination record and 1,124 (19 %) could not be matched to either a COMLEX-USA Level 1 or 2 CE examination record.

Selection variables and outcome measures

The authors investigated two selection variables: MCAT total scores and cumulative UGPA. These variables were chosen because they typically are regarded as two important variables utilized by admissions committees [4, 12, 13]. The MCAT total score, ranging from 472–528, is the sum of scores from the 4 sections of the examination (Biological and Biochemical Foundations of Living Systems; Chemical and Physical Foundations of Biological Systems; Psychological, Social, and Biological Foundations of Behavior; and Critical Analysis and Reasoning Skills). We studied the most recent MCAT total scores.

To determine if MCAT total scores and UGPA provide comparable prediction of students’ performance on COMLEX-USA Level 1 and Level 2-CE examinations by sex, race and ethnicity, and SES, we divided students into two groups for each demographic variable. Students self-reported their sex as either male or female. Race and ethnicity were categorized into UIM or non-UIM based on the aforementioned definitions. Because parental educational attainment is strongly associated with family income and wealth, parental educational attainment was utilized as a proxy for SES [14]. Students for whom neither parent held a bachelor’s degree were categorized as lower-SES, and students for whom at least one parent held a bachelor’s degree were categorized as upper-SES.

The outcome variables investigated in this study were student scores based on their first attempt at the COMLEX-USA Level 1 and COMLEX-USA Level 2-CE licensure examinations. According to the NBOME, the COMLEX-USA Level 1 assesses “foundational biomedical sciences with other areas of medical knowledge relevant to clinical problem-solving and the promotion of health maintenance” [15]. The COMLEX-USA Level 2-CE assesses “application of knowledge in clinical and foundational biomedical sciences with other physician competences related to the clinical care of patients and promoting health in supervised clinical settings” [16]. Both COMLEX-USA examinations produce a scaled score and a pass/fail outcome measure. The scaled score range for both examinations is 9–999. This study was deemed exempt by the American Institutes for Research in Behavioral Sciences Institutional Review Board (Protocol # EX00591).

Data analysis and procedures

The authors utilized three linear regression models to determine if MCAT total scores and UGPA predict performance on students’ first attempt of COMLEX-USA Level 1 and Level 2-CE examinations. The first model examined the predictive value of MCAT scores alone; the second model examined the predictive value of UGPA alone; and the third model examined the predictive value of MCAT scores and UGPA combined. Due to the many nuances and variations in medical schools (e.g., mission, applicant pools, curricula, etc.), all analyses were estimated within each medical school and then summarized across schools. Because COMLEX-USA scores can only be obtained for matriculants, the bivariate UGPA and MCAT score distribution of the analytical sample is restricted relative to the population. We utilized the simultaneous multivariate correction method given in Sackett and Yang [17] to adjust the correlation in the restricted sample to estimate the correlation between the predictors and COMLEX-USA scores. This procedure alleviated the underestimation bias associated with utilizing only the matriculant sample. For each model, correlations across schools were summarized with median and interquartile range (IQR).

The authors utilized a logistic regression on pass/fail outcomes. Schools with 100 % pass rates on either the COMLEX-USA Level 1 or Level 2-CE examinations were excluded from analysis because relations between variables cannot be estimated when one or both is constant in a sample. Thus, a logistic regression was performed on 35 schools for the Level 1 analysis and on 31 schools for the Level 2-CE analysis. We examined the area under the receiver operating curve (AUC) to determine how well each model differentiated students. We then summarized the AUC for each model across schools with the median and IQR.

For the comparable prediction analysis, we calculated the aggregate prediction error within each group and then summarized the values across schools. For scores, this was computed as the average of the difference between the predicted and observed scores. For pass/fail indicators, this was determined by dividing the observed success rate by the predicted success rate. Standardized effect sizes (Cohen’s d and Cohen’s h, respectively) were then calculated to discern the practical significance of any observed differences [18]. We performed all statistical analyses utilizing R statistical computing programming language version 4.2.3 for Windows (R Core Team).

Overall summary of MCAT scores, UGPA, and COMLEX-USA performance outcomes

The mean MCAT total score for matriculants (n=5,208) was 503.0 (SD=5.5). The mean cumulative UGPA for matriculants was 3.55 (SD=0.26). With respect to matriculants’ performance on COMLEX-USA examinations, the mean COMLEX-USA Level 1 score was 533.1 (SD=87.0), with 4,941 (94.9 %) passing on their first attempt. The mean COMLEX-USA Level 2 score was 573.5 (SD=95.4), with 4,677 (97.5 %) passing on their first attempt. A statistical summary of the overall results by subgroup is presented in Table 1.

Predictive value of MCAT total scores and UGPA for COMLEX-USA examination performance

For COMLEX-USA Level 1 scores, MCAT scores alone provided superior predictive value to UGPA alone, yielding a median (IQR) correlation of 0.47 (0.42–0.51) compared to 0.28 (0.22–0.33) for UGPA alone. The best predictive value was provided by MCAT scores and UGPA combined, which yielded a median (IQR) correlation of 0.51 (0.45–0.55). A similar pattern was found for COMLEX-USA Level 2-CE scores. MCAT scores alone provided superior predictive value to UGPA alone, yielding a median (IQR) correlation of 0.39 (0.36–0.47) compared to 0.25 (0.23–0.32). MCAT scores and UGPA combined provided the best predictive value, yielding a median (IQR) correlation of 0.45 (0.40–0.49).

Regarding the prediction of success and failure on COMLEX-USA Level 1, MCAT scores alone provided superior classification accuracy to UGPA alone, each yielding a median (IQR) AUC of 65 % (57 %–72 %) and 57 % (55 %–64 %), respectively (Table 2). MCAT scores and UGPA combined provided the best classification accuracy, with a median (IQR) AUC of 70 % (64–75 %). For COMLEX-USA Level 2, MCAT scores alone provided superior classification accuracy than UGPA alone, each yielding a median (IQR) AUC of 71 % (59 %–78 %) and 65 % (62 %–70 %), respectively. MCAT scores and UGPA combined provided the best classification accuracy, with a median (IQR) AUC of 75 % (70–81 %).

Comparable prediction of COMLEX-USA examination scores by sociodemographic characteristics

Overall, participants belonging to both under- and well-represented sociodemographic groups did not score meaningfully higher or lower on the COMLEX-USA Level 1 or 2-CE examinations than what would be predicted for any other student with the same MCAT total score and UGPA (Table 3). Notably, for all comparisons, Cohen’s d effect size estimates ranged from 0.00 to 0.20 in magnitude, indicating negligible practical differences between the predicted and observed scores [18]. Although practically equivalent, predictions based on MCAT scores alone deviated less from the observed scores than those based on UGPA alone; and the predictions based on MCAT scores and UGPA together provided the most comparable prediction.

Comparable prediction of passing the COMLEX-USA examinations by sociodemographic characteristics

Overall, participants belonging to both under- and well-represented sociodemographic groups passed both COMLEX-USA Level 1 and Level 2-CE examinations at similar rates to what would be predicted for any other student with the same values on the selection variables (Table 4). The Cohen’s h effect sizes for all comparisons ranged from 0.00 to 0.06, indicating negligible practical differences between predicted and observed pass rates [18]. As with the prediction of COMLEX-USA examination scores, predictions based on MCAT scores alone deviated less from the observed pass rates than those based on UGPA alone; in addition, the predictions based on MCAT scores and UGPA together provided the most comparable prediction.