A retrospective and correlative analysis of academic and nonacademic predictors of COMLEX level 1 performance

Methods

This study was reviewed and approved by the Institutional Review Board at Kansas City University under 45 CFR 46.104 Exempt Category (d-2 Criteria #i). The IRB number for this study is: 1480505-1. No additional external or internal funding was utilized to conduct this study. Research subjects were informed about the study through an online study information sheet (SIS) that served as the first page of the online survey. The SIS outlined the research investigators, title of the study, commitment from the research participant, probability of harm, how results will be utilized, benefits of participation in the research study, how to withdraw or decline participation, and contact information for the principal investigator and the IRB at Kansas City University. The SIS was the first page of the survey. Clicking on the link for the study from the recruitment materials gave the research participant access to the SIS and the online survey. Participants had to click that they acknowledged the SIS to continue to the survey. No compensation was provided to research participants.

Participants for this study included primarily third-year (OMS III) osteopathic medical students who had completed the COMLEX Level 1 examination in the 2019 academic year by the time the study was conducted. Fourth-year (OMS IV) osteopathic medical students were also eligible if they met this criteria. Participants were recruited utilizing student email distribution lists and via Kansas City University Facebook student groups. An estimated 390 osteopathic medical students had the opportunity to take the survey via email.

The anonymous online survey was developed by the authors and was administered through Qualtrics (Qualtrics, Provo, UT) over a four-week period in January 2020. The single recruitment email and private social media advertisement provided a link to click on to participate, which navigated prospective participants to a consent page and the survey. Participants provided their student identification numbers to allow for student records pertaining to GPA and national licensing board scores to be linked with the data collected in the survey. An independent analyst, who was not involved in the study, linked the data sets together and removed student identification numbers, resulting in an anonymous data set. The independent analyst was a staff member who serves as the manager of assessment for Kansas City University.

Responses were obtained from 72 medical students. Thirteen participants in the sample did not provide their student identification number, prohibiting the independent analyst from being able to link their academic performance data to survey responses, and subsequently their inclusion in statistical analyses. The final sample consisted of 59 participants. There were 37 male (62.7%) and 22 female (37.3%) participants. Among the sample, 98.3% (n=58) were OMS III students with only one participant reporting as an OMS IV student. The average age of the sample was 27.14 ± 2.58 (mean ± standard deviation; range: 24–36).

Research participants were surveyed for information regarding demographics, study behavior, and nonacademic variables such as preferred specialty of interest, financial burden, and resilience. Questions were in binary, multiple-choice, or free-text format with programmed follow-ups dependent on participant answers. Example survey questions included: how many hours per week did you take for personal time and recreation during dedicated board study, how many practice exams did you complete, and how much money did you roughly spend on board preparation materials. In regard to financial burden, participants were also asked to estimate to the best of their ability the total amount of money spent on preparation materials. Resilience was evaluated by the Connor Davidson-Resilience 10-item Scale (CD-RISC 10), with the highest score of 50 [29]. The scale utilizes a modified five-point Likert scale, with 1 corresponding to “not true at all” and 5 corresponding to “true nearly all the time.” Higher overall scores indicate greater levels of resilience. See Appendix, Supplementary Material for a full copy of the survey.

When participants were asked about the hours spent on personal time during NLE preparation, some did not report engaging in that behavior. Only the participants who provided an estimate of hours spent for personal time were utilized for analysis (Table 1). Similarly, one participant did not complete the CD-RISC and two participants did not report the dollar amount spent on preparation materials. All analyses were run with complete case data. Independent sample t-tests were conducted to determine demographic differences in academic and nonacademic variables. Pearson correlations were conducted to evaluate relationships between study variables. A regression was conducted to determine if financial, psychological, and study habit variables were associated with a student’s COMLEX score after controlling for GPA. A p value of 0.05 was considered statistically significant. Statistical analyses were completed utilizing SPSS software version 26.

Results

Preferred specialties among the participants were queried and stratified into two groups: low-competitiveness (LC) and high-competitiveness (HC). This stratification was conducted utilizing established competitiveness groupings based on National Resident Matching Program (NRMP) charting outcomes match data, which were modified from previously published criteria [30]. The moderate and high competitiveness specialties were combined into the HC group. Over half (52.5%) of the sample indicated an interest in a high-competitive specialty, whereas 39% reported interest in a low-competitive specialty, 3.4% in multiple specialties, and 5.1% were undecided.

Descriptive statistics for academic and nonacademic variables are presented in Table 1. The average GPA for the final sample was 3.33 ± 0.37. The average COMLEX Level 1 score was 598.4 ± 78.7, and the goal pretest COMLEX Level 1 score was 587.97 ± 38.33. Further demographic data reveal money spent on board preparation ($1,319.12 ± $689.17), hours of sleep per night during dedicated study (7.34 ± 0.92), hours per week spent for personal time during dedicated study (n=46, 19.7 ± 18.53), and CD-RISC 10 (40.53 ± 4.24). Independent samples t-tests were conducted to determine differences in academic and nonacademic variables between the LC and HC groups. There was a statistically significant difference between these groups in the number of practice exams completed [t(52)=−3.12, p<0.01]. Those in the HC group (6.81 ± 2.36) completed on average approximately two additional practice exams compared to the LC group (4.78 ± 2.35). However, this difference in practice exams did not influence overall COMLEX Level 1 performance. Pearson correlations between academic and nonacademic factors are presented in Table 2.

An ordinary least squares regression was conducted to determine the contribution of academic and nonacademic factors in the prediction of COMLEX Level 1 scores after controlling for GPA. GPA was entered as a predictor in the first model. GPA, number of questions completed, money spent on board preparation, and number of weeks of study were entered as predictors into the second model. Overall, the first model with GPA was statistically significant in accounting for 53.2% (adjusted R²) of the variance in COMLEX Level 1 scores [F (1, 55)=64.76, p<0.001]. The second model accounted for 75.8% of the variance in COMLEX Level 1 scores [F (4, 52)=44.91, p<0.001]. Of the variables in the second model, GPA was still a statistically significant predictor of COMLEX scores (β=0.62, p<0.001), along with the number of weeks of study (β=0.30, p<0.001), money spent on preparation (β=0.21, p<0.01), and number of questions completed (β=0.20, p<0.01). See Table 3 for regression results.

Because money spent on board preparation was found to be a statistically significant predictor of COMLEX scores (β=0.21, p<0.01), the quantity of resources utilized was assessed. Three student spending categories were stratified as high ($1,600–$3,000), average ($1,000–$1,500), and low ($100–$900) spenders. These three groups did not statistically differ in the average number of resources (content or question banks) they reported (F statistics <1; p>0.05), and the number of resources utilized was not related to COMLEX performance (Pearson correlations ranging from −0.11 to 0.13, p>0.05). See Table 4 for exam preparation resources by spending group based on answers to survey questions.

Discussion

Medical students prepare for NLEs in different ways and with different motivations. In this study, we surveyed OMS III and OMS IV osteopathic medical students to determine correlations between academic variables, such as GPA and COMLEX Level 1 scores, with nonacademic variables such as study habits and resilience. On average, the medical students in our sample spent greater than $1,000 on board preparation materials, yet the number of resources purchased did not correlate with higher COMLEX Level 1 scores. In addition, students pursuing highly competitive residencies take, on average, two additional practice exams, yet these additional practice exams do not correlate with higher COMLEX Level 1 exam scores.

Results indicated that there were no statistically significant relationships between resilience, amount of sleep, and amount of free time spent with performance on the COMLEX Level 1 exam. Resiliency scores in the sample were generally high, indicating that many students subjectively endorsed having resilient traits. This may be a feature of the sample or indicative of medical students’ ability to persist in the face of many challenges. Sleep and amount of free time varied in the sample and may speak more to individual differences around responsibilities outside of medical school (such as family obligations) or to the nature of the sample size. These variables should be further evaluated for their relationship with NLE performance as well as other factors such as motivation and confidence.

After controlling for GPA, three factors were able to explain a significant amount of the variance in COMLEX Level 1 scores. These additional factors were the number of questions completed, number of weeks of study, and money spent on board preparation. Retrieval practice, spaced practice, and self-testing have been shown to be successful study techniques because their use improved student retention of study material and enhanced student learning [31], [32], [33]. In one such study, long-term retention and application of materials were improved (p<0.0001) in medical students (n=47) that utilized repeated testing [33]. The results of this study corroborate these other studies by demonstrating that time spent studying and the number of questions completed are significant factors associated with a student’s COMLEX Level 1 performance.

Money spent on board preparation is also found to be an additional factor associated with a student’s COMLEX Level 1 performance. This result is potentially problematic and may place students that are socioeconomically disadvantaged in an unfair situation. The data indicate that the high spending group ($1,600–$3,000) did not utilize more resources for studying than other groups. The high spending group utilized less total content resources, on average, when compared to the other spending groups. This may indicate that the high spending group purchased fewer, more expensive resources. For example, all of the students in the high spending group utilized the study resource UWorld, whereas 88% of students from the low spending group utilized UWorld. Overall, the cost of NLE preparation may disproportionately impact socioeconomically disadvantaged medical students. More research is needed to tease apart the difference in board preparation investment, to assess where the money from the high spending group is going, and to evaluate how NLE preparation cost impacts career development outcomes such as residency match success and long-term debt burden.

The findings in this study are limited in a few different ways. The final sample size is small, and the results from this study may not be generalizable to other medical student populations. Students with higher levels of resilience and academic performance may have been more likely to respond to the survey than other students. Indeed, the average COMLEX Level 1 score and GPA reported in the survey results are greater than the average for the entire class, leading to sampling bias and response/nonresponse bias. Despite the small sample size, our results share trends with other similar studies on key factors such as COMLEX and USMLE correlation and with the number of questions completed during NLE preparation predicting NLE performance [34, 35]. Specific prior study results revealed that attempts by medical students (n=274) at more practice questions was significantly associated with USMLE Step 1 scores [35]. Several of the factors surveyed, including financial investment and study behaviors, are reported in a subjective fashion. Students reported information retrospectively and may have misremembered or guessed how they studied or the amount they spent on resources.

The finding that money spent on NLE preparation is associated with a student’s COMLEX Level 1 exam performance prompts future research. It is not known whether students felt adequately prepared for the NLEs by the medical school curriculum or if students are motivated to purchase outside board preparation materials based on feelings of anxiety to increase their NLE performance. It also remains to be seen whether the transition of the COMLEX Level 1 and USMLE Step 1 to pass/fail grading will change the amount of money that students spend on board exam preparation. It may be that COMLEX Level 2 CE and USMLE Step 2 CK become more important for postgraduate residency candidacy. Further studies could evaluate how resilience, financial investment, and other factors relate to performance on these NLEs. Students take COMLEX Level 2 CE and USMLE Step 2 CK later in medical school, so they likely have a better understanding of their career plans. Future studies could also assess how clinical experience has influenced student resilience.

Our data indicate that time spent studying and the number of questions completed are essential factors associated with COMLEX Level 1 performance. A potential intervention that may operationalize this finding is the practice of spaced retrieval. It is entirely possible that the introduction of spaced retrieval practice to students as a study technique may enhance student performance on NLEs by giving students an effective study strategy to interweave NLE practice questions within the standard medical curriculum. In addition, the use of self-testing and spaced repetition would allow medical students to study for a longer period of time and to see more practice questions prior to taking an NLE, all of which may improve a student’s NLE performance. These study techniques represent the philosophical opposite of cramming and have the added benefit of enhancing long-term retention of the studied material [36]. Additionally, spaced-retrieval practice has been shown to protect memory against acute stress, which may be an additional positive effect to potentially enhance NLE performance [37]. Informing medical students of spaced retrieval practice early in their career would be a simple and inexpensive way to potentially enhance student performance on NLEs.

Conclusions

Noncognitive factors, such as psychological traits, time management skills, and socioeconomic status, are increasingly being elucidated as relevant to scholarly performance. This is especially pertinent given the evolving landscape of medical education and heightened emphasis on student well-being. This study preliminarily highlights money spent, practice questions completed, and amount of time spent studying as being associated with a student’s COMLEX Level 1 performance. Furthermore, amount of sleep, amount of free time, and resiliency are not correlated with performance on the COMLEX Level 1. Lastly, despite the amount of money that students spent on extracurricular NLE preparation resources, the number of extracurricular resources utilized is not correlated with increased COMLEX Level 1 performance. In conclusion, this research study suggests that one possible inexpensive operationalization of these results would be the inclusion of spaced retrieval practice in NLE preparation. Further research is required to identify how these and other factors should be accounted for in osteopathic medical education curricula and resource allocation.