A validity study of COMLEX-USA Level 3 with the new test design

Measures

Study sample

The analysis data for this study contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The first-attempt test scores of the L2-PE for the same group of candidates were also utilized to study the association between the scores and subscores from the two examinations. Demographic data, including gender, race and first language, were self-reported by the candidates during registration; as such, there would be missing data in the demographic categories if the candidates chose not to answer. Among the 8,366 candidates, 8,087 (96.7 %) reported demographic information for gender, race, and first language.

Analysis

Multiple potential sources of validity evidence were explored for the newly designed L3 examination scores utilizing Kane’s validity framework. For Scoring, this includes content coverage of the L3 examination, standardization across forms, and QC in the scoring and reporting processes. Here, the validity evidence is based primarily on the documentation of test assembly and scoring processes. For Generalization, we estimated the reliability of the L3 examination scores, rater consistency in the scoring of the CDM SA items, and decision accuracy and decision consistency of pass/fail classifications. For Extrapolation, we summarized the association between the scores from the MCQ component and CDM component and between scores from different competency domains of the L3 examination, the association of L3 scores with scores from a performance-based assessment that measure related and unrelated constructs (L2-PE), and subgroup comparisons for candidates taking the L3 examination. We expected that the MCQ and CDM scores would be moderately related. We also hypothesized that, based on the measured domains, specific components of the L3 and L2-PE examinations would be more highly related than others (e.g., interpersonal skills and communication and professionalism as measured in L3 and L2-PE, L3 CDM component and ability to write a SOAP note [Level 2-PE]). Finally, based on the prevailing literature, we postulated that certain groups of candidates (e.g., those whose first language was not English) would not perform as well, on average, on L3. For the Implication/Decision inference, we documented and justified the pass/fail standard. Table 1 shows how these sources of evidence fit into Kane’s validity framework and the analysis methodology for each piece of evidence.

The investigation was considered exempt for review by the Institutional Review Board (IRB) of the NBOME. Only de-identified data was utilized for the analyses.

Scoring

The blueprint for the L3 examination was constructed by osteopathic physicians in the context of clinical problem-solving [23]. To control item exposure and ensure test security, 16 parallel forms were administered in this cycle. Before the forms were assembled, SMEs reviewed the item pools utilized in the current cycle for accuracy and applicability with respect to current medical practice; only items deemed eligible were kept in the pools for form assembly. The forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties [24]. All 16 forms in this study met the blueprint specifications and were reviewed by SMEs through a standardized procedure to confirm content coverage. Items were replaced and reviewed again by SMEs if they were found to have content overlap with other items on the same form. Through the ATA procedure, the forms were parallel in average item difficulty and item discrimination, and test information was maximized at the cut score points to ensure the highest precision [16].

Slight differences in form difficulty were adjusted through an item-response theory-based concurrent calibration procedure [25]. The passing scores for each examination form were adjusted accordingly to account for any differences in form difficulty. QC procedures were implemented in the scoring and reporting process to ensure the accuracy of the computed scores and score reports delivered to the candidates. These include independent rating of SA items, monitoring rating consistency, key validation (ensuring the administered items are psychometrically sound), and independent replication of item calibration, equating, and scoring.

The SA items are scored by the scoring engine through auto matches if the responses correspond to the prescored responses in the machine or by human scorers if the responses differ. Currently, all unmatched responses are scored independently by at least two human scorers. In case of disagreement between the two scorers, a third scorer will review responses and scores from the first two scorers, providing an additional score. The scoring engine also conducts a consistency check and flags inconsistent scores for similar responses. Following this process, a QC scorer will review all scoring results from the third scorer and those flagged as inconsistent by the scoring engine and make a final determination of the score. The SA scorers are board-certified osteopathic physicians who have gone through a rigorous training process. They receive instructions on the scoring rubrics and the scoring engine and then complete a practice assignment. Scorers are provided feedback on their scores for the practice assignment, and additional practice is assigned if they do not meet certification criteria. Once new scorers are certified, they are assigned to rate pretest cases first and be paired with experienced scorers. After they can maintain an inter-rater reliability above 0.95 for three batches, they are promoted to primary scorer and are permitted to score any CDM case. Scorers at any level receive additional training and feedback if their scores consistently differ from the final scores.

All items counted toward the candidates’ total scores have undergone pretesting and validation. Item analyses are conducted to evaluate the quality of these pretest items. Item statistics, including item difficulty, item discrimination, and distractor information, are utilized to flag items for content review. Additionally, based on candidate comments on specific items, a content review could also be initiated. During the content review, SMEs review the flagged items, validate the correct answers, and determine whether the items can be utilized in future forms or need to be revised and re-pretested.

All psychometric processes that lead to the final scores for the candidates, including item calibration, equating, and scoring, are independently replicated by at least two analysts to ensure scoring accuracy. In addition, candidates may request score confirmation for individual scores after the scores have been reported. In this case, additional score validation will be conducted.

Generalization

The estimated reliabilities of the L3 examination total scores, as measured by stratified alpha [26], ranged from 0.86 to 0.89 across the 16 forms with an average of 0.88. For CDM SA items, the inter-rater consistency between the primary and secondary raters was above 95 %. Decision accuracy and decision consistency indices indicate the accuracy and consistency of classification of the examinees into the pass/fail categories. Utilizing Rudner’s method [27], the overall decision accuracy estimate of the L3 examination is 0.98 and the overall decision consistency estimate is 0.97.

Extrapolation

Table 2 shows the Pearson correlations between different L3 competency domains and between the MCQ component and the CDM component; reliability estimates are also provided. Pearson correlations between each competency domain and the L3 total score ranged from 0.41 to 0.92 and were all statistically significant (p<0.01). The L3 total score was most strongly correlated with osteopathic patient care and procedural skills (0.92) and application of knowledge (0.82) and least strongly correlated with interpersonal and communication skills (0.41) and professionalism (0.43).

Pearson correlations between the competency domain subscores for L3 ranged from 0.15 to 0.69, with the strongest of those correlations between osteopathic patient care and procedural skills and application of knowledge (r=0.69) and the weakest between interpersonal and communication skills and osteopathic principles, practice, and manipulative treatment and between professionalism and osteopathic principles, practice, and manipulative treatment (both 0.15). In general, interpersonal and communication skills and professionalism had weaker correlations with other competency domains. There was a moderate association between the MCQ component and the CDM component, with a Pearson correlation of 0.51. Correlations between the MCQ component score and the competency domain scores were generally higher than those between the CDM component scores and the competency domain scores.

Table 3 shows the disattenuated correlations (corrected for measurement error) between the L3 scores and L2-PE scores. The L3 total score and subscores had higher disattenuated correlations with the BM domain score than with the HM domain score. The disattenuated correlations between the L3 total score or subscores and the BM domain score ranged from 0.22 to 0.44, with that between professionalism and the BM domain score the lowest (0.22). The disattenuated correlations between the L3 total score and subscores and the HM domain score ranged from 0.13 to 0.31. As expected, interpersonal and communication skills and professionalism had the highest disattenuated correlations with HM among all the L3 subscores. Because systems-based practice had a relatively higher correlation with interpersonal and communication skills and professionalism, it also had relatively higher correlation to HM. The CDM component also had higher disattenuated correlations with HM than the MCQ component.

The disattenuated correlations between the L3 total score and subscores and the subscores of BM ranged from 0.15 to 0.41. Interpersonal and communication skills tend to have higher correlations with the BM subscores than other L3 competency domains. The CDM component, designed to assess higher-order clinical application functions, also had higher correlation with SOAP and OMT than the MCQ component.

Table 4 shows the average standard scores, standard deviation of standard scores, effect size of standard score difference, and passing rates by subgroups for the L3 examination. In general, subgroups of male candidates, white candidates, and candidates with English as their first languages had higher average scores than their corresponding counterpart in the same category, and their passing rates were slightly higher. The standard score difference [28] is bigger in the race (effect size=0.44) and first language categories (effect size=0.3) and smaller in the gender category (effect size=0.15).

Implication/Decision

The pass/fail standard is criterion-based for the L3 examination and is based solely on a candidate’s performance on the total examination. For the L3 examination, a passing score means that the candidate has demonstrated at least a minimal level of competency required for the safe and effective unsupervised, independent practice of osteopathic medicine. The NBOME follows industry-standard best practices to determine minimum pass/fail standards for its COMLEX-USA examinations [2, 29]. This includes gathering data from stakeholders concerning expected passing rates and convening a standard-setting panel to participate in a modified Angoff standard-setting study. The NBOME also periodically reviews and updates the blueprint (in 2-year cycles) and the minimum pass/fail standards (every 3–5 years) to ensure that the tested knowledge and skills on the examination and the standards reflect current osteopathic medical education and practice required for licensure.