Response to “COMSAE phase 1: value added”

To the Editor:

We thank Dr. Sefcik and Prof. Petsche for their interest [1] in our article, “Meaningful Use of COMSAE Phase 1 in Preparation for COMLEX-USA Level 1” [2]. We agree that the osteopathic profession must work together to help qualified candidates succeed and progress in their pursuit of osteopathic licensure and board certification. As always, we need to be careful about how assessment scores are used for secondary purposes and to limit these uses by exercising appropriate caution. The Comprehensive Medical Self Assessment Examination (COMSAE) is designed for the purpose of self-assessment and as a formative tool for candidates to gain experience with the types of test questions, content, format, and timing for COMLEX-USA examinations. The National Board of Osteopathic Medical Examiners (NBOME) cautions users against other uses or inferences based solely on COMSAE data.

The NBOME developed COMSAE and other formative assessment materials, such as WelCOM [3], in response to student requests for additional examination preparation materials in the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) style, beyond those used as part of the osteopathic medical college curriculum. With numerous studies [4], [5], [6], [7] citing correlation between performance on COMLEX-USA and performance in the curricular program at an osteopathic medical school, NBOME advises that full engagement in the curricular program leading to a DO degree is the best overall preparation strategy for COMLEX-USA. Unfortunately, the articles cited in Sefcik and Petsche’s letter [1] are limited in their scope, making it difficult to draw any generalizable conclusion. For example, the data from Jackson et al. [8] comes from a single school and is based on a self-reported survey of only 102 (from 171 solicited) student respondents from a single class year in which only 58 students self-selected to take United States Medical Licensing Examination (USMLE). The fact that this group of students performed better on COMLEX-USA than the larger group who did not take USMLE may have to do with their underlying ability and motivation to secure a certain residency position or specialty, not the additional effort taken to study for two separate licensure examinations.

At the end of their letter [1], Sefcik and Petsche highlight the need to better understand five issues concerning the performance of osteopathic medical students on medical licensing examinations. While some of these issues go well beyond the scope of the Wang et al. study [2], additional discussion is warranted.

1. We agree that utilizing tools to help at risk candidates succeed on their licensure examinations and in their growth mindset for their future careers is a laudable goal. But it is the combination of many tools rather than the reliance on just one which should be explored. For example, NBOME has developed the Comprehensive Osteopathic Medical Achievement Tests (COMAT), a series of nationally standardized subject exams to assess osteopathic medical students’ knowledge and ability in core osteopathic medical and foundational biomedical sciences principles. Examinations such as these will help identify specific strengths and weaknesses in certain subject areas, as will performance on the Colleges of Osteopathic Medicine (COMs) curricular assessments. They also provide experience with COMLEX-USA test item style and other features such as test timing. The COMSAE Phase 1 can help predict future performance on COMLEX-USA Level 1 but, other than indicating that a student is likely to pass or fail, it cannot sufficiently be relied upon to help students allocate their study time nor the amount of study time that should be allocated to a specific subject area. Performance in other assessments (e.g., COMAT, course assessments, COM performance) where more robust discipline-specific feedback is available will certainly be more useful for this purpose [9].

2. It would be interesting to know whether time spent focusing on specific content in weaker disciplines leads to higher scores. However, this will require much more detailed information on student study habits, including their real (or perceived) discipline-based strengths and weaknesses. Student surveys, combined with a data sharing agreement between the NBOME and the COMs, would be needed to answer this question.

3. It is reasonable to postulate that preparation for a second medical licensing exam (e.g., USMLE Step 1) will improve performance on COMLEX-USA Level 1. While these two examinations have unique elements and designs, they do measure some overlapping constructs. This is one variable which addresses the relationship between total study time and examination performance. Perhaps students who dedicated double the time to prepare for COMLEX-USA Level 1 would do just as well as those who split their time studying for USMLE Step 1 and COMLEX-USA Level 1. Once again, more detailed information of study habits is needed to answer the question and further research is warranted.

4. Neither COMSAE nor COMLEX-USA Level 1 were designed to help faculty evaluate and improve instructional and assessment activities at their schools. While some information based on aggregate student performance could be helpful for this purpose, the primary purpose of these assessments is to provide students with a reasonably precise ability estimate, one that can be used for assessing readiness and preparing for licensure examinations (e.g., COMSAE) or for making reliable and defensible competency decisions in the pathway for licensure (e.g., COMLEX-USA Level 1). As a service to the COMs, the NBOME does provide aggregate performance data on an annual basis based on exam blueprint dimensions. While it may be possible to provide more detailed feedback, it is likely to be of questionable reliability, thus negating its value for identifying areas for curricular program improvement.

5. It is not clear what information residency program directors will substitute for COMLEX-USA Level 1 numeric scores after that change is made in 2022. It may well be that program directors utilize COMLEX-USA Level 2-CE scores for making interviewing decisions. The NBOME advocates for holistic review of applicants and anticipates that numerous other factors will be used. Regardless of what happens when both COMLEX-USA Level 1 and USMLE Step 1 stop reporting numeric scores, it is clear that the selection process (and the information used to make these decisions) has numerous opportunities for improvement. The Coalition for Physician Accountability’s Undergraduate Medical Education (UME) to Graduate Medical Education (GME) Review Committee has proposed changes to improve this process, reduce bias, and enhance diversity, equity, and inclusion [10]. The NBOME believes that if a program director chooses to use any exam or scores as part of a holistic review of a DO applicant’s qualifications, COMLEX-USA is the licensure assessment best aligned with their curricular program and, therefore, should be the one that is used.

The answers to the questions posed by Sefcik and Petsche will require continued collaboration between the NBOME, the COMS, and various other organizations involved in the education, licensure, and certification of osteopathic students, graduates, and practicing physicians. The NBOME appreciates the contributions of all our colleagues who work to help to continuously improve the value of our assessments and provide information to learners and other stakeholders that is meaningful. Ultimately, our patients will benefit.