Pass/Fail Patterns of Candidates Who Failed COMLEX-USA Level 2-PE Because of Misrepresentation of Clinical Findings on Postencounter Notes

The 1999 Institute of Medicine report To Err Is Human: Building a Safer Health Care System emphasized the need for healthcare professionals to address patient safety. The report estimated that 44,000 to 98,000 deaths per year in the United States were due to preventable medical errors.¹ Negative patient outcomes may result from errors in diagnosis, treatment, monitoring, and communication.^1-3 In particular, documentation errors have been identified as contributing to a substantial number of preventable negative patient outcomes.^4,5

Errors in documentation have not only been reported in the patient-care literature but also in medical education.^6-10 Errors include underdocumentation (errors of omission), overdocumentation (errors of commission), or inaccurate documentation (errors in fact or recall).^6,7 Errors of commission are of particular concern not only in the assessment of practitioner professionalism, but also with respect to patient safety: Documenting information in a patient chart that was not obtained during a clinical encounter may lead to diagnostic or treatment errors. This type of overdocumentation has been considered unprofessional, careless, harmful, and dangerous.^6,7,10

In 2007, the National Board of Osteopathic Medical Examiners (NBOME) implemented a policy to address the accuracy and integrity of information documented in written postencounter notes (SOAP notes) that states the following:

Misrepresentation of clinical findings of the medical record (written SOAP Notes) in COMLEX-USA Level 2-PE by documenting medical history that was not elicited, or physical examination maneuvers or techniques that were not performed, is considered irregular conduct and will be thoroughly investigated and dealt with as specified in NBOME's Bulletin of Information.^6,11,12

This policy was instituted not only to protect the integrity of the examination, but also to raise awareness that overdocumentation in clinical practice may jeopardize patient safety.

Sandella et al⁶ describe the processes for identifying candidates who may have misrepresented elements of their SOAP notes during the examination. These processes involve the identification of discrepancies in candidates' documentation by trained physician examiners who score SOAP notes, as well as the identification of scoring disparities between data collection and SOAP note scores. When a candidate is identified by one of these screening methods, the contents of the SOAP notes are compared to video recordings of the standardized patient encounters. These videos and SOAP notes are reviewed first by physician staff of the NBOME and again by an external panel of reviewers. Following these reviews, external reviewers decide whether the errors in the written notes represent a consistent pattern of misrepresentation in the candidate's notes. If the external panel of reviewers deem that such a pattern is present, the candidate with discrepant notes will receive a failing score report with an annotation of “Irregular Conduct.”

The purpose of the present study was to review the pass/fail patterns of candidates affected by the implementation of this policy. Of those who failed because of misrepresentation of clinical findings on the written postencounter notes, we will investigate each candidate's status as a “first-time candidate” or “repeat candidate” in the examination. Secondly, we will examine how these candidates fared on subsequent attempts to pass the COMLEX-USA Level 2-PE.

Methods

Historical records from NBOME's COMLEX-USA Level 2-PE database were accessed to analyze the pass/fail patterns of candidates who failed the examination because of misrepresentation of clinical findings for the 2007-2008, 2008-2009, and 2009-2010 testing cycles. When candidates take the COMLEX-USA Level 2-PE, they give permission for their deidentified data to be used for research purposes. Because no individual scores or outcomes are reported, institutional review board approval was not required.

Results

Throughout the 2007-2008, 2008-2009, and 2009-2010 testing cycles, 12,510 candidates (first-time and repeat) took the COMLEX-USA Level 2-PE (Table 1). Of the sample, 24 (0.2%) failed the examination because of misrepresentation of clinical findings on postencounter SOAP notes.

Overall first-time pass rates for the 2007-2008, 2008-2009, and 2009-2010 testing cycles were 93.5%, 94.7%, and 94.8%, respectively (Table 2). Previous and subsequent examination pass/fail results for the 24 candidates who failed because of misrepresentation of clinical findings are presented in Table 3.

Of the 24 candidates, 20 were first-time examinees, 2 were on their second take (1 failed the Biomedical/Biomechanical Domain and 1 failed the Humanistic Domain on first attempt), and 2 were taking the COMLEX-USA Level 2-PE a third time (1 failed both domains on both first and second attempts, and 1 failed one of each domain on each previous attempt).

After failing COMLEX-USA Level 2-PE because of misrepresentation of clinical findings during their first examination attempt, 19 of 20 candidates passed on their second attempt. The remaining candidate failed the Humanistic Domain on a second attempt and passed the examination on the third attempt. Of the 4 candidates who failed the examination because of misrepresentation on the second or third attempt, 2 went on to pass in subsequent administrations, and 2 had not scheduled subsequent examinations at the time of data collection (Table 3).

Of the 24 candidates, 22 ultimately passed the examination at the time data were obtained (August 2010), none failed twice because of misrepresentation, and 2 candidates had not yet scheduled retake examinations.

Comment

Failing the COMLEX-USA Level 2-PE because of misrepresentation of clinical findings is a rare occurrence: Only 24 of 12,510 candidates failed the examination (0.2% failure rate) for this reason during the 2007-2008, 2008-2009, and 2009-2010 testing cycles. Of these 24 candidates, most failed on their first attempt because of misrepresentation of clinical findings, and only 4 (16.7%) failed because of misrepresentation of clinical findings during subsequent examination attempts. It is therefore unlikely that candidates intentionally misrepresent clinical findings after an unsuccessful previous attempt in order to improve examination scores.

Of those candidates who had previously failed the examination in 1 of the 2 performance domains prior to failing because of misrepresentation of clinical findings, there was no clear pattern of prior performance. Students failed both the Humanistic Domain and the Biomedical/Biomechanical Domain with equal frequency. Intentionally misrepresenting SOAP notes would theoretically provide an advantage only to those scores within the Biomedical/Biomechanical Domain. Therefore, one might again draw the conclusion that this misrepresentation is not necessarily the result of a previously unsuccessful candidate intentionally “gaming” the examination, particularly for a candidate trying to compensate for previous poor performance in the Humanistic Domain.

Most candidates in our sample went on to successfully complete COMLEX-USA Level 2-PE on their next subsequent attempt. Of the 20 candidates who failed because of misrepresentation on the first attempt, 19 passed on their next attempt (95% retake pass rate). Therefore, the candidates who failed their first attempt because of misrepresentation demonstrated higher pass rates on second attempt (95%) than all candidates who failed on their first attempt (87.1% pass rate for candidates on their second-attempts overall).¹¹

Although the number of candidates misrepresenting clinical findings on SOAP notes appears small, overdocumention in postencounter notes is a disturbing act that represents careless and unprofessional behavior. However, is it considered cheating? Cheating is defined as “depriving something valuable by the use of deceit or fraud” or “influencing or leading by deceit, trick, or artifice.”¹³ Implied in the use of the term “cheating” is intent, and if intent cannot be established, then the behavior cannot be considered cheating with certainty. In a previous study including respondents from 120 medical schools in the United States and Canada, Stimmel et al¹⁴ reported 181 formal allegations of cheating with 30 (16.6%) of those representing cheating on the National Board Examinations. Because Stimmel et al predates the incorporation of clinical skills testing for high-stakes licensing examinations,¹⁵ the analysis excludes issues of overdocumentation on postencounter notes.

Under the Health Insurance Portability and Accountability Act of 1996, fraud is defined as knowingly and willfully executing a scheme to defraud; this is contrasted with abuse, which is unintentional practice.^16,17 Therefore, as is the case with cheating, intent is used to distinguish fraud from abuse. Sandella et al⁶ reported that medical students may overdocument, particularly for high-stakes clinical skills examinations, for a variety of reasons: feeling pressure to pass the examination, presuming that standardized patients are unlikely to have positive findings on examination, failing to recognize overdocumentation as inappropriate because the behavior was not identified during previous standardized patient-based exercises at school, and assuming the consequences will not have negative patient outcomes because the testing environment is artificial and the patients are not “real.”⁶

Other investigators have speculated that overdocumention may be due to recall bias, misunderstanding of physical examination, misuse of acronyms or abbreviations, inference of findings of a clinical case, and intentional reporting of findings after realizing they neglected to ask questions or perform maneuvers pertinent to the case.⁹ However, without understanding or knowing the intent of candidates when writing post encounter notes that contain information that was not obtained during the clinical encounter, we cannot consider this behavior as cheating with certainty. The fact of the matter is that the recorded information may not be accurate, the information was not obtained during the patient encounter, and this overdocumentation may lead to serious negative patient outcomes if continued in clinical practice. Identifying candidates who misrepresent clinical findings in their documentation is therefore not necessarily a mechanism to recognize cheating, but rather to identify a practice that may compromise patient safety.

The consequence for a candidate who has demonstrated a consistent pattern of misrepresentation on his or her postencounter notes is severe: a failure on the COMLEX-USA Level 2-PE. This outcome results in preserving examination scoring validity by not scoring notes that are inaccurately formulated, as well as raising awareness of the importance of documentation accuracy throughout medical education and beyond. Consistent with NBOME's mission to protect the public, identifying overdocumentation practices that may lead to negative patient outcomes is an important step toward reducing medical errors.

Conclusion

Among candidates found to have misrepresented clinical findings on postencounter written documentation on the COMLEX-USA Level 2-PE, no clear pass/fail pattern was identified in their past or subsequent examination performance, and the majority of these candidates passed on subsequent examination attempts. The NBOME will continue to monitor the accuracy and integrity of contents included in written postencounter documentation to continue to fulfill its mission to protect the public.