Radiologic errors, past, present and future

The past

Nearly 65 years ago a California radiologist named L. Henry Garland shocked the medical community with publication in a radiology journal his article entitled, “On the Scientific Evaluation of Diagnostic Procedures” [2]. Summarizing investigations that revealed a “surprising” degree of inaccuracy in many non-radiologic clinical and laboratory as well as radiological tests in that article and several others that were published soon thereafter [3–5], Garland enumerated studies that found a 34% error rate in the diagnosis of myocardial infarction, an only 15% agreement among eight experienced internists in determining the presence of “the most simple signs” of emphysema when examining the chests of patients afflicted with that disease, a marked disparity in clinical evaluation of 1000 school children of the indications for tonsillectomy, an agreement rate of only 7% among five experienced pediatricians determining clinically whether children were suffering from malnutrition, a 20% error rate in the interpretation of electrocardiograms, a 28% error rate among 59 different hospital clinical laboratories in reporting the results of chemical analyses, and a 28% error rate among clinical laboratories in measuring the erythrocyte count.

Most of Garland’s attention, however, was focused on radiologic errors. He found that experienced radiologists missed 30% of chest radiographs positive for radiologic evidence of disease, overread 2% of them that were actually negative for disease, and disagreed with themselves 20% of the time [5]. Garland commented:

Many clinicians continue to believe that their observations are accurate, and are unaware of the need … to reduce error. They feel … that Roentgen tests may be subject to faulty interpretation, but not careful “observation.” Not only should clinicians recognize their own errors; they should admit them.

Garland went on to relate that when one of his friends, a well-known professor of radiology, learned that Garland’s research disclosed that radiologists missed about one-third of roentgenologically positive films, the friend expressed the hope that Garland would discontinue his “investigations in this field because they were so morale-disturbing.” When other radiologists were confronted with this data, continued Garland, their usual reaction was, “Well, in my everyday work, this does not apply; I would do better than those busy investigators.” How wrong they were.

The present: error rates confirmed

Garland’s revelations about the incidence of errors in clinical medicine have been confirmed and expanded upon by subsequent researchers. Agreement among academic faculty physicians performing physical examination for spleen enlargement [6], liver enlargement [7], abdominal ascites [8], acute otitis media [9], and other assorted physical findings [10–12], has shown to be remarkably poor. Large autopsy studies have disclosed frequent clinical errors and misdiagnoses, with error rates as high as 47% [13, 14]. Errors ranging from 25% to 49% in pathologists’ interpretations of biopsy specimens, and a 24% error rate in laboratory results have also been reported [15, 16].

In the decades following Garland’s classic articles, a number of investigators replicated Garland’s findings relative to radiologic interpretations [17–26]. In a 1976 study at the University of Missouri, an error rate of 30% was reported among staff radiologists in their interpretation of chest radiographs, bone exams, gastrointestinal series, and special procedures [27]. Elsewhere researchers found that as many as 20% of colonic tumors were missed on lower gastrointestinal examinations [28]. Harvard University researchers [29] reported that radiologists disagreed on the interpretation of chest radiographs as much as 56% of the time. Additional studies conducted by researchers at major academic medical centers disclosed that from 26% to 90% of all lung carcinomas were missed by radiologists interpreting plain chest radiographs [30–32].

Numerous reports also documented similarly high error rates among the more recent “high-tech” modalities utilized in radiologic practice, such as sonography [33, 34], arteriography [35], MR angiography [36], MRI when evaluating lumbar disk herniation [37], MR when evaluating rotator cuff injury [38], MR when evaluating prostatic cancer [39], and radionuclide scans [40]. Similar error rates were also reported with chest CT scans harboring lung cancer [41, 42].

In a 2010 study in which three experienced radiologists who specialize in abdominal imaging initially reviewed 90 abdominal and pelvic CT examinations and then at a later date while blinded to previous interpretations were asked to reinterpret exams that had been read not only by themselves but by their colleagues, the interobserver discrepancy rate was 26%, the intraobserver discrepancy rate 32% [43] – almost identical to Garland’s data 60 years earlier. Although these figures today are readily acknowledged, in the 1950s they were indeed astonishing to all radiologists.

Still other studies have confirmed a 35% error rate among radiologists interpreting radiologic studies obtained in patients who had undergone trauma [44–46]. Statistics disclosing inaccuracies in the interpretation of mammograms are startling [47–54]. A report from Yale University School of Medicine found that upon retrospective review of mammograms originally interpreted by experienced radiologists as normal, from 15% to 63% of breast carcinomas had been overlooked at initial readings [55]. A University of Arizona study found that in 75% of mammograms initially interpreted as normal, breast carcinomas could be seen on retrospective evaluation [56].

Retrospective experimental vs. “real-time” error rates

As explained by Garland, error rates can be calculated in two different ways, depending on the denominator used [5]:

If a series of 100 roentgenograms contains 10 positive and 90 negative films, and a reader misses three of the positive films and over reads two of the negative films, he may be regarded as having only a 5% error. On the other hand, since the series of 100 roentgenograms is being examined to detect patients with disease, the reader who misses three of the ten positive films has an error rate of 30%. Coupled with an over reading of two of the ninety negative films, the combined error rate in the example mentioned is about 32%.

In virtually all of the studies of error rates to which this article has thus far referred, the denominator consists of a preselected number of abnormal radiologic studies. Thus, if a radiologist participating in a research project is given 100 radiographs known to be abnormal and misses 30 of them, the error rate is obviously 30%. This should not be construed as indicating that radiologists commit an average 30% error rate in their everyday practices. Several studies have measured “real-time” error rates of radiologists by determining how many errors were committed among a large number of radiologic exams interpreted by a radiologist in a practice situation over a selected period of time – in other words, with a denominator including both normal and abnormal exams.

University of Texas researchers [57] reviewed imaging interpretations rendered in radiology departments of six community hospitals and found a 4.4% mean rate of interpretation error. Other researchers [58] reviewed the performance of more than 250 radiologists who had interpreted more than 20,000 examinations as part of clinical testing of a performance improvement product, Radpeer. They found an all-case error rate of 3% to 3.5%. Still another group of researchers [59] reviewed the results of a quality improvement study conducted among 26 radiologists who read 6703 cases, and found an overall error rate of 3.48%. To summarize, if the denominator consists only of radiology studies that harbor abnormalities, the error rate averages 30%; if the denominator consists of an “everyday” mixture of abnormal and normal cases as is usually found in daily practices, the error rate averages 3.5%–4%.

It should be emphasized that none of the studies referred to in this article reflect the degree to which patient care is injured or otherwise jeopardized because of reader misinterpretation. “Extrapolation of reader error to medical care is complex” [29]. Although some radiologic errors may indeed result in serious injury and/or mismanagement of a patient, most are either corrected quickly or, fortunately, not clinically important, and thus exert no adverse effect on the health or management of the patient.

Causes of radiological errors: perceptual

Diagnostic errors in radiology may be perceptual or cognitive. Although it is not known exactly what percentage of diagnostic errors in radiology are due to perceptual misses, it has been estimated to be in the 60%–70% range [17, 23].

The failure to detect a radiologic abnormality is often attributed to the subtlety of the radiologic finding, or poor conspicuity, a term that is defined as the ratio between the contrast enhancement of the lesion relative to the surrounding tissues. While this definition may adequately explain how a truly subtle lesion can be missed, it is woefully inadequate to explain how an obvious abnormality can be missed. The phenomenon of simply not “seeing” initially an abnormality that is easily and clearly seen on a second look has never been explained to anyone’s satisfaction. Referred to as the “human factor” [2, 60], as the “foibles of human perception” [24], as an “irreducible necessary fallibility emanating from uncertainties inherent in medical predictions based on human observation and the laws of natural science” [13], and as a constant “inherent in all human activity” by Leape [61], the missing of an overt lesion remains as much a mystery and enigma today as it was 61 years ago.

Probably no event is more perplexing or frustrating to radiologists than the realization that they have committed a perceptual error – that they did not “see” on a radiologic imaging exam an abnormality that later is plainly evident. Despite the voluminous material that has been published in the radiologic literature on the subject of perceptual misses, we still do not know exactly why we miss obvious radiographic findings. We still do not know the answer to the question that all too often the erring radiologist asks in exasperation; “How and why did I not see that abnormality?”

One pioneering researcher in radiologic perception commented [60]:

So long as human beings are responsible for Roentgen interpretation, the process will be subject to the variability of human perception and “reader-error” due to the interpreter’s failure to perceive critical detail. The processes governing search behavior and mediating visual perceptual are correspondingly complicated, and our knowledge of them is fragmentary. Enough is known, however, to suggest that errors of perception are for the most not the result of carelessness or willful bias on the part of the radiologist, but rather a consequence of the physiologic processes of perception. Errors of perception are an unavoidable hazard of the “human condition.”

A British radiologist-researcher observed, “Although technology has made enormous progress in the last century, there is no evidence for similar improvement in the performance of the human eye and brain” [62].

Author Malcolm Gladwell in an article published in The New Yorker [63] made the following insightful observation regarding perceptual errors in radiologic interpretation:

The reason a radiologist is required to assume that the overwhelming number of ambiguous things are normal is that the overwhelming number of ambiguous things really are normal. Radiologists are, in a sense, a lot like baggage screeners at airports. The chances are that the dark mass in the middle of the suitcase isn’t a bomb, because you’ve seen a thousand dark masses like it in suitcases before, and none of those were bombs – and if you flag every suitcase with something ambiguous in it, no one would ever make his flight. But that doesn’t mean, of course, that it isn’t a bomb. All you have to go on is what it looks like on the X-ray screen – and the screen seldom gives you quite enough information.

Cognitive errors: alliterative and satisfaction of search

Whereas up to 70% of diagnostic radiologic errors are perceptual in nature – failing to “see” something on the radiologic image – the remainder is due to cognitive errors or errors in judgement – i.e., attaching the wrong significance to a finding that is seen. It is impossible to delve into the minds of radiologists who have rendered erroneous conclusions, but one possible explanation referred to as “faulty reasoning” [17], is the radiologists’ failure to think of possibilities when interpreting radiographs: radiologists simultaneously combine perception of an abnormality with the notion of it, and the notion is often so strong that other features or information that might have modified the decision are rejected. Thus, “more things are missed through not being thought of, and so not looked for, than through not being known.”

Alliteration is defined as the occurrence in a phrase, or line of speech or writing, of two or more words having the same initial sound [64]. In the context of radiologic errors, the alliterative error results from the influence that one radiologist exerts on another: if one radiologist fails to detect an abnormality or attaches the wrong significance to an abnormality that is easily perceived, the chance that a subsequent radiologist will repeat the same error is increased. Alliterative errors occur because radiologists read the reports of previous examinations before or while reviewing the newly obtained radiologic studies and therefore are more apt to adopt the same opinion as that rendered previously by a colleague (or oneself) [65]. To what extent radiologists repeat the same errors as those committed by predecessor radiologists is not known with certainty, but they are not uncommon.

“Satisfaction of search” refers to the fact that the detection of one radiologic abnormality may interfere with the detection of additional abnormalities in the same examination. In other words, when viewing radiologic studies, there is a tendency to become “satisfied” after identifying the first abnormality that leads to a failure to search for additional findings [66]. In a study in which radiologists were shown in random order a number of exams in which one abnormality was present and the same number of exams in which two or three abnormalities were present, 75% of the abnormalities were reported when the examination contained one or two abnormalities. In examinations that contained three or more abnormalities however, only 41% were detected [67].

Computed assisted detection (CAD)

Over the past two decades, a multitude of articles discussing new technologies has appeared in the radiologic literature. In 1998, the US Food and Drug Administration approved computed assisted detection (CAD) to assist radiologists in their interpretation of radiologic examinations. Since that time, CAD has been used primarily in mammography [68–70], where it has improved sensitivity by raising the level of the radiologist’s suspicion for breast cancer. CAD was initially shown to be of value in reducing radiologic errors and improving interpretation in mammography [71, 72]. Early studies disclosed a significant increase in breast cancer detection when CAD was utilized, but more recent studies have suggested that the value of CAD may have been overrated.

A study published in 2011 that examined records from 685,000 women who received more than l.6 million mammograms from 1998 through 2006 disclosed that although CAD was associated with a statistically significant decrease in specificity, there was only a non-statistically significant increase in sensitivity but no statistically significant improvement in cancer detection rates [73]. It was also found that CAD increased a woman’s risk of being recalled unnecessarily for further testing. An accompanying editorial concluded that millions of women are being exposed to “a technology that may be more harmful than it is beneficial” [74]. Thus far, CAD has not been shown to reduce radiologic error.

Radiologic errors: the future

Awareness and understanding of all medical errors – clinical and radiologic – have grown rapidly in the past decade and a half. Nevertheless, it has been estimated that up to 80,000 US hospital deaths occur due to misdiagnosis annually, 5% of autopsies reveal lethal diagnostic errors for which a correct diagnosis would have averted death, and physician errors resulting in adverse events were more likely to be diagnostic than drug related [75]. In 2007, the Agency for Healthcare Research and Quality (AHRQ) announced special emphasis on funding diagnostic errors research. Health information technology, improved education, and increasing acknowledgment of diagnostic errors hold promise in error reduction [76], although such efforts still remain a goal rather than a reality. Diagnostic accuracy remains low [77]. Cognitive errors in radiology have been reduced through continuing medical education and by providing more complete patient history and clinical findings to the interpreting radiologist. Reducing perceptual errors, however, remains a challenge. A lament 44 years ago [60] that “the ultimate solution to the problem of ‘reader error’ is not yet clear,” still remains true today.

Fifty-four years ago, Garland exhorted future radiologists to continue “attempts at elucidation and correction of the factors involved” in causation of radiological errors. Urged on by this charge, radiologist and non-radiologist researchers today still pursue this goal, and will undoubtedly continue to do so for many years to come.