Is language an issue? Accuracy of the German computerized diagnostic decision support system ISABEL and cross-validation with the English counterpart

Introduction

Getting the right diagnosis is a key aspect of health care. It provides an explanation of a patient’s health problem and informs subsequent health care and treatment. Misdiagnosis occurs in about 5–10 % of emergency room (ED) patients, sometimes with devastating medical and economic consequences [1], [2], [3]. The causes for diagnostic error can be diverse, but one major cause is human error. This includes the consideration of incomplete patient histories, failure to consider alternatives, lack of knowledge and lack of recognition of clinical findings by physicians [4], [5], [6], [7], [8]. Moreover, diagnostic hypothesis generation in the ED is often based on physician intuition and experience [9]. Because an accurate diagnosis is the basis for all treatment and care, improving the diagnostic process and accuracy is key to improving patient safety and outcomes.

Because healthcare is becoming increasingly digitalized, one possibility to reduce human error in the diagnostic process is the use of computerized diagnostic decision support systems (CDDS). Such programs can prompt physicians to ask relevant questions, can close knowledge gaps and suggest diagnostic steps and differential diagnoses (DDx) and thus have potential to improve the diagnoses and outcomes of patients [10, 11]. CDDS collect data on patient characteristics, suggest potential causes, and propose next steps in the diagnostic workup. While CDDS originally only supported physicians (Dr-CDDS), many today offer an interface to be used by patients (Pat-CDDS), and propose DDx and next steps to patients directly, such as doctor visits or self-care. In a recent review and meta-analysis of CDDS providing DDx, the best performing system was ISABEL, which was associated with the highest rates of accurate diagnosis retrieval compared to all other types of CDDS tools (pooled rate=0.89; 95 % CI=0.83–0.94) [11]. It should be noted, however, that the vast majority of studies evaluating CDDS used vignette cases (i.e. cases that authors constructed) as opposed to real-world cases (ED cases).

ISABEL Healthcare Ltd (UK) has recently developed a German version of both the Dr-CDDS and the Pat-CDDS. Neither of these have been validated so far. In contrast to other CDDS using chatbots and rule-based algorithms to generate DDx, ISABEL is based on natural language processing. More specifically, the users enter symptoms as free text either in medical terminology when using the Dr-CDDS (Isabel Pro) or in lay terminology when using the Pat-CDDS (Isabel Symptom Checker). These terms are then compared to a reference library using a search algorithm and matched with natural language processing to those DDx with the highest degree of matching symptoms. Each language version of Isabel is based on the same database, which is maintained in English. Each query runs first through a professionally translated synonym file. The database search is then expanded with all matched synonyms (e.g. “upper abdominal pain” with “upper stomach pain”). German terms not found in the synonym file are translated to English by the Google API and matched again with the synonym file. If a term or phrase cannot be found in Isabel’s synonym file, an NLP algorithm first removes ‘stop words’ such as ‘and’, ‘is’, ‘no’ and tries to match the remaining text with the synonym file. If still no match is found, the NLP algorithm searches the synonym file by using combinations of 2 or 3 words and lastly by searching single words. The natural language processing approach employed by ISABEL might be more vulnerable to errors in translations compared to systems using rule based algorithms (where there usually is one storyline for each of the predictable user request). Indeed, users can enter terms in many ways and combinations, which highlights the importance of taking language issues into consideration. Hence, it is critical to investigate the performance of the German versions of ISABEL. Consequently, we aimed to assess the accuracy of diagnoses suggested by the German version of Isabel Pro and the Isabel symptom checker on typical ED vignettes and real ED cases. In addition, we aimed to compare the provided DDx from the German and English Isabel Pro on the same cases for direct comparison between language versions. Further, we aimed to explore the potential of the Dr-CDDS to increase diagnostic quality, namely to evaluate whether the Dr-CDDS provides the correct diagnosis for a sample of ED patients who initially received a wrong ED discharge diagnosis.

Subjects and methods

Subjects

Vignette cases

The clinical vignettes were derived from a German text book containing 100 cases occurring in Emergency Medicine including cases from various medical disciplines such as cardiology, pulmonology, gastroenterology and others [12]. Of those 100 cases, 50 were purposefully selected by two independent investigators (AL, RG) in order to sample each discipline. Selection criteria were the presence of a correct diagnosis in the book and information regarding medical history and physical examination. Cases with an unspecific final diagnosis were excluded as well as resuscitation, trauma or psychiatric emergency cases, because the need for and value of CDDS for such cases is considered low.

Real ED cases

Real ED data were used from hospitalized patients who participated in a recent prospective observational study on diagnostic errors conducted in the ED of the university hospital of Bern [13, 14]. That former study assessed discrepancies between the ED diagnosis at hospital admission and the hospital discharge diagnosis, whereas the latter was considered as the correct diagnosis. To assure that the final diagnosis can be detected given the symptoms at ED presentation, only patient cases with a confirmed ED diagnosis (i.e. no discrepancy between ED and hospital discharge diagnosis) were considered eligible for the validation analysis. Patients with a diagnostic discrepancy were considered for analysing Isabel’s diagnostic potential. The original study excluded patients admitted to any internal medicine ward for reasons of age, comorbidities, palliative care, social reasons, surgical ward crowding and patients with an acute traumatic injury. For the present study, patients were further excluded in case of ED revisit, withdrawal of the hospital’s general informed consent, triaged with an acute life-threatening condition, missing information in EHR history and unspecific discharge diagnosis indicated by an ICD-10 code starting with R from Chapter 18 – Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified within ICD-10, because we did not expect ISABEL to provide unspecific DDx. In addition, cases were included where the hospital discharge diagnosis was neither completely identical nor different from the ED discharge diagnosis. From the eligible cases, 50 cases were randomly selected for the validation analysis and 20 cases for the diagnostic potential analysis using the statistical software R. The full flow chart is shown in Figure 1.

Figure 1:

Flowchart.

The study was approved by the Ethics Committee of the Canton of Bern, Switzerland (ID 2021-01449).

Results

Overall, 100 cases were used to analyse the accuracy for both, the German Dr-CDDS and the Pat-CDDS. Age of the included cases ranged from 12 to 90 years with a mean (standard deviation; SD) of 54.4 (19.5) years. Half of all discharge diagnoses could be categorized into diseases of the circulatory, digestive or respiratory system (Table 1).

Discussion

We aimed to assess the accuracy of differential diagnoses suggested by the German version of Isabel Pro and Isabel symptom checker on vignettes and ED cases. Our analysis shows robust results for both, vignettes and real ED cases.

Overall, we found an accuracy comparable to previous studies on DDx generators. A recent systematic review found an overall rate of accurate diagnosis retrieval of 0.70 (95 % CI 0.63–0.77). Isabel Pro was associated with the highest rate of accurate diagnoses among all of the investigated tools with a pooled rate of 0.89 (95 % CI 0.83–0.94), although a high heterogeneity was found between the 7 included studies [11]. One of the studies included queried Isabel Pro with data from 594 patients presenting to the ED and found a substantially greater accurate diagnosis retrieval rate of 95 % overall and 78 % in top 10 than we did [15]. The higher accuracy may be explained by the more strictly controlled data collection and extraction in the prospective trial compared to our retrospective data analysis. However, as the authors did not provide details regarding discharge diagnoses, we may only speculate about this discrepancy.

Interestingly, we did not find differences in accuracy between the German and English language version, when sending the same (translated) queries to the Dr-CDDS. However, the provided DDX list did substantially differ between the language versions. Therefore, we assume that the algorithms in place are somewhat susceptible to minor changes in text entries.

Case vignettes, which are widely used to test Dr-CDDS, are often prototypical and illustrative. They may thus not be the best choice to test CDDS since ED case scenarios are hardly ever prototypical. Using both, ED cases and vignettes, we saw differences in the results of the DDx lists. The accuracy tended to be higher for vignette cases and if the correct diagnosis was retrieved, it was ranked substantially higher compared to ED cases. Therefore, we emphasize to use real-world cases for validation studies.

The authors of the aforementioned systematic review already pointed out the problem of lengthy DDx lists provided by CDDS. Although having an increased likelihood of retrieving the correct diagnosis, the value to clinicians may decrease, especially in a busy ED setting. Therefore, the importance of the rank where the correct diagnosis is listed should not be neglected.

Regarding the accuracy of the Pat-CDDS, Semigran et al. measured accuracy of self-diagnosis and triage advice provided by 23 symptom checkers by using 45 standardized patient vignettes. On average, the correct diagnosis was listed in 58 % of the cases among the first 20 DDx across all investigated CDDS. In this study, Isabel symptom checker provided the correct diagnosis in 69 % of the cases within the top 20 and thus performed better than in our study with 58 % [16].

In contrast to simulated vignette cases where usually a specific diagnosis can be defined, diagnoses in the real world are often rather vague and not definite, especially in emergency medicine. As we cannot expect a CDDS to come up with unspecific diagnoses, we excluded all diagnoses from analyses that had unspecific ICD-10 codes. Nevertheless, 10 patients included in this validation study still had some rather unspecific discharge diagnosis such as unspecified infectious disease, lower back pain or disorder of muscle (unspecified). For six of these 10 patients, Isabel could not provide any DDx that could have been assigned to the discharge diagnosis. Specific diagnoses that were not identified by the Dr-CDDS included colon diverticulum, immune thrombocytopenia or gastric perforation. The full list of failures including Isabel query and provided DDx list is shown in Supplementary Table 1. The full list with correctly identified diagnoses is provided in Supplementary Table 2 (in German).

The accuracy of differential diagnoses provided by the CDDS seems to differ regarding organs, respectively ICD disease category. Accuracy was higher for circulatory and endocrine conditions than for musculoskeletal and respiratory conditions. However, the sample size was rather low for many of the ICD-10 disease chapters.

Querying symptoms and clinical features available from the ED electronic health record system for real ED cases with known diagnostic errors (i.e. discrepancy between ED diagnosis and hospital discharge diagnosis), the Dr-CDDS provided the ‘correct’ hospital discharge diagnosis in half of cases. Of course, it can only be speculated whether the DDx list would finally have improved the diagnosis in ED. However, it is likely that the Dr-CDDS would have at least broadened the DDx, as none of the ED diagnoses included the final hospital discharge diagnosis as differential diagnosis.

Conclusions

We have found a somewhat lower accuracy using the German Version of Isabel Pro compared to previous studies on the English language version. Translating the German queries and querying the English version did not lead to improved accuracy, but a substantial part of the provided DDX differed between the language versions. CDDS have the potential to guide clinicians to the right diagnosis. However, further studies are needed to evaluate the effects when such tools are applied in a real world clinical setting (for example concerning an increase in diagnostic tests or increased length-of-stay due to additional investigations).