Evaluating disease outbreaks with syndromic surveillance using medical student clinical rotation patient encounter logs

Methods

This study was reviewed and was approved by the VCOM institutional review board (#1172780, effective until 1/15/2021). We retrospectively analyzed International Classification of Diseases, 10th Revision (ICD-10) codes logged by medical students completing core clinical (US-based) and international rotations at VCOM-affiliated hospitals and clinics during the COVID-19 pandemic period. Data was accumulated throughout the Appalachian and Delta regions of the US and in developing communities across Central America; it had been entered into the CREDO system, in which ICD-10 codes were recorded in real-time for each patient encounter.²² The CREDO logging system collects only basic patient demographics (e.g., age range, sex); unique identifiers that conflict with the Health Insurance Portability and Accountability Act regulations are not captured by the system.

Data from November 1, 2019, through March 13, 2020 was gathered; on March 13, 2020, VCOM was closed because of the COVID-19 pandemic and students were no longer able to participate in clinical rotations. During the study period, approximately 900 third- and fourth-year medical students from VCOM on clinical rotations logged approximately 3,000 patient encounters per day from over 1,500 clinical sites. A CREDO database query was conducted for ICD-10 codes using WHO’s 2011 revised case definitions for ILI and severe acute respiratory infections (SARI); these codes are presented in Table 1 and included J00, J01, J06, J09, J10, J11, J12, J18, J20, J22, J40, R05.^23,24 Each patient interaction (encounter) can generate multiple ICD-10 codes; each diagnostic, procedure, and drug entry are independently tracked and counted.

We calculated the sum for each ICD-10 code reported each day, then entered the data into a Microsoft Excel spreadsheet (Appendix). The total daily syndrome count (the sum of all reported instances of all relevant ICD-10 codes per day) was also recorded (Table 2). A modified cumulative sum (CUSUM) algorithm was applied to the syndromic data to detect temporal aberrant events (Figure 1). The CUSUM algorithm is used in statistical quality control to monitor variance over time. The algorithm uses sequential analysis to display the cumulative sums of deviation from a target value. The CUSUM control chart provides an objective measure to monitor changes in disease activity and has been proven useful in monitoring respiratory syndromic data.^1,25,26

Figure 1:

Equation for syndrome mean and alert threshold using a modified cumulative sum technique.

The mean of daily syndrome counts from days three through nine before the day of interest (“moving mean”) was calculated. The alert threshold, or the upper limit, was calculated as the moving mean plus 2.5× the standard deviation during the same time period:

Total daily counts that exceeded this threshold represented unusual levels of disease and indicated a potential outbreak. A sample calculation for the alert threshold on interest day January 31, 2020 is shown in Table 3. In this example, the total daily count (501) exceeded the alert threshold (32.6951), indicating a notable change in disease activity. Total syndrome counts were also obtained for the 2018–2019 flu season using the same case definitions. Lastly, total daily counts for breast malignancy—a disease with a stable incidence during the specified time frame—were used as a control.

Results

Daily total counts that exceeded alert thresholds for ILI/SARI are depicted in red in Figure 2. The first significant alert would have occurred on January 6 (daily count 120/alert threshold 53), which was three days before the Chinese Center for Disease Control and Prevention linked the mysterious cluster of pneumonia cases to COVID-19,²⁷ nine days before the first confirmed case in the United States, and 24 days before WHO declared the novel virus a Public Health Emergency of International Concern (PHEIC).²⁸ These alerts were tracked along a timeline of critical moments in the COVID-19 pandemic (Figure 3).

Figure 2:

This figure shows total daily influenza-like illness (ILI) and severe acute respiratory infections (SARI) counts from November 1, 2019 through March 13, 2020 (blue), the cumulative sum (CUSUM) + 2.5 standard deviation (SD) alert threshold (broken line). Exceeded alert threshold dates are depicted in red. Breast malignancy acted as a control and is shown in black.

Figure 3:

A timeline of the coronavirus outbreak may be explained by the statistically significant increases in influenza-like illnessactivity, shown here as an exceeded alert threshold.

The second alert would have been triggered on January 31 (daily count 501/alert threshold 333), two days before the US suspended entry for people who had traveled to China²⁹ in the prior 14 and five days before a briefing on Capitol Hill where Senators urged administration officials to take the threat more seriously.³⁰ The third important alert would have been triggered for February 24 (daily count 281/alert threshold 280), five days prior to the first confirmed COVID related death in the United States,³¹ 16 days before WHO declared a pandemic,³² and 18 days before the US declared a national emergency.³³

Additional data from the 2018–2019 flu season was included for comparison, with an adjusted timeframe to overlay ILI/SARI activity for both seasons (Figure 4). The total monthly counts of the designated ICD-10 codes were higher in 2019–2020 compared with 2018–19 (17,738 vs. 6,598)

Figure 4:

Influenza-like illness (ILI) and severe acute respiratory infections (SARI) total daily counts comparing 2018–2019 flu season (black) to 2019–2020 season (blue). Data was accumulated from November 1 through March 13 for both seasons. Note, the diminished data from approximately December 20 to January 1 in each year is a consequence of students on holiday and thus not reporting patient encounters.

The CUSUM control chart provides an objective measure to monitor changes in disease activity and has been proven useful in monitoring respiratory syndromic data.³⁴ This method was effective in identifying statistically significant spikes in ILI activity. These alerts were overlaid along the pandemic timeline in Figure 3 as an indicator of the potential clinical significance and likelihood of a true outbreak. Further, the overall ILI activity level as illustrated in Figure 4 was significantly larger than the previous year (17,738 vs. 6,598).

Discussion

Diagnostic codes entered by medical students to document each patient encounter across all rotations as a standard component of their clinical training were used for surveillance in this study. For respiratory syndromes, prior studies have shown the superiority of diagnosis codes over chief complaint alone for surveillance.^5,35,36 A study⁵ by George Washington University compared surveillance between complaint and diagnostic codes and found despite a potential delay in detection using diagnostic codes, the additional information obtained is more reliable and beneficial than surveillance using chief complaint. Children’s Hospital of Boston also demonstrated that diagnosis was superior to chief complaint for respiratory illness in the pediatric population.³⁵ The use of diagnosis codes also demonstrates good sensitivity and specificity. A study by the University of Pittsburgh Center for Biomedical Informatics showed the sensitivity of detection of acute respiratory illness using ICD-9 coded diagnoses was 44% with a 97% specificity.³⁶

The results of our syndromic surveillance COVID-19 validation demonstration are consistent with recent reported studies.^37,38 In one study,³⁷ a group at University of California-Los Angeles Health System performed a retrospective observational EMR inpatient and outpatient chart review across three hospitals for patients who presented with a “cough” and for patients hospitalized with acute respiratory failure. The researchers found a statistically-significant excess in symptoms as early as December 22, 2019, and concluded these were most likely early COVID-19 infections. In another recent study, a group from the CDC and the American Red Cross, among others, tested blood bank samples for COVID-19 reactive antibodies.³⁸ They found samples with reactive antibodies from California, Oregon, and Washington as early as December 13–16, 2019.

Traditional surveillance methods rely on confirmed laboratory testing and physician reporting. This has important health implications, as traditional surveillance may delay the time to response in an evolving outbreak; for example, our results indicated a lag of nine days. Today, approximately 70% of emergency departments and 4,000 health care facilities across the nation transmit electronic health data to the CDC’s BioSense Platform daily. Therefore, this method can be used to monitor infectious disease activity across much larger populations enhancing detection of evolving disease outbreaks. Our results have potential instructional value for medical students, as they illustrate a unique approach that utilizes student generated data for entirely new purposes, as well as a relevant statistical approach for time-dependent surveillance data analysis. Given the annual influenza season and the potential for new and emerging infectious diseases to surface from anywhere on the planet and affect health in America, it is important for future clinicians to be familiar with approaches like the one illustrated here, because it is likely to be valuable for future epidemics and pandemics.

A limitation of this study is that the increase in ILI/SARI activity could have resulted from any number of possible infectious diseases, limiting our ability to conclude causation; however, it is consistent with the timeline of the COVID-19 pandemic and other similar reports.^37,38 Additionally, surveillance data is subject to confounding factors. Another limitation is the data is representative of the VCOM associated hospitals and clinics throughout Appalachia and Central America (Dominican Republic, Honduras, and El Salvador) and does not represent the entire US population. A strength of the study was that data was accumulated from thousands of patients at many hospitals/clinics. Another advantage was the accuracy of discharge diagnosis codes as compared with chief complaints as this eliminates variability and results in higher outbreak detection sensitivity. Finally, the study was quick and easy to conduct with few resources and can easily be reproduced and enhance collaboration among public health, academic investigators, and industry.

As a result of this study, we are installing a real-time alert for ILI into CREDO, so it can be added to those diseases that are currently monitored by VCOM faculty. This significant, high quality data is amenable to automated computational analysis for generating automated alerts for identifying and tracking the progress of future pandemics. There are a number of advantages to using CREDO data. First, it is real-time, as medical students are continuously logging their clinical training patient encounters. Second, data are collected over a very large number of distributed rural and urban sites, from large, urban metropolitan hospitals to small clinics and individual practices in socioeconomically disadvantaged areas, and thus provides a method for surveilling a large region of the US and the Caribbean area without the need to extract and merge data from a variety of EMR systems. Third, the data is uniformly collected by clinically trained professionals simultaneously across a plurality of indications, thus providing well matched controls. Last, the CREDO system is extensible and if used by most or all health care trainees (medical, physician assistants, nursing, pharmacy students), could ensure coverage across the entire US at even higher data rates, thus improving surveillance performance.

Conclusion

One of the tenets of public health is prevention, and we deduce that syndromic surveillance is timelier than traditional surveillance methods and can be used in conjunction with them to enhance outbreak detection. Earlier detection can lead to more efficient laboratory testing and prompt infection control measures to interrupt human-to-human transmission during disease outbreaks. This study demonstrates the potential to use medical student clinical experience logs and the VCOM CREDO medical student patient encounter logging system as a real-time surveillance and rate-triggered alert tool.