Application of machine learning tools for feature selection in the identification of prognostic markers in COVID-19

Introduction

One of the most relevant tasks for a clinician caring for a patient with coronavirus disease (COVID)-19 is to identify the risk of developing complications in order to plan how to handle the patients (outpatient or in-hospital management) as well as the implementation of specific therapeutic measures. To this end, a large amount of research has been done around the world, which is focused on identifying the variables related to the prognosis of individuals at various stages of this infection (Figliozzi et al. 2020; Liang et al. 2020).

From a medical standpoint, risk factor analysis has historically consisted of the initial assessment of each particular variable, or univariate analysis, which is then supplemented by the results from an evaluation of a subset chosen under criteria related to the evidence of their individual association, or multivariate analysis. This analytical framework, which was initially designed to establish associations between risk factors and diseases, was subsequently extended in order to identify prognostic variables or markers (Fletcher and Fletcher 2014). Nevertheless, significant limitations for this strategy have been identified, and this has led to a search for new approaches on how to deal with prognostic markers (Goldstein, Navar, and Carter 2017; Pepe et al. 2004).

In the research on COVID-19, after reviewing the studies that attempted to identify these prognostic markers, variability in the lists of the variables presented is evident (Rod, Oviedo-Trespalacios, and Cortes-Ramirez 2020). Although a coincidence in the majority of the factors identified could be expected, this procedure was found to be sensitive to variation due to the experimental conditions determined by the type of population explored, the geographical area, timing of the pandemic outbreak, along with other aspects (Di Castelnuovo et al. 2020; van Halem et al. 2020; Xu et al. 2020). An additional factor that changes in these studies and could affect the final results is how the variables are selected (Bursac et al. 2008; Núñez, Steyerberg, and Núñez 2011).

From the point of view of the fields of machine learning, pattern recognition, and data science, the selection of variables or characteristics is also a preliminary procedure done before the development of prediction (preprocessing) tools. The goal of this selection is threefold: (a) improve prediction performance, (b) provide faster and more cost-effective predictors, and (c) provide a better understanding of the underlying process that generated the data (Guyon and Elisseeff 2003). The management of huge amounts of data (bigdata) has led to the development of various strategies that are increasingly sophisticated and successful and have resulted from the improvement in the generalizing capacity of the various models fed by the variables resulting from these processes (Li et al. 2017).

Artificial intelligence tools have been used in different processes in the fight against the COVID-19 pandemic, including diagnosis, public health, clinical and therapeutic decision-making (Chen and See 2020; Rasheed et al. 2021); but so far, it’s possible usefulness in identifying prognostic markers has not been explored. The objective of this study is to explore an alternate route for identifying prognostic markers by leveraging the capacity of machine learning strategies in the feature selection.

Methodology

A sub-analysis was done of a multi-center retrospective observational study of patients hospitalized in the emergency department or general ward for SARS-CoV-2/COVID-19 viral pneumonia confirmed by real-time polymerase chain reaction (RT-CRP) test in nasal swabs between April 15 and December 31, 2020 in three fourth-level clinical care hospitals in Bogota, Colombia. The following patients were excluded: those who were admitted directly to the Intensive Care Unit (ICU) having been referred after a 72-h stay in another institution and were unaware of the outcome under study, women in gestation, and patients with any condition that could seriously affect their short-term survival as recorded on their medical chart.

The patients were screened based on the census of hospitalized patients in the three institutions and their data were entered into a virtual format built using data recommended by the WHO International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC).

Results

Between April 15 and December 31, 949 patients were included of which 399 came from San Jose Hospital, 394 from El Tunal Hospital, and 156 from San Jose University Children’s Hospital. Complete data for all variables were obtained on 466 patients, of which 220 were selected for the development database, in 83 cases the main outcome occurred and representing [36 (16.4%) deaths and 72 (32.7%) were transferred to ICU]. The remaining 266 patients were used for the validation. Among them, the primary outcome occurred in 52 cases [44 (16.5%) deaths and 73 (27.4%) were transferred to ICU]. Figure 1 shows the flowchart of selected patients and the applied methodology.

Figure 1:

Flowchart of methodology.

The characteristics of the population and laboratories are described in Tables 2 and 3. Like other cohorts of hospitalized patients, men predominate – 137 (62.3%). The median age was 58 (IQR 21.5). The most frequent comorbidities were diabetes – 39 (17.7%), high blood pressure – 62 (28.2%), and chronic lung disease – 32 (14.5%). The median duration of symptoms prior to admission was 8 days (IQR 4). The most frequent symptoms were cough in 200 patients (90.9%), fever in 175 (79.5%), and dyspnea in 176 (80.0%). The majority had infiltrates on the chest X-rays, 192 (87.3%). In the laboratories, the median number of lymphocytes was 7,870 cel/µL (IQR 4692); for ferritin, it was 787 (1,086); for dehydrogenase, 429 (380.8); and C-reactive protein, 14.4 (18.8). Troponin was positive in 39 patients (17.7%).

Discussion

The experiments done through our analysis identify, based on a relatively small population, a series of variables that are related to a requirement for intensive care or death in patients hospitalized for COVID-19. Therefore, they are considered prognostic markers. The ten most frequently identified by various algorithms were LDH, the PaO₂/FiO₂ ratio, CRP, age, and neutrophil and lymphocyte count.

The prognostic markers identified in our study have also been identified in other studies done in various parts of the world. Age and CRP are the most consistent (Rod, Oviedo-Trespalacios, and Cortes-Ramirez 2020) as are also the neutrophil count, lymphocyte count, and LDH which have been documented in several studies and have been cataloged as average in consistence (Rod, Oviedo-Trespalacios, and Cortes-Ramirez 2020). In a meta-analysis involving 58 studies, Noor et al. (Noor and Islam 2020) identified age >65 years (RR 3.59, 95% CI [1.87–6.90], p<0.001) and admission to ICU (RR 3.72, 95% CI [2.70–5.13], p<0.001) were the most statistically significant factors related to in-hospital mortality. In another meta-analysis that evaluated hematological and immunological markers, Elshazli et al. (Elshazli et al. 2020) identified 9 variables related to progression to a severe phenotype among which are found neutrophil counts (OR=2.62). Four were related to admission to ICU including the neutrophil count (OR=6.25), and another four variables were related to mortality among which a neutrophil count (OR=6.25) and the CRP (OR=7.09) were found. Another systematic review done by Izcovich et al. (Izcovich et al. 2020), evaluated 207 studies and identified 49 variables that could provide prognostic information on mortality and severe disease and among these there were variables that were identified by the strategies used in our study.

Of the models evaluated for selecting variables, three (SW, LASSO, and RFS) are derived from traditional statistics and are considered to be improvements on the usual practice of building multivariate models (Bursac et al. 2008; Taylor and Tibshirani 2015). The remaining five are tools used in machine learning. Three of them are based on decision trees. Even when strategies with different mathematical foundations were used, the generation of logistic regression models from each of the subsets of selected variables demonstrated a similar prognostic performance (with confidence intervals superimposed on all the accuracy measurements). This can be interpreted as a similar ability to identify prognostic markers. In general, these models were well-calibrated and the results in this population were not very different from those in the training population suggesting that there was no substantial overfitting. There was a low percentage in the variance of the results identified by the R-squared with the methodological limitations that this statistician may have (Cox and Snell 1989). This situation requires consideration of the complexity and the wide range of risk patterns that COVID-19 may have as highlighted by the various results obtained from a large number of studies.

The evaluation of the prognostic markers remains a major challenge in medical research. The traditional statistical methods used in the clinical study to evaluate etiological associations or “risk factors” are not always suitable for determining the potential performance of a marker for classifying or predicting results for individuals. Relatively often the performance of many of these markers can be described as disappointing (Pepe et al. 2004). There are three analytical challenges in these types of studies that limit the usefulness of logistic regressions for this by decreasing: the assumption of linearity between risk factors and outcome, the diversity of the effects, and the presence of many prognostic-related variables. These considerations may affect the performance of the logistic regressions as real-world prognostic evaluators (Goldstein, Navar, and Carter 2017). In addition, the purpose for feature selection for establishing prognostic models is to determine which combination of variables will offer the best performance when a regression or classification model is generated (Chandrashekar and Sahin 2014). A variable considered useless by itself can provide a significant improvement in performance when taken with others. Furthermore, perfectly correlated variables are actually redundant in the sense that no additional information is obtained when they are added together (Guyon and Elisseeff 2003). The challenge of correcting the effects of the selection is complex since selective decisions may occur at different stages in the process of analysis (Pepe et al. 2004). There are several proposals to solve this problem such as the false discovery rate strategy (Benjamini, Heller, and Yekutieli 2009), sample division, adaptive regression techniques (Taylor and Tibshirani 2015), Bayesian probabilistic models (García-Donato, Castellanos, and Quirós 2021), and automatic learning techniques (Nilsson et al. 2019). This last one was the one evaluated in the current paper.

One of the areas where the identification of prognostic markers has been changing in the last few years is in genetics. Here the association technique through genome-wise scan for associations (GWA) has been used. This makes inferences from variations in polymorphism areas associated with the problem condition within a large number of these evaluated areas (García-Donato, Castellanos, and Quirós 2021; Nilsson et al. 2019). This technique has suffered from a reproducibility problem that has been called “selective inference” (Taylor and Tibshirani 2015). One possible explanation is that logistic regression models ignore “heterogeneous effects,” and this can generate misleading evidence for large subgroups of individuals (Benjamini 2020). This is a problem that is similar to the one faced in clinic practice where using a large number of initial variables an effort is made to identify some of them as prognostic markers.

This study has several limitations. One of them is the size of the population since a calculation of the sample size was not carried out. Another was the impossibility of recognizing variables that boosted the performance of others within a model but that alone, do not mark a difference. Although there is a significant number of works that have developed models for the forecast of COVID-19 from which the variables that constitute them can be extracted (Bottino et al. 2021; Wang et al. 2021), the discussion around individual variables does not consider their results at the time of their analysis (example: obesity (Yang et al. 2021) or ferritin (Taneri et al. 2020)); so it is a field to be explored in which there is still much to say. We must note that the parameters used for the feature selection could not be modified since they were those preset by the packages used in the R program.

The possibility of identifying or at least corroborating the information already produced with respect to prognostic factors shows the possibility of favoring local decision-making with greater certainty. Future work will include testing the hypothesis of different populations (greater in size or other spectrum of disease) as well as in other clinical entities. A hypothesis that is worth studying is how assembling a set of prediction tools fed with different sets of variables might show better predictive performance along with a better ability to generalize.

Conclusions

The combined use of strategies for characteristic selection derived from machine-learning made it possible to identify a broader variety of prognostic markers for death or hospitalization in intensive care in patients who were hospitalized for COVID-19 starting from a small population of patients coinciding with those who were identified by other studies.