Predicting functional outcome in acute ischemic stroke patients after endovascular treatment by machine learning

2.1 Subjects

We analyzed acute ischemic patients who received EVT in Zhongnan Hospital of Wuhan University from January 2018 to December 2020. Inclusion criteria were as follows: (1) age ≥18 years; (2) conformed to the diagnostic criteria of acute ischemic stroke in Chinese guidelines for diagnosis and treatment of acute ischemic stroke 2018; (3) with LVO in the anterior circulation; (4) received EVT, with or without intravenous alteplase treatment within 4.5 h after symptom onset; and (5) completed 90-day follow-up. Exclusion criteria were as follows: (1) modified Rankin Scale (mRS) score ≥3 before stroke onset; (2) without magnetic resonance imaging or CT examination after EVT; and (3) without complete clinical data. In total, there were 156 patients enrolled in our study. Patients were assigned to the favorable outcome group (90-day mRS ≤2) and the poor outcome group (90-day mRS >2).

Demographic characteristics, including age, gender, mRS score before stroke onset, baseline NIHSS score, blood pressure at admission, baseline blood glucose levels, blood routine test, cholesterol, and kidney function. Cardiovascular risk factors, such as hypertension, diabetes mellitus, hyperlipidemia, atrial fibrillation, history of smoking, and history of ischemic stroke, were recorded as well. Imaging parameters, such as regional leptomeningeal score (rLMC) [25], TAN collateral circulation scores [26], ASPECT score, and hyperdense middle cerebral artery sign, were also analyzed in the study. All images were analyzed by two neuroradiologists with more than 5 years of image analysis experience. Reproducibility of imaging feature quantification was calculated by comparing the results from the two reviewers in 30 random samples.

Successful reperfusion was defined as a modified thrombolysis in cerebral infarction (mTICI) score of 2b or 3. Symptomatic intracerebral hemorrhage (sICH) was defined as any type of ICH with an increase of NIHSS score of ≥4 points compared to baseline within 24 h or leading to death [27]. ASPECT score was divided into dichotomous variables according to <8 and ≥8. TAN scores ≥2 were defined as good collateral circulation, while <2 was defined as poor collateral circulation [26]. Complications included pulmonary infection, gastrointestinal bleeding, and cardiovascular events after EVT.

2.2 Primary outcome

The primary outcome was functional dependence (defined as a mRS score of ≤2) at 90 days after EVT [10,11].

2.3 Feature selection and model building

First, least absolute shrinkage and selection operator (LASSO) regression algorithm was performed to retain the most predictive features, followed by univariate logistic regression. Variables with a p value of <0.05 were incorporated into subsequent models. The importance of the selected variables was evaluated by a random forest algorithm. Then, various machine learning models were built, including logistic regression, LDA, SVM, KNN, and decision tree. Receiver operating characteristic (ROC) curves were constructed, and models were compared by area under the curve (AUC). After comparisons, we chose a multivariate logistic regression algorithm to construct an individualized discriminatory nomogram. Collinearity of combinations of variables in the model was evaluated by variation inflation factors (VIF, <10 indicates no multicollinearity). The concordance index (C-index) and AUC were used to measure the discrimination ability of the nomogram, the calibration plot was performed to visually assess the calibration, and the Hosmer–Lemeshow test was used to evaluate the goodness of fit. Decision curve analysis (DCA), by quantifying net benefits against a range of threshold probabilities, was used to evaluate the clinical utility of the nomogram.

Lastly, a pairwise comparison of ROC curve was used to compare the nomogram with previously published prognostic scoring systems, including THRIVE (Total Health Risks in Vascular Events-calculation) score [12], HAIT2 (Houston Intra-Arterial Therapy 2) score [13], SPAN-100 (Stroke Prognostication using Age and NIHSS) score [14], wSPAN (Weighted Stroke Prognostication using Age and NIHSS) score [15], and PRE (Pittsburgh Response to EVT) score [16]. The THRIVE, HIAT2, SPAN, wSPAN, and PRE index were calculated as previously described [12,13,14,15,16]. The THRIVE score assigned 1 point for age 60–79 years, 2 points for age ≥80 years, 2 points for NIHSS score 11–20, 4 points for NIHSS score ≥21, and 1 point each for hypertension, diabetes mellitus, and atrial fibrillation. The HIAT-2 score assigned 2 points for age 60–79 years, 4 points for age ≥80 years, 1 point for NIHSS score 11–20, 2 points for NIHSS score ≥21, and 3 points for ASPECT score ≤7. The SPAN-100 score was defined as (NIHSS + age), and the score is positive if the sum of age and NIHSS ≥100. wSPAN index was defined as ([3 × NIHSS] + age), and PRE score was defined as (age + 2 × NIHSS − 10 × ASPECT score) score. Furthermore, we made a subgroup analysis of patients in different time windows by the model. The flow of feature selection and model building is shown in Figure S1.

2.4 Statistical analysis

Statistical analysis was performed using R software (version 4.1.2, http://www.R-project.org). Continuous variables were expressed as mean ± standard deviation or median (interquartile range, IQR) for normal or non-normal distributions, respectively, followed by an unpaired t-test or Wilcoxon rank sum test. Categorical variables were summarized as counts (percentages) and compared using the chi-square test or Fisher’s exact test, as appropriate. All tests with two-sided p < 0.05 were considered statistically significant. The R package “tableone” was used in these analyses.

The reproducibility of imaging features of lesion was evaluated by intraclass coefficient (ICC) using a two-way random model with absolute measurements. The “irr” package was used for the ICC algorithm, “tidyverse” package for data collation and exploration, “glmnet” for LASSO regression, “car” for multicollinearity detection, “rms” for nomogram, “Hmisc” for C-index calculation, “rmda” for DCA, “pROC” for ROC analysis, and “ResourceSelection” for goodness of fit test. The packages “MASS,” “e1071,” “class,” “rpart,” “rpart.plot,” “randomForest,” and “caret” were applied in different machine learning models’ construction, respectively.

1. Ethical approval: The research related to human use has been complied with all the relevant national regulations, institutional policies, and in accordance with the tenets of the Helsinki Declaration and has been approved by the authors’ institutional review board or equivalent committee. The studies involving human participants were reviewed and approved by the Ethics Committee of Zhongnan Hospital, Wuhan University (No. 2023007K).

2. Informed consent: Written informed consent for participation was not required for this retrospective study in accordance with the institutional requirement.

3.1 Demographic and clinical characteristics

According to the forementioned inclusion and exclusion criteria, a total of 156 patients were finally enrolled in our study (Table 1). The median age of the patients was 65(55–74) years, and 84 patients (53.8%) were men. An unfavorable outcome at 90 days was observed in 109 (69.9%) of the cases, and the overall mortality at 90 days was 21.2%. The age of patients in the poor outcome group was elder than that in the favorable outcome group (66.00 [57.00, 75.00] vs 62.00 [48.50, 70.00], p = 0.04). There was no significant difference in the cardiovascular risk factors and laboratory blood tests between two groups. Compared with the favorable outcome group, the poor outcome group had higher NIHSS score (18.00 [14.00,23.00] vs 13.00 [11.50,15.00], p < 0.001), lower rates of ASPECT score ≥8 ( 32 [68.1%] vs 44 [40.4%], p = 0.003), successful recanalization (mTICI ≥ 2b) (78 [71.6%] vs 42 [89.4%], p = 0.027), and more rates of complications (103 [94.5%] vs 27 [57.4%], p < 0.001) and sICH (37 [33.9%] vs 2 [4.3%], p < 0.001).

3.2 Identification of significant predictors for poor outcome following EVT

A total of 39 potential predictors were enrolled in LASSO regression as shown in Figure 1(a) and (b), and 13 candidate predictors with nonzero coefficients were selected, including age, gender, baseline NIHSS score, smoking history, coronary heart disease, diabetic mellitus, blood urea nitrogen, lymphocyte count, low-density lipoprotein, ASPECT score, mTICI score, complications, and sICH. These variables were further analyzed by univariate logistic analysis, and characteristics with p < 0.05 were age, baseline NIHSS score, mTICI score, ASPECT score, complications, and sICH (Figure 1c). A variable importance measure was used to evaluate the impact of each variable. The order of importance of the variables was as follows (highest to lowest): baseline NIHSS score, age, complications, ASPECT score, sICH, and mTICI score (Figure 2a). Based on the forementioned significant variables, we constructed different models. The ROC curves indicated that logistic regression was as good as LDA, SVM, and KNN (p > 0.05) and non-significantly better than the decision tree (AUC, 0.886 vs 0.831, p = 0.075) (Figure 2b, Table 2). Logistic regression model was used for further analysis due to its better performance. No effect of collinearity was observed for the variables in the logistic regression analysis. The results of these variables in the multivariate logistic regression demonstrated that age (odds ratio [OR] = 1.04, 95% confidence interval [CI]: 1.00–1.07, p = 0.033), baseline NIHSS score (OR = 1.17, 95% CI: 1.05–1.30, p = 0.005), ASPECT score (≥8 vs <8: OR = 0.37, 95% CI: 0.15–0.96, p = 0.041), mTICI score (≥2b vs <2b: OR = 0.21, 95% CI: 0.06–0.78, p = 0.020), complications (yes vs no: OR = 7.74, 95% CI: 2.46–24.37, p < 0.001), sICH (yes vs no: OR = 5.13, 95% CI: 1.05–25.10, p = 0.043) were significant independent predictors for outcome after EVT (Figure 1c).

Figure 1

Identification of significant predictors for poor outcome after endovascular treatment in acute anterior circulation stroke patients. (a) LASSO coefficient profiles of the candidate predictors. (b) Selection of the optimal penalization coefficient in LASSO regression. (c) Univariate and multivariate logistic regressions of the predictors.

Figure 2

(a) Importance value of clinical variables. Mean Decrease Gini indicates a decrease in the Gini coefficient after variable substitution. A higher value indicates a higher importance. (b) ROC curves of different predictive models using logistic regression (red), LDA (orange), SVM (blue), KNN (purple) method, and decision tree (green).

3.3 Construction and validation of the prediction nomogram

Based on the above-mentioned significant predicting factors, we established a predictive nomogram. Each of the six predictors was assigned a score ranging from 0 to 100 on a point scale. After adding the scores of all variables to “Total Point”, the corresponding value was the probability of unfavorable outcome event (Figure 3a).

Figure 3

Construction of the predictive nomogram for poor outcome after endovascular treatment in acute anterior circulation stroke patients. (a) Development of the nomogram. For example, if a patient with an age of 55, baseline NIHSS score of 10, mTICI <2b, ASPECT score ≥8, with sICH and without complications, the corresponding points for each factor were 19, 14, 29, 0, 0, and 38, respectively. The total point for this patient was 100, with a probability of 0.65 for having unfavorable outcome after endovascular treatment. (b) ROC curves of the nomogram. (c) Calibration curve of the nomogram. (d) DCA in the cohort.

Based on the maximum Youden index, the optimal cutoff value of the nomogram predicted probability was 0.695. At this cutoff value, the sensitivity, specificity, positive predictive value, and negative predictive value for differentiating the presence from the absence of unfavorable outcome events were 81.7, 83.0, 91.8, and 66.1%, respectively. The patient would have poor outcome when the total prediction probability is beyond the cutoff point. ROC curve of the model is demonstrated in Figure 3b. Figure 3c shows the calibration curves for the nomogram model. The calibration curve and an insignificant Hosmer–Lemeshow test statistic (X-squared = 7.09, p = 0.527) showed a good agreement between the predicted model and actual observation in the study, along with the C-index (0.886, 95% CI 0.832–0.940) indicating that the nomogram is a discriminant tool. Five-fold 100 times cross-validation was performed, in which the C-index was 0.850 (95% CI 0.843–0.857). Given that a C-index >0.75 is generally considered to indicate reliable discrimination, this nomogram performed well in the study. DCA results showed that the nomogram had more benefits in the clinic practice than taking the measures that treat all patients or treat none of patients (Figure 3d).

3.4 ROC curve analysis comparing nomogram with other outcome prediction scores

We compared the AUC of the nomogram with previously published risk models (Figure 4a). The AUC of the nomogram for predicting poor functional outcome was superior to the THRIVE score (AUC, 0.886 vs 0.688, p < 0.001), HAIT2 score (AUC, 0.886 vs 0.737, p = 0.002), SPAN-100 score (AUC, 0.886 vs 0.687, p < 0.001), wSPAN score (AUC, 0.886 vs 0.750, p = 0.008), and PRE score (AUC, 0.886 vs 0.757, p = 0.007). This result indicated that our nomogram is a distinctive tool for the outcome prediction after EVT. A similar significant difference was also observed in the prediction of mortality (Figure 4b). Besides, our nomogram was superior to the other two prediction models [28,29], as it is shown in Figure S2.

Figure 4

Discriminability analysis. (a) Models predicting poor outcome after EVT. (b) Models predicting mortality after EVT. Comparison of discriminability of the nomogram, THRIVE (Total Health Risks in Vascular Events) score, HAIT2 (Houston Intra-Arterial Therapy 2) score, SPAN-100 (Stroke Prognostication using Age and NIHSS) score, wSPAN (weighted Stroke Prognostication using Age and NIHSS) score, and PRE (Pittsburgh Response to EVT) score was performed by ROC curve analysis. AUC of the nomogram was apparently superior to that of any other prediction model.

3.5 Subgroup analysis in patients with different time windows

We divided patients into two groups according to whether they were admitted to hospital within 6 h. The AUC of the nomogram in the within 6-h time window group was 0.914 (95% CI, 0.833–0.995), and the AUC in the other group was 0.889 (95% CI, 0.828–0951).

3.6 Reproducibility of imaging data

The ICCs of the imaging data for the two reviewers in measuring mTICI, infarct site, ASPECT score, hyperdense middle cerebral artery sign, rMLC, and TAN score were 0.819, 0.929, 0.963, 0.806, 0.747, and 0.804, respectively.