Artificial intelligence-based predictive model for relapse in acute myeloid leukemia patients following haploidentical hematopoietic cell transplantation

Study design and participants

This study was conducted using the transplant database of Peking University (PKU), Institute of Hematology. The inclusion criteria were as follows: (1) patients with AML; (2) ≥ 12 years of age; (3) received HID HSCT in CR between January 1, 2017, and March 5, 2021; and (4) having complete medical information (Figure 1). The final follow-up was conducted on October 31, 2022. The study was conducted in accordance with the Declaration of Helsinki and the protocol was approved by the Institutional Review Board of Peking University People’s Hospital.

Figure 1

Flow chart of the study and data analysis process.

Transplant regimen

The protocols of major preconditioning regimen,^[10,34] graft-versus-host disease (GVHD) prophylaxis (i.e., antithymocyte globulin [ATG], cyclosporine A, mycophenolate mofetil, and short-term methotrexate), and infection prophylaxis are presented in Supplemental Information.^{[35, 36, 37, 38, 39, 40]} MRD before and after HID HSCT was detectable by multiparameter flow cytometry (MFC) and a lower limit of detection (LOD) of 0.01% was targeted (Supplemental information).^[21] Patients who showed MRD occurrence after HID HSCT received preemptive immunotherapy including donor lymphocyte infusion (DLI)^[41] or interferon-α treatment^[42] as previously reported (Supplemental information).

Data collection

The collected data included demographic characteristics of the patients (age, sex, and comorbidities), characteristics of leukemia and treatment before HSCT (white blood cell count and AML risk category at diagnosis, number of courses of induction chemotherapy for first CR, time from diagnosis to HSCT, and disease status before HSCT), characteristics of transplantation (donor/recipient sex-matched, donor/ recipient relation, blood group disparity, preconditioning regimen, graft type, and mononuclear cell and CD34⁺ cell counts in the graft), and MRD status before HSCT (Figure 1, Supplementary Table S1).

Building machine learning models and nomogram

The proposed method consisted of the following three steps: First, feature selection was conducted based on the entire dataset (n = 951). We then randomly selected 70% of the entire population (n = 665) as the training cohort for developing the machine learning model and nomogram, and the remaining 30% (n = 286) were used as the validation cohort.

Therefore, a predictive model was established for the training cohort. Multivariate logistic regression analysis was used for feature selection. We included variables with coefficients having P values < 0.1 as the input for the machine learning model.

A logistic regression model was selected as the machine learning model to predict relapse. It was developed using data from a training cohort. This model assumes that the probability of relapse P x i can be computed based on Equation (1) using input variables x i , where w and b can be trained from the training cohort. We chose w and b to minimize the loss function (with l 2 regularization) represented in Equation (2). When a specific instance of data is entered, the logistic regression model yields a probability (between 0 and 1) of relapse. After obtaining the probability, determining the threshold for producing negative or positive results remains important. We constructed receiver operating characteristic (ROC) curves and calculated the g-means for each threshold. The threshold with the highest g-mean was selected.

The accuracy, area under the curve (AUC), sensitivity, and specificity were computed for the training cohort.

A nomogram was developed using the well-trained logistic regression model. We first assigned each variable a point between 0 and 100 based on their estimated coefficients and ranges. We then summed all the points of the variables and used a sigmoid function to map the probabilities.^[43] Finally, we drew a horizontal line as a representative of the threshold to facilitate probability assignment. Additionally, we distinguished the nominal variables using dashed axes and applied grids for computational assistance. The workflow is shown in the Supplemental Information.

Validated machine learning models and nomogram

We validated the machine learning models and nomogram in the validation cohort, which was further validated in an independent historical cohort (n = 213).^[16] The accuracy, AUC, sensitivity, and specificity were computed for both cohorts. Calibration and decision curves were plotted to determine the usefulness of the nomogram. We also compared the AUCs of our AI-based model with those of other existing predictive models.

Additionally, we validated the discrimination and clinical usefulness^[44] of the nomogram by applying it clinically. We developed a questionnaire based on the clinical information and nomogram (Supplemental Information). Five experienced clinicians received the questionnaires and were required to compute the relapse probabilities and binary outcomes (relapse or non-relapse) based on clinical information and nomogram; each clinician was required to evaluate 10 patients. We plotted a calibration figure and confusion matrix to check the agreement between the clinical applications and the real performance of the nomogram.

Definitions

The AML risk category was assessed using the European LeukemiaNet (ELN) genetic risk.^[45] The definitions for engraftment, relapse, non-relapse mortality (NRM), event-free survival (EFS), leukemia-free survival (LFS), and overall survival (OS) are shown in Supplemental information.

Statistical analysis

Data were censored at the time of death or last available follow-up. The primary outcome was the relapse rate. Secondary outcomes included MRD, EFS, NRM, LFS, and OS. The minimum sample size was 472 according to calculations carried out in PASS version 11.0.7 (α = 0.05, power[1–β] = 0.9, and R2 = 0.15). In this study, data from 665 patients in the training cohort were used to construct the nomogram. Mann–Whitney U-test was used to compare continuous variables, and χ² and Fisher’s exact tests was used for categorical variables. The Kaplan–Meier method was used to estimate the probability of survival. We used competing risk analyses to calculate the cumulative incidence of MRD occurrence, NRM, and relapse.^[46] Testing was two-sided, with statistical significance set at P < 0.05. Statistical analyses were performed using R software (version 4.2.0) (http://www.r-project.org), Python (version 3.9.12), and SPSS 26.0 software (SPSS, Chicago, IL).

Patient characteristics

Characteristics of the 951 patients are presented in Table 1. Neutrophil engraftment was achieved by 948 (99.6%) patients, and the median time from transplantation to neutrophil engraftment was 13 days (range, 6–33 days). Platelet engraftment was achieved by 900 (94.6%) patients, and the median time from transplantation to platelet engraftment was 16 days (range, 5–184 days). Notably, 516 (54.2%) patients developed acute graft versus host disease (aGVHD) after allo-HSCT. The cumulative incidences of grade I–IV, grade II–IV, and grade III–IV aGVHD 100 days after allo-HSCT were 54.3% (95% CI, 51.1%–57.5%), 23.5% (95% CI, 20.8%–26.2%), and 7.5% (95% CI, 5.8%–9.2%), respectively. Furthermore, 401 (42.1%) patients developed chronic GVHD (cGVHD) after allo-HSCT. The cumulative incidences of moderate to severe and severe cGVHD at three years after allo-HSCT was 24.0% (95% CI, 21.2%–26.8%) and 8.2% (95% CI, 6.4–10.0%), respectively. Twenty-six patients (24.7%) with FLT3-ITD mutations simultaneously received sorafenib as a maintenance therapy. The median duration of maintenance therapy was 28 days (range, 11–210 days).

Furthermore, 111 patients experienced relapse, and the median time from HSCT to relapse was 220 days (range, 22–1738 days). Eighty patients died of NRM. The median follow-up duration was 945 days (range, 21–2190) days. The probabilities of relapse, NRM, LFS, and OS at three years after HID HSCT were 12.5% (95% CI, 10.2%–14.7%), 8.7% (95% CI, 6.8%–10.5%), 78.9% (95% CI, 76.2%–81.7%), and 83.2% (95% CI, 80.7%–85.7%), respectively.

Development of machine learning model

Five variables (AML risk category at diagnosis, number of courses of induction chemotherapy for first CR, disease status before HSCT, measurable residual disease before HSCT, and blood group disparity; Supplementary Table S2 and S3, Figure 2A and 2B) were included in the PKU-AML model, and the equation was as follows:

Figure 2

The process of model development. SHAP value summary plot for the logistic regression model. SHAP value (x-axis), feature (y-axis), feature values (color). Explanation: Each point on graph is a feature and corresponding SHAP values of an instance, the position on the y-axis is determined by the feature, the position on the x-axis is determined by the SHAP value, the color represents the feature value from small to large, and the overlapping points are on the y-axis direction, so we can understand the distribution of SHAP values for each feature. (A) SHAP = SHapley Additive exPlanations. The mean absolute SHAP values of the top 5 features (B). The x-axis (instances) values are sorted by (C) similarity, and (D) output values. Values higher on the vertical axis indicate higher likelihood of relapse. Values lower on the vertical axis indicate a lower likelihood of relapse. Values that are red drive the relapse risk up. Values that are blue drive the relapse risk down. Receiver operating characteristic (ROC) curve and confusion matrix for relapse model in the training (E) and validation cohorts (F).

where Y = 0.5677 × (AML risk category at diagnosis) + 0.0690 × (number of courses of induction for first CR) + 0.4583 × (disease status before HSCT) + 0.4061 × (MFC before HSCT) – 0.1623 × (blood group disparity) – 2.9641.

The threshold of probability was set at 0.1106, and the g-mean was 0.668. The force plot (Figure 2C and 2D) illustrates how the features contributed to the prediction of the model for all observations. The sensitivity, specificity, AUC, and accuracy scores of the training cohort are shown in Figure 2E and were 0.7000, 0.6250, 0.7071, and 0.6329, respectively, in the validation cohort (Figure 2F).

Development of prediction nomogram

A nomogram was designed using the training cohort based on the machine learning model (Figure 3A), and the validation cohort showed that the concordance index was 0.707 (95% CI 0.645–0.770). The calibration plots in the training and validation cohorts (Figure 3B and 3C) revealed satisfactory agreement between the nomogram prediction and actual observations for the probability of relapse. Based on the decision curve analysis (Figure 3D and 3E), if the threshold probability was > 0.1, using this nomogram to predict relapse would provide a greater net benefit than either a treat-all-patients scheme or a treat-none scheme. The optimal cutoff value of the total nomogram scores was determined to be 95 (Figure 3F), and the patients were separated into low- and high-risk groups. The Hosmer−Lemeshow test showed that the model had a good fit (P = 0.205).

Figure 3

Nomogram for estimating the probability of relapse and its predictive performance. Nomogram predicting the probability of relapse for AML patients receiving haploidentical hematopoietic stem cell transplantation in complete remission based on training cohort. AML risk category at diagnosis: favorable = 0, intermediate = 1, adverse = 2; number of courses of induction chemotherapy for first CR: numerical value; disease status before HSCT: CR1 = 1, CR2 = 2, ≥ CR3 = 3; MFC before HSCT: negative = 0, ≥ 0.01% but < 0.1% = 1, ≥ 0.1% but <1% = 2; ≥ 1% = 3; blood group disparity: matched = 0, minor mismatched = 1, major mismatched or minor and major mismatched = 2 (A). Calibration plot of actual probability versus predicted probability of relapse based on training (B) and validation cohort (C). Decision curve analysis demonstrating the net benefit associated with the use of our model for predicting relapse based on training (D) and validation cohort (E). Accuracy of the prediction score of the nomogram for estimating the probability of relapse (F).

In the training cohort, the three-year cumulative incidence of relapse after HID HSCT were 18.9% (95% CI, 13.8%–24.0%) and 9.2% (95% CI, 6.1%–12.2%) in the high- and low-risk groups, respectively (P < 0.001; Figure 3G). In the validation cohort, the three-year cumulative incidence of relapse after HID HSCT were 19.5% (95% CI, 11.9%–27.1%) and 5.9% (95% CI, 2.1%–9.7%) in the high- and low-risk groups, respectively (P < 0.001; Figure 3H). We observed that the three-year cumulative incidence of relapse was significantly higher in high-risk patients than in low-risk patients in all subgroups and was as high as 26.8% in high-risk patients who received HID HSCT after CR1 (Supplementary Table S4).

Validation of the PKU-AML model in an independent cohort

A total of consecutive 213 patients with AML were included, and their characteristics are shown in Supplementary Table S5. The AUC and accuracy scores of the PKU-AML model were 0.7074 and 0.8685, respectively (Figure 4A). The concordance index was 0.7074 (95% CI 0.300–1.000). The calibration plots (Figure 4B) revealed a satisfactory agreement between the nomogram prediction and the actual observation of the probability of relapse. Based on decision curve analysis, if the threshold probability was > 0.1, using this nomogram to predict relapse would provide more net benefit than either a treat-all-patient scheme or a treat-none scheme (Figure 4C). The three-year cumulative incidence of relapse after HID HSCT were 16.9% (95% CI, 10.7%–23.0%) and 3.6% (95% CI, 0–8.7%) in the high- and low-risk groups, respectively, (P = 0.018; Figure 4D) in this cohort.

Figure 4

Validation of AI-based model in an independent cohort. Receiver operating characteristic (ROC) curve and confusion matrix for relapse model in the independent cohorts (A). Calibration plot of actual probability versus predicted probability of relapse based on independent cohort (B). Decision curve analysis demonstrating the net benefit associated with the use of our model for predicting relapse based on independent cohort (C). The 3-year cumulative incidence of relapse after transplantation in the low‑ and high‑risk groups in independent cohort (D).

Validation of the nomogram in clinical practice

A total of 50 questionnaires were returned. The calibration curve obtained from the questionnaires (Figure 5A) showed that the nomogram-maintained consistency with the predictive probability when applied clinically. It tended to slightly overestimate the probability of relapse when the actual probability was small. The confusion matrix (Figure 5B) illustrated an accuracy of 0.92 for the actual usage. Among the four false-positive instances, three were from one patient predicted by the nomogram with a relapse probability of 0.1006, which was close to the threshold of 0.1106. However, the proportion for patients with nomogram-predicted relapse probability between 0.10 and 0.11 was only 0.031. In this case, false distinguishments rarely occurred.

Figure 5

Clinical performance for nomogram. Calibration plot of probability from questionnaire versus actual probability of relapse based on nomogram (A). Confusion matrix of relapse outcome from questionnaire versus actual outcome of relapse based on nomogram (B).

Comparison of predictive value between our PKU-AML model and other existing models

Five existing models were included: Hematopoietic cell transplantation comorbidity index (HCI-CI) score, HCT-CI/Age score,^[47] AML-specific disease risk group,^[33] haploidentical European Group for Blood and Marrow Transplantation (EBMT) risk score (haplo-EBMT),^[26] and haploidentical DRCI (haplo-DRCI).^[16] The ROC and precision-recall curves of our PKU-AML model and these existing models for relapse prediction are shown in Figure 6 and Supplementary Table S6. The AUC and average precision of our PKU-AML model were superior to those of the other existing models for predicting post-transplant relapse after HID HSCT. We compared the PKU-AML model with the MRD before HSCT and ELN genetic risk. The AUC and average precision of our model were superior in predicting post-transplant relapse after HID HSCT (Supplementary Figure S1).

Figure 6

Models performance evaluation and comparisons in validation cohort. Receiver operating characteristic (ROC) curve (A) and precision-recall curves (B) of our AI-based model and other existing models for relapse prediction.

Validation of the prediction nomogram for other outcomes after HID HSCT

As the training cohort was developed based on relapse and not on other outcomes, we combined the training cohort with the validation cohort to analyze secondary outcomes.

Notably, 244 patients developed MRD after HID HSCT. Low-risk patients showed a lower cumulative incidence of MRD occurrence than high-risk patients (Table 2 and S7), and the cumulative incidence of MRD occurrence at three years after HID HSCT were 30.8% (95% CI 26.0%–35.6%) and 23.2% (95% CI 19.6%–26.7%) for the high- and low-risk group, respectively (P = 0.011, Supplementary Figure S2). The low-risk group showed a lower probability of EFS than the high-risk group (Tables 2 and S8), and the probability of EFS at three years after HID HSCT were 55.8% (95% CI 50.8%–61.2%) and 65.4% (95% CI 61.4%–69.5%) for the high- and low-risk groups, respectively (P = 0.003, Supplementary Figure S2). A total of 210 patients received preemptive immunotherapies, 173 patients received preemptive Interferon (IFN) α treatment, and 37 patients received preemptive DLI. The cumulative incidence of relapse at three years after preemptive immunotherapy were 33.7% (95% CI, 22.5%–44.9%) and 15.9% (95% CI, 9.1%–22.7%) in the high- and low-risk groups, respectively (P = 0.011). The probability of LFS at three years after preemptive immunotherapy were 60.6% (95% CI, 50.3%–73.1%) and 80.2% (95% CI, 73.1%–88.1%) in the high- and low-risk groups, respectively (P = 0.006). In patients who received preemptive IFN-α therapy, the cumulative incidence of relapse at three years after IFN-α therapy were 23.8% (95% CI, 11.9%–35.6%) and 10.1% (95% CI, 4.1%–16.1%) in the high- and low-risk groups, respectively (P = 0.054). The probability of LFS at three years after IFN-α therapy were 68.8% (95% CI, 57.4%–82.5%) and 85.4% (95% CI, 78.6%–92.8%) in the high- and low-risk groups, respectively (P = 0.023). The high-risk group showed a trend toward a higher incidence of relapse and lower probability of LFS than the low-risk group in patients receiving preemptive DLI.

The probabilities of LFS and OS at three years after HID HSCT for patients in the high-risk group were significantly lower than those for patients in the low-risk group (Table 2). The three-year cumulative incidence of NRM after HID HSCT was comparable between groups (Table 2).