Survival risk prediction in hematopoietic stem cell transplantation for multiple myeloma

3.1 Dataset description and pre-processing

The dataset utilized in this study is provided by the Hematology Department of the University Hospital Virgen de la Arrixaca in Murcia, and contains medical records of transplants performed between 1995 and 2021, with patient follow-up extending up to the latter year.

Initially, the data was compiled into a spreadsheet consisting of 1,115 records and 48 attributes to track the progress of patients as monitored by healthcare professionals. The attributes covered a range of diagnoses requiring hematopoietic stem cell transplants, including both allogeneic and autologous procedures, as well as post-transplant information.

For the purposes of this study, we filtered the dataset to include only autologous transplants in patients diagnosed with MM. After removing irrelevant attributes and crafting time-related features, the resulting dataset comprised 177 instances and 12 attributes.

As shown in Table 1, most of the attributes are categorical, except for the patient’s age at the time of transplantation. To effectively handle missing values – a common issue in clinical datasets – a multi-step imputation strategy was applied. Firstly, domain experts reviewed the dataset and manually completed missing entries when supporting clinical information was available from external sources, such as patient medical records or laboratory results. This approach ensured that critical clinical variables were imputed based on accurate and context-specific data. For cases where manual completion was not feasible, an AI-based imputation method was employed using the k-Nearest Neighbors algorithm.

Since these models cannot directly process nominal variables, we transformed the categorical attributes into numerical format using the encode_categorical() function from the sksurv library, encoding attributes with M categories into M-1 binary variables using one-hot encoding.

3.2 Survival models and risk predictions

The metric utilized to assess the performance of survival prediction models, known as the C-index, gauges the model’s capability to accurately rank individuals based on their predicted risks and observed survival times. Specifically, the C-index measures how effectively the model differentiates between two randomly selected patients, one of whom experiences the event of interest before the other. A pair is considered concordant if the patient with the higher predicted risk indeed experiences the event earlier. The C-index score is computed as the ratio of concordant pairs to all comparable pairs, with a score of 1.0 indicating perfect concordance (the model accurately ranks all patients), while a score of 0.5 suggests random prediction.

The selection of these SA models from the Python library scikit-survival [15], was driven by their well-established use in time-to-event data analysis. These models can be categorized into three main types:

1. Linear Models, such as Cox’s proportional hazard with elastic net penalty (Coxnet), model the relationship between covariates (independent variables) and individual survival times. This model posits that the hazard function is a product of a baseline hazard function and an exponential function of a linear combination of covariates, without imposing assumptions on the shape of the baseline hazard, thus classifying it as semi-parametric.

2. Ensemble Models amalgamate multiple models to enhance predictive performance and robustness, adeptly capturing intricate relationships in the data and managing non-linear patterns that might be overlooked by individual models. Notable ensemble models include Random Survival Forest (RSF), derived from traditional Random Forests, Gradient Boosting (GB), and Componentwise Gradient Boosting (CWGB).

3. Survival Support Vector Machines (SVMs) seek to identify a hyperplane that optimally separates the data into distinct classes. In the context of SA, these classes correspond to different survival times, ranked using a loss function that penalizes prediction errors. SVMs are particularly effective in high-dimensional data scenarios and can capture both linear and non-linear relationships. The SVMs employed in this study are the Fast Survival Support Vector Machine (FS) and the Fast Kernel Survival Support Vector Machine (FKS).

This diverse set of models ensures a comprehensive evaluation of survival prediction techniques, balancing accuracy, interpretability, and computational efficiency.

The process for utilizing these models is as follows. First, each model undergoes optimization by identifying the best hyperparameters, which involves creating a dictionary containing the model’s parameter settings and employing the C-index metric as a means to evaluate the performance of the cross-validated model.

Subsequently, each model is assessed using a 10-fold cross-validation with three repetitions on the dataset. This approach entails dividing the dataset into multiple subsets, or folds, and systematically training and testing the model across various combinations of training and validation sets. Utilizing cross-validation enables us to evaluate the model’s generalization capabilities over different data subsets, strengthening the evaluation against the limitations posed by a single static train-test split and reducing the risk of overfitting that may occur when using the entire dataset for both training and evaluation.

The SA models utilized in this research apply the predict() method to generate a risk score, which is an arbitrary value indicating the likelihood of the event occurring sooner for a particular patient compared to others in the dataset. This risk score is typically unbounded and not linked to a specific scale, serving primarily as a relative indicator of risk rather than an absolute probability.

In practice, higher risk scores signify that a patient is more likely to experience the event of interest earlier, whereas lower risk scores imply a longer anticipated time until the event occurs. This prediction mechanism allows for the ranking of patients based on their predicted risk, which is especially beneficial in clinical decision-making. Clinicians can leverage these risk scores to prioritize patients for closer observation, make informed choices regarding the necessity of proceeding with a transplant, and assess the potential requirement for additional interventions.

3.3 Feature importance techniques

To gain insights into the significance of each variable, we employ feature importance methods, specifically permutation importance and SHAP (Shapley Additive exPlanations), within our SA models.

1. Permutation Importance. This technique evaluates the importance of each feature by randomly shuffling its values and observing the resultant impact on the model’s performance. During this process, the values of a feature are permuted, and the model is re-evaluated using the modified data. The change in performance serves as a measure of that feature’s significance. A positive permutation importance indicates that the feature contributes positively to the model’s predictive accuracy – shuffling it degrades performance. Conversely, a value near zero suggests minimal impact. In contrast, negative values imply that the feature might introduce noise or mislead the model, as the model performs better when the feature is randomized. This can occur in cases where the feature is weakly informative or interacts negatively with other variables.

2. SHAP. This method, grounded in game theory, allocates a contribution value to each feature for every model prediction. It calculates the contribution of each feature for each prediction by considering all possible combinations of features. Ultimately, it leverages game theory to assign a value to each feature, representing its average contribution across all scenarios. The sign of the SHAP values indicates the direction of the feature’s effect, depending on whether a feature increases or decreases the model’s prediction relative to the baseline; the magnitude of the SHAP value reflects the strength of the feature’s influence on the prediction.

While both permutation importance and SHAP values offer valuable insights into model interpretability, they present certain limitations, particularly in the context of clinical data. Permutation importance can be sensitive to correlated features, potentially underestimating the importance of variables that share predictive power or may produce unstable results in smaller datasets or when dealing with high-dimensional data. On the other hand, SHAP values, despite providing more granular and consistent feature attributions, assume feature independence and can be computationally intensive.

Following the methodology outlined in the previous section, we conduct a 10-fold cross-validation with three repetitions. Both feature importance techniques are applied to the resulting 30 subsets throughout this process, and the mean outcomes are calculated.

By utilizing feature importance techniques, we aim to identify which features play the most pivotal role in determining the predicted survival outcomes for each model. This analysis enables us to assess the consistency of results across various models and to ensure that the most significant attributes identified correspond with established medical literature on the key factors influencing patient prognosis following aHSCT.

In conclusion, the analysis of feature importance is essential not only for validating the predictive capabilities of the models but also for enhancing their interpretability and ensuring alignment with clinical knowledge. This exploration provides deeper insights into how effectively the models identify the most relevant factors influencing patient outcomes, ultimately facilitating more informed clinical decision-making.

4.1 Survival models: C-index comparison

Table 2 displays the mean C-index results obtained from training and evaluating the models through 10-fold cross-validation with three repetitions, along with the hyperparameters selected during the optimization process. These results reflect the average performance across the 30 individual splits for each of the six models.

Overall, the C-index comparison results indicate that all SA models demonstrate a commendable level of concordance, with minimal variation observed among them. Each model consistently achieves a performance score exceeding 0.82, highlighting their efficacy in predicting patient outcomes based on OS.

4.2 Feature importance analysis

While all models demonstrated strong performance according to the C-index, it is crucial to conduct a comprehensive analysis of how these models interact with the attributes in our dataset.

Comparing the results of feature importance across different models, we can identify whether certain clinical parameters consistently emerge as significant predictors of risk, and whether these findings align with established research on aHSCT. In this study, the ISS, history of disease progression or relapse, and the number of previous treatment lines consistently emerged as strong predictors across multiple models, supporting their recognized roles as critical risk factors in the clinical literature. This reinforces the reliability of the models in identifying key determinants of survival outcomes. Therefore, it is vital to verify if these attributes are similarly represented in the models’ behaviors. Any inconsistent or unexpected results may warrant further investigation into potential biases or limitations within the models.

Tables 3 and 4 present the ranges of values obtained from the two feature importance techniques applied to all models.

The findings across all evaluated models reaffirm the critical impact of key clinical variables, particularly the ISS, disease progression or relapse, and the number of treatment lines, on survival outcomes following MM aHSCT.

However, some deviations from expected patterns were observed. For example, while age is traditionally considered a significant factor in transplant outcomes, certain models assigned it a lower relative importance compared to other clinical variables. This could suggest that age may have a more nuanced impact when considered alongside other comorbidities or treatment factors.

While models based on gradient boosting achieve strong C-index scores, they tend to focus on a limited set of key attributes, potentially neglecting clinically relevant variables. This narrow focus can reduce the model’s ability to differentiate between patients with similar characteristics, limiting its effectiveness in complex scenarios. These findings highlight that, beyond traditional performance metrics like the C-index, evaluating the interpretability and clinical relevance of feature importance is essential for robust risk stratification.

Unexpected patterns or model inconsistencies could indicate areas where data complexity, feature interactions, or sample size limitations influence the model’s behavior. Addressing these gaps through further research could enhance the accuracy and clinical applicability of predictive models in HSCT.

Consequently, although a preliminary analysis based solely on the C-index may suggest that any of these models could be appropriate for clinical application, a more thorough examination of feature importance reveals that models like Coxnet or those utilizing decision trees (e.g., RSF) offer a more balanced and comprehensive analysis of all available data, allowing for a more nuanced interpretation of patient characteristics.

4.3 Risk stratification

To evaluate the effectiveness of the SA models, we randomly selected four patients from the dataset and performed risk predictions following their aHSCT. The predictions can be validated since we have access to the actual status and survival times for these patients. These individuals were excluded from the training set to assess the generalization performance of our predictive model. Table 5 summarizes the key characteristics of the four selected patients.

Considering the values of their parameters and the most influential attributes for overall survival identified in the preceding section, we can anticipate a favorable prognosis for patients 1 and 2, a moderate risk for the third patient, and an unfavorable prognosis for the final patient due to a high ISS, advanced disease stage, and a history of relapse post-transplantation. Table 6 displays the predicted risk scores for each patient, with each model predicting this index on an arbitrary scale. Higher risk scores indicate a lower probability of survival following the transplant procedure.

While we observe that the predicted risk scores increase consistently across patients with each model, these scores can still be challenging to interpret and compare directly. To address this issue, we propose the following approach: for each model, we calculate the minimum and maximum possible risk values based on patients with the most favorable and least favorable characteristics at the time of transplantation. We then categorize the predicted risk scores for each of the three patients within these defined ranges. By positioning each patient within the risk spectrum of each model, we can classify them into low, medium, or high-risk levels. This process is visually illustrated in Figure 2.

Figure 2:

Predicted risk scores for patient 1 (green), patient 2 (yellow), patient 3 (orange) and patient 4 (red) within the range of risks for each model with the best and worst diagnosis.

After implementing this procedure, where each model determines the minimum and maximum risk values by testing all possible combinations of parameters from our dataset, and subsequently positioning the initial predictions for our test patients within these ranges, we can draw several conclusions. These individualized risk assessments provide clinicians with a data-driven tool to evaluate the potential benefits and risks of proceeding with aHSCT, facilitating more informed decision-making tailored to each patient’s unique clinical profile. These results are consistent with the criteria set by the physicians and the actual outcomes observed for these patients.

The models corroborate the lower risk expectation for Patients 1 and 2, who were anticipated to have a lower risk post-transplant. Both patients remain alive for 47 and 26 months following the procedure, respectively. The transplant has benefited these individuals, leading to improved survival outcomes as anticipated.

In the case of Patient 3, who passed away a few years after the operation, the models suggest that the intervention was still advisable, albeit with additional precautions. Although there was an associated risk, the procedure presented a reasonable chance of success, contingent upon careful monitoring and potential post-operative interventions.

Conversely, for the last patient, who exhibited more unfavorable characteristics at the time of transplantation and had a history of relapse following previous transplants, the outcome was markedly different. This patient succumbed within a year after the procedure. The models predicted that proceeding with the transplant would not be advisable in this scenario, as the likelihood of obtaining significant benefits from the treatment was minimal.

Critical ethical and medical considerations are raised when the predicted survival time with the operation does not substantially differ from the expected time without intervention. In such instances, the burden of undergoing an intensive procedure like aHSCT, combined with the associated risks and costs, may outweigh the potential benefits, making a non-interventional approach a more compassionate and medically sound decision.

The integration of these risk stratification results into clinical workflows can significantly enhance the decision-making process. By embedding risk scores into clinical practice, physicians could receive automated risk assessments alongside standard patient data, enabling real-time evaluation during multidisciplinary transplant board meetings. These risk predictions can aid in patient selection, inform treatment planning, and help prioritize post-transplant monitoring strategies based on individual risk profiles. In high-stakes procedures like aHSCT, where the balance between potential benefit and risk is critical, the use of predictive risk scores can contribute to more personalized, transparent, and ethically sound clinical care.