Augmenting heart disease prediction with explainable AI: A study of classification models

1 Introduction

Heart disease significantly contributes to mortality, but it is crucial to emphasize that not all individuals diagnosed with heart disease face fatal outcomes. Timely and precise diagnosis and appropriate treatment can empower those dealing with heart diseases to lead fulfilling lives well into their later years, substantially enhancing their overall quality of life. Recent advancements have introduced potent tools like machine learning (ML) classification models, which have become instrumental in delivering accurate heart disease diagnoses. These models harness patient data to provide promising predictions about heart disease outcomes. In our ongoing research, we have harnessed these classification models using a comprehensive heart disease dataset. A central focus of our work revolves around ensuring the fairness and reliability of the models we develop. To achieve this, we have integrated explainable artificial intelligence (XAI) techniques, focusing on the conformal classifier, designed to identify the most dependable model. Our study extensively analyzes diverse classification models, including logistic regression, K-nearest neighbors (KNNs), decision trees (DTs), XGBoost, and CatBoost classifiers. The results explicitly highlight the outstanding performance of boosting algorithms, both XGBoost and CatBoost, displaying highly promising metrics. Furthermore, these models have successfully pinpointed the type of chest pain (e.g., TA, typical angina, ATA, atypical angina, NAP, non-anginal pain, or ASY, asymptomatic) and the slopes of the ST segment of the PQRST ECG wave as vital indicators of heart disease onset. Our findings provide evidence that the XGBoost and CatBoost classifier models, in particular, exhibit significantly higher coverage and lower set size, affirming their fairness and suitability for deployment in heart disease prediction.

2 Literature review

The results of the work by Bhatt et al. [8] underscore the promising capabilities of k-mode clustering in precisely predicting heart disease, indicating its potential as a valuable asset in shaping tailored diagnostic and treatment approaches for this ailment. Nagavelli et al. [14] adopted naive Bayes with weighted approach-based prediction, support vector machine (SVM) with XGBoost-based prediction, an improved SVM based on duality optimization technique-based prediction, and an XGBoost-based prediction for heart disease detection. Their analysis showed that the XGBoost algorithm has high-performance metrics, indicating its suitability. Tao et al. developed an automated system for detecting and pinpointing ischemic heart disease (IHD). The findings of Tao et al. [19] showed that the synchronicity of T-wave repolarization is a crucial factor in distinguishing IHD from normal subjects. Further, magnetic field patterns are closely linked to the location of stenosis in the heart. Mohan et al. [13] introduced an innovative approach that leveraged ML to identify crucial features, thereby enhancing the accuracy of cardiovascular disease prediction. Their model, hybrid random forest (RF) with a linear model, achieved an impressive 88.7% accuracy, surpassing other feature combinations and classification methods. Fitriyani et al. [9] introduced an advanced heart disease prediction model (HDPM) combining density-based spatial clustering of applications with noise, synthetic minority oversampling technique and edited nearest neighbor, and XGBoost-based ML techniques to enhance prediction accuracy. When validated on two publicly available heart disease datasets, their model outperformed existing models with an accuracy of up to 95.90 and 98.40%. In addition, they integrated HDPM into a heart disease clinical decision support system for efficient diagnosis and data management, offering potential benefits for clinicians and healthcare practitioners.

Baviskar et al. [7] integrated ensemble learning techniques with feature variance maximization that excelled in binary and multiple disease classification, surpassing existing models by 3.4% in the latter. These enhancements enabled efficient classification across various datasets and clinical applications. Shorewala [18] assessed ML’s efficacy in coronary heart disease prediction, wherein initial models were found to give an accuracy plateau of 75%. Base models averaged 71.92% accuracy, with neural networks reaching 73.97%. Ensemble methods such as bagging, boosting (notably gradient boosting with optimized parameters at 73.89%), and stacking effectively enhanced accuracy. Stacking, combining diverse models like KNN, RF classifier, and SVM with logistic regression as the meta-classifier, achieved the highest accuracy at 75.1%. Mamun et al. [12] employed a heart disease dataset to assess the effectiveness of ML methods in predicting heart disease. Notably, three classification algorithms – KNN, RF, and DT – demonstrated exceptional performance, achieving a flawless 100% accuracy rate. Furthermore, they computed feature importance scores for each attribute using all applied algorithms except multilayer perceptron and KNN. Guleria et al. [10] studied artificial intelligence (AI) and ML’s role in healthcare, highlighting their benefits and challenges. They explored an experimental method for heart disease prediction using ML techniques, with SVM standing out, achieving an 82.5% accuracy, surpassing other classifiers. Aghamohammadi et al. [1] devised an innovative method, merging the adaptive neural fuzzy inference system with genetic algorithm (GA) for heart attack prediction, capitalizing on fuzzy rule interpretability, neural network training, and GA optimization. Applied to the Cleveland dataset, the root-mean-square error improved from 0.82754 to 0.75372. For clinical benefit, their algorithm categorized patients by heart attack risk. In addition, they introduced an importance evaluation function to gauge feature significance in prediction, systematically assessing feature removal’s impact on model performance. Alsaleh et al. [2] presented a systematic review of the application of explainable AI for predicting comorbidities. They suggested that integrating additional variables, such as social determinants of health and genetic characteristics, could lead to even more accurate and personalized predictions. Rajjliwal and Chetty [16] demonstrated that shallow ML models, such as DT, exhibit exceptional interpretability and explainability, even if their predictive performance may be relatively modest. They suggested that when detection and prediction accuracy reach acceptable levels, post hoc processing can enhance interpretability, explanations, and trustworthiness for integration into clinical decision workflows. Anand et al. [4] applied deep neural networks to the PTB-XL ECG dataset for cardiac disorder detection. Their proposed model outperformed existing results with a 93.41% area under the receiver operating characteristic curve (AUC). They also tested this architecture on another arrhythmia ECG dataset, achieving 95.8% accuracy and a 99.46% AUC, competitive with state-of-the-art models. They assessed the model’s interpretability using Shapley additive explanations and established its ability to highlight relevant ECG wave alterations for diagnostic purposes. Pawar et al. [15] recommended integrating XAI techniques in healthcare to harness benefits such as improved transparency, traceable outcomes, and model refinement. The proposal suggests harnessing well-established XAI models with clinical expertise to optimize the advantages within AI-driven healthcare systems. Sheu and Pardeshi’s [17] XAI survey article provided a comprehensive approach to advancing research in XAI. Their work addressed legal and ethical concerns, focusing on vital aspects of medicine, all aimed at bolstering user trust through enhanced transparency. In their study, Amann et al. [3] utilized AI-powered clinical decision support systems as a case study, adopting a multidisciplinary approach to evaluate the significance of explainability in the context of medical AI. Their research demonstrated that various perspectives, including technological, legal, medical, and patient viewpoints, required distinct fundamental considerations and values to grasp explainability’s role in clinical practice fully. Band et al. [6] investigated the potential of AI and ML in reshaping healthcare, highlighting their applicability and dependability in various applications, such as using XGBoost for mediastinal cysts and tumor detection, 3D brain tumor segmentation, and the trace method for medical image analysis. Their research contributed to the growing domain of XAI, providing valuable insights for healthcare professionals. In addition, the article delved into the performance of XAI techniques in medical systems and elucidated their implementation.

3 Problem identification and research objectives

Building upon the ongoing endeavors to merge XAI with ML classification models, our research aims to construct ML models utilizing a labeled dataset related to heart disease sourced from Kaggle [11]. The description of the dataset is in the Appendix.

Given the availability of labeled data, we focused on supervised ML techniques. Since the target variable is categorical, we naturally gravitated toward employing classification algorithms to gain deeper insights into the dataset. In the realm of classification algorithms, we can classify them into two broad categories: eager learners and lazy learners. In our study, we decided to explore both categories: specifically, we selected logistic regression and decision tree algorithms within the eager-learner category, which are known for their proactive approach to building a model based on the entire dataset. On the other hand, in the lazy-learner category, we chose the KNN algorithm, characterized by its more adaptive, instance-based approach, relying on local patterns in the data. We intend to assess and compare the performance of these selected algorithms on the heart disease dataset. By doing so, we aim to comprehensively understand their effectiveness in predicting heart disease, ultimately contributing to our knowledge of this critical medical condition.

The following research question is put forth as the primary goal of the analysis: to construct ML models using the dataset.

4 Proposed approach

To begin with, we obtained a dataset on heart disease from Kaggle [11]. We employed it to train several ML classification models, including logistic regression, KNN, DTs, XGBoost, and CatBoost. Subsequently, we utilized accuracy, precision, and F1-score as performance metrics to identify the most suitable model to make predictions. Subsequently, we utilized the Mapie conformal classification algorithm to evaluate the reliability of the developed ML models. Ultimately, we turned to XAI algorithms to extract meaningful interpretations from the results.

We present the following flowchart (Figure 1) outlining the key elements of the current workflow. This diagram shows the “DATA” component, representing the heart disease dataset from Kaggle [11]. This dataset serves as the input for the subsequent component, the ML classification models. As mentioned in Section 3, our approach involves investigating eager and lazy learning models to assess their suitability for the current dataset. These models encompass logistic regression, KNN, DT, XGBoost, and CatBoost. Following an evaluation of performance metrics, the most appropriate model is advanced to the next prediction stage. At this point, we forecast the target variables based on the selected model and determine the feature importance. This information is transferred to the subsequent component, XAI. We employ the conformalized quantile classification model. It is worth noting that standard learning algorithms typically offer binary YES/NO classifications without any measure of uncertainty. On the other hand, the conformalized quantile classification model regulates prediction certainty through two primary measures: coverage score and prediction set size. Subsequently, we employed an explainable classifier that aligns with the most reliable model pinpointed by the conformalized quantile classifier to render the obtained outcomes interpretable. The ultimate stage of this study involves presenting definitive findings and offering valuable recommendations.

Figure 1

Workflow diagram.

5 Results and discussion

The performance metrics generated through the Python code we developed for the classification models are illustrated in Figure 2. Notably, the KNN classifier appears less suitable for the current dataset, with accuracy, precision, and F1-score for both the train and test sets falling within the 70–78% range. Conversely, the other models exhibit greater reliability regarding these metrics than the KNN model. However, when examining the statistics on false positives (FPs) and false negatives (FNs), as presented in Table 1, it becomes evident that KNN and DT struggled to accurately identify the presence or absence of heart disease. It is worth noting that a higher number of cases identified as FN can be detrimental to the recommendation system, constituting a critical failure.

Figure 2

Performance metrics.

In Figure 3, we display a bar chart that visually represents the degree of importance of each feature on the target variable. Analyzing this chart in the context of our dataset, we can recognize that the features “(#2) Chest Pain Type” and “(#10) ST Slope” emerge as the most influential factors contributing to the occurrence of heart disease. It suggests that, within the dataset, these two particular features play a substantial role in determining the presence or absence of heart disease.

Figure 3

Feature importance chart.

The subsequent step in the workflow diagram involves the incorporation of conformality quantile classification model. This approach yields a binary prediction and furnishes an estimate of the prediction’s confidence or probability level. Two pivotal metrics are linked to this approach: the coverage score and set size. The coverage score quantifies the ratio of correctly classified cases, and a model characterized by a good coverage score and a minimal prediction set size is deemed reliable and trustworthy.

This article employed the Mapie conformal quantile classification across all the ML models considered for traditional classification tasks. The schematic representation of this technique is depicted in Figure 4. As depicted in the framework, the first stage entails splitting the dataset into training and testing subsets, using a chosen division ratio of 90% for training and 10% for testing. A further division is made into calibration and primary training sets within the training set, distributed in an 80–10% ratio. The ML models, including logistic regression, KNN, DT, Catboost, and XGboost, are trained using the 80% dataset. The calibration dataset is utilized to fine-tune the model parameters, and the resulting calibrated models are subsequently evaluated for final predictions across different significance or quantile levels ( α = 0.05 , 0.1 , and 0.2).

Figure 4

Conformal quantile classification framework.

Figures 5, 6, 7, 8, 9 display the model predictions. In these figures, the top-left plot represents the prediction generated by the base estimator, while the remaining three subplots each illustrate a distinct α score. Dots that appear in colors other than dark purple indicate instances where the models struggled to predict the outcomes. Therefore, selecting a suitable value of α is essential because the predictions are predominantly represented in dark purple. Figure 5 shows a lack of substantial differentiation between the classifications within these quantiles for the logistic regression model. In the case of the KNN and other models, the α = 0.05 level exhibits minimal classification errors (Figures 6–9).

Figure 5

Predictions from the logistic regression model.

Figure 6

Predictions from the KNN model.

Figure 7

Predictions from the decision tree model.

Figure 8

Predictions from the XGBoost model.

Figure 9

Predictions from the CatBoost model.

Further, we used the explainable boosting classifier to enhance result interpretability, driven by its high predictive accuracy. It unveils critical factors, making our findings more explicit and essential for real-world applications and medical communication.

Figures 10, 11, 12, 13 present predictions based on patient-specific data from the dataset, offering an insight into how the model’s predictions relate to individual features. For example, Figure 10 highlights the prominent influence of a flat ST_Slope (encoded as 1), followed by a high level of fasting blood sugar (120 mg/dl). Interestingly, despite the absence of chest pain symptoms related to heart disease, the predictions favor the presence of heart disease. The other contributing features are represented by bars in the color orange. The prediction in Figure 11 shows that a higher heart rate does not indicate heart disease.

Figure 10

Predictions for a patient’s data falling into class 1.

Figure 11

Predictions for a patient’s data falling into class 0.

Figure 12

Predictions for a patient’s data falling into class 0.

Figure 13

Predictions for a patient’s data falling into class 0.

Figure 14

Feature importance from XAI boosting algorithms.

Figure 12 demonstrates an instance of misclassification, potentially linked to the significance of particular symptoms (such as Flat ST_Slope, asymptomatic chest pain, and moderate ST_Depression or old peak) highlighted as crucial features by the boosting algorithms (refer to Figures 3 and 14). Figure 13 depicts another misclassification scenario where a patient diagnosed with heart disease is erroneously categorized as being healthy. This discrepancy may be attributed to the upsloping of the ST curve (designated as “2”), a notable feature associated with a healthy individual. This highlights that although boosting algorithms have been deemed the most reliable by the conformality quantile classification model, this predictive tool is not entirely foolproof [5], thus necessitating expert intervention in decision-making.

Figure 15

Interaction of age and max heart rate.

6 Conclusions and recommendations

Based on the available dataset, this study combines conventional classification models with XAI to draw insights regarding an individual’s health status. The analysis leveraged the conformalized quantile regression model to interpret outcomes generated by the logistic regression, KNN, DT, Xgboost, and Catboost algorithms. Notably, the boosting algorithms demonstrated a remarkable ability to offer binary classifications with 97% confidence for the specific dataset. A detailed analysis of the patient-specific dataset is carried out using explainable boosting algorithms, and the results are presented and discussed. After conducting a comprehensive global term analysis of the provided dataset, we suggest the following recommendations based on Figures 15 and 16. For individuals aged 60 years and above, a heart rate of more than 150 bpm is a high-risk factor. It is recommended for those individuals to maintain a heart rate in the range of 120–150 bpm (Figure 16). Furthermore, individuals in the age group 30–35 years who have a heart rate in the range of 110–150 bpm need to be cautious, and it is recommended to maintain a heart rate below 110 bpm. For individuals who are 63 years and older, having a cholesterol level exceeding 350 mg/dl indicates a significant risk. In addition, middle-aged individuals between 30 and 55 years with lower cholesterol levels (0–40 mg/dl) are also at risk of developing heart disease. It is advisable to maintain cholesterol levels within the normal range to prevent potential complications (Figure 16). In conclusion, our approach will sustain trust and facilitate decision-making within the healthcare domain, where precision, reliability, and transparency hold utmost significance. In the future, the research emphasizes collaborative efforts between researchers, clinicians, and data scientists, aligning with the evolving landscape of healthcare research. Moreover, considering the dynamic health data over time with emerging technologies, federated learning comes into play to address privacy issues.

Figure 16

Interaction of age and cholesterol.

Limitations: It is crucial to emphasize that the findings presented in this study are founded upon ML models trained with the currently available dataset. Over time, a balanced and dynamic health dataset is essential to enhance the reliability of these results. Otherwise, the implications of misclassifications in real-world clinical settings underscore the importance of robust diagnostic processes, evidence-based decision-making, and continuous quality improvement initiatives in the healthcare sector.