A generalized deep learning model for heart failure diagnosis using dynamic and static ultrasound

Introduction

Heart failure (HF) is a chronic progressive condition that imposes a heavy disease burden and affects nearly 24 million people worldwide, with a 5-year mortality rate of approximately 50%.^[1] Therefore, accurate diagnosis, longterm observation, and a “before-and-after” comparison of HF conditions are crucial. One critical indicator is the cardiac ejection fraction (EF, output per beat as a percentage of the end-diastolic volume of the ventricle).^[1] Cardiac ultrasonography is the most commonly used clinical tool for the diagnosis of HF. However, it relies on the operator’s experience; hence, the examination results may not truly reflect the patient’s condition.^[2,3] Additionally, training specialized ultrasonographers or cardiologists who can competently perform the examination does not adequately meet the growing demand for testing.^[4,5] Patients in remote areas often need to travel to cities with concentrated medical resources for medical care because of the uneven distribution of medical resources across regions.^[6,7] The difficulty of communicating, carrying, and displaying dynamic ultrasound images across regions as static images could prevent a comprehensive understanding of the patient’s disease progression.

To the best of our knowledge, studies reported to date that use artificial intelligence (AI) methods to assist in the diagnosis of HF have primarily focused on dynamic ultrasound.^{[8, 9, 10, 11]} Studies that classify static ultrasound images or are compatible with dynamic and static ultrasound data are still lacking. Therefore, the aim of this study is to demonstrate the feasibility of this approach to integrate multi-dimensional data and to provide more ideas for subsequent studies. Overall, the findings improve the accuracy of cardiac ultrasound, and help physicians and patients to better manage the progression of HF.

Materials and Methods

Static recognition capability of the model

Data were allocated and models were trained separately using two schemes to explore the ability of the model to be applied in migration applications on static datasets.

r2plus1d-Pan1: The CAMUS dataset was not involved in training and the model was applied directly to perform the classification tasks.

r2plus1d-Pan2: 80% of the CAMUS dataset was involved in training. Additionally, the model was applied directly to 20% of the validation set and compared with human experts.

1, 2, 4, and 8 static images from the same ECG were randomly used as input while the classification task was performed to observe the classification accuracy of the two models.

Results

Data distribution and dynamic recognition capability of the model

The CAMUS dataset (Figure 1D) and EchoNet-Dynamic dataset (Figure 1E) were visualized separately. The data in both datasets with EF < 40 and EF ≥ 40 were mixed and were difficult to segregate from each other.

The model achieved an AUC value of 0.95 (95% confidence interval [CI]: 0.947 to 0.953) on the dynamic EchoNet-Dynamic dataset with increasing training (Figure 1B). Moreover, the stable convergence proved that training was sufficient. An AUC value of 1 (95% CI: 1 to 1) was achieved on the local independent validation dataset (Figure 1C). These results suggest that the model had excellent classification ability on the dynamic ultrasound dataset. The complete model structure can be obtained from Figure S1.

Static recognition capability of the model

The classification accuracy (ACC) values of the r2plus1d-Pan1 model were 0.66, 0.70, 0.75, and 0.74 with 1, 2, 4, and 8 images, respectively. By contrast, the classification accuracy values of the r2plus1d-Pan2 model were 0.85, 0.81, 0.93, and 0.92 with 1, 2, 4, and 8 images, respectively. The classification accuracy of the r2plus1d-Pan2 model was significantly higher than that of the r2plus1d-Pan1 model (P = 0.003, Figure 2). This is because the r2plus1d-Pan2 model uses 80% of the total data from the CAMUS dataset for training and is more "adapted" to static ultrasound data than the r2plus1d-Pan1 model. This suggests that even if the compatibility of dynamic and static ultrasound data is achieved, further research is required to understand the practical applications of the r2plus1d-Pan2 model as it would require a large amount of co-learning of both types of data.

Figure 1

Partial structure of the model, data visualization, and training effect on dynamic data. A. Structure diagram of the input and output of the model. B. Variation of AUC of the model with the number of trainings on the dynamic dataset. C. Classification effect of the model on the local dataset. D. Visualization of the CAMUS dataset. E. Visualization of the EchoNet-Dynamic dataset.

Figure 2

Classification effect of the model on the static dataset. A–D. r2plus1d-Pan1 classification effects when 1, 2, 4, and 8 static images from the same echocardiogram were randomly input, respectively. E–H. r2plus1d-Pan2 classification effects when 1, 2, 4, and 8 static images from the same echocardiogram were randomly input, respectively.

Discussion

The objectification, standardization, and paper-based nature of cardiac ultrasonography are necessary for clinical work.^[14] For example, a patient with HF needs to travel from an area of scarce medical resources to a relatively abundant area. In this case, the patient could carry only a paper ultrasound report with some printed images. However, upon arrival at the destination, the patient often does not have immediate access to such services because the demand for cardiac ultrasound examinations at each medical unit is greater than the supply. As a result, physicians often have to rely on available examination reports for processing and wait for the results of their own unit’s ultrasound examinations. Because ultrasound is highly influenced by operator subjectivity, results given by different operators can vary, thereby resulting in inappropriate medical measures during the early stages of treatment.^[2,3]

Figure 3

Comparison of the classification results between all human experts and artificial intelligence on static data. A-D. Classification results using 1, 2, 4, and 8 graphs at a time. Artificial intelligence models are indicated using arrows. H1: specialists who have been working in cardiology specialties for less than 1 year; H1-3: specialists who have been working in cardiology specialties for 1–3 years; H3+: specialists who have been working in cardiology for more than 3 years; U1: specialists who have worked in ultrasound for less than 1 year; U1-3: specialists who have worked in ultrasound for 1–3 years; U3+: specialists who have worked in the ultrasound specialty for more than 3 years.

The information deficit caused by missing dimensions is a major obstacle in achieving compatibility between static and dynamic data. If dynamic video is considered as a series of consecutive frames, it contains information as both frames and sequences. A frame could contain spatial information at that moment, whereas a sequence would contain the temporal variation of this spatial information. However, static images lose temporal information. The necessity of temporal information for accurate action recognition and whether the static frames of a sequence contain this information is still debated.^[13,15]

The degradation of classification caused by missing temporal information can be compensated for using model restructuring and training with multiple data types. The classification accuracy of r2plus1d-Pan2 is significantly higher than that of r2plus1d-Pan1 (P = 0.003), which suggests that the classification capacity of the model could be significantly improved by supplementing the relevant training data. Compared with AI models that are compatible with either dynamic or static ultrasound data, the models that are compatible with both static and dynamic data have several advantages. They could be used for a wider range of applications in paper-based examination reports (static) and electronic device (dynamic) settings. Better standardization of cardiac ultrasound examinations reduce the subjective influence of the operator and variations in examination results in different regions with different levels of healthcare resources. They compensate for the reduced classification effect due to the missing temporal information while making the conversion from dynamic to static data.

However, there are some limitations to the practical application of the model compatible with both static and dynamic data. Achieving compatibility with multimodal data would require a large amount of data to train and further improve the performance of the AI model and higher training costs. AI-assisted diagnosis and treatment models can only provide reference advice to clinicians but cannot replace human experts in diagnosis and treatment. The actual environment in which the model will be applied would be far more complex than the laboratory environment. This, in turn, requires further training to improve stability.

There are several highlights in this study. First, this is the first deep learning model compatible with dynamic and static ultrasound, and it can accurately perform HF diagnosis. The model outperformed most human experts from the National Cardiovascular Centers who have worked in ultrasound and cardiac specialties for more than 3 years. Second, the proposed model is the largest trained AI ultrasound diagnostic model. Its training and human-expert comparisons were performed using a multi-center study approach, thus confirming the stability of the results.

This study has some limitations. First, larger-scale multi-center data are needed, including more application populations, data types, and data collection devices. Second, the model needs further lightweighting because it is too large, with a file size of 57.2 GB.

In this study, a new deep spatio-temporal convolution model was constructed. The proposed model accurately identified patients with HF with reduced EF (< 40%) using dynamic or static cardiac ultrasound images. The model outperformed most senior specialists in diagnostic performance, and thus, it can improve overall HF diagnosis. Moreover, it allows patients to be referred and reexamined with a portable static ultrasound report, thereby reducing medical risk and saving healthcare resources.