A multimodal deep learning-based algorithm for specific fetal heart rate events detection

Data set and annotation source

The dataset used in this study is derived from the publicly available CTG database of the Czech Technical University-University Hospital in Brno (CTU-UHB) [33]. This dataset contains 552 original signal recordings, each lasting up to 90 min, and includes FHR time series, UC signals, as well as detailed clinical information on the mother, labor, and fetus. All signals were sampled at a frequency of 4 Hz.

The annotations used in this study were obtained by a gynecological expert with the support of a CTG analyzer [29]. The annotations include the start and end points of specific CTG events detected in the FHR and UC signals. The annotated events in the FHR signal are bradycardia, tachycardia, acceleration, and deceleration, represented by BC, TC, ACC, and DEC, respectively. The UC annotations indicate uterine contractions. The dataset also reports the classification of each deceleration associated with uterine contractions as early, late, variable, or prolonged, represented by E, L, V, and P, respectively.

Data preprocessing

The first step in data preprocessing is to verify that the length of the annotation files matches the source files. We found that 9 out of 552 files had mismatched lengths between the annotation files and the source files; these were deemed unsuitable for annotation and were thus excluded to ensure accurate subsequent processing.

During actual clinical monitoring, the CTG signals often encounter various noise interferences due to fetal movements, poor probe fixation, or changes in the maternal position [34], 35]. Interference noise in FHR signals typically manifests as spiky artifacts and missing values [36].

First, the zero-value data in the FHR signals were removed. Next, the absolute difference between adjacent points was calculated, and unstable points with a difference greater than 10 were corrected using linear interpolation, which effectively smooths small-scale irregularities by utilizing the values of stable points before and after these points. For extreme values exceeding 200 or below 50, corrections were made using the Hermite spline interpolation method. Segments with extreme values lasting longer than 2 s were removed directly to maintain the authenticity and accuracy of the data. Finally, a median filter was applied to smooth the data, reducing noise within the signal. The smoothing window length was set to five sampling points. The effectiveness of this preprocessing scheme is illustrated in Figure 1.

Figure 1:

Comparison of raw and preprocessed data. The (a) graph shows the raw data, while the (b) graph shows the preprocessed data, with different colored lines representing various specific FHR events.

FHR signal feature extraction

This study aims to identify specific CTG events by extracting multidimensional features from FHR signals. This section describes all the extracted features and their calculation methods, categorized into basic statistical features, morphological features, HRV time-frequency domain features, and nonlinear features [37], [38], [39], [40], [41], [42].

Time-frequency domain feature extraction

Studies have shown that transforming signals into images for processing is more effective in predicting fetal hypoxia than processing time-domain signals alone [21], 52], 56]. Wavelet transform is a signal processing technique that decomposes signals into different frequency components and analyzes them at various time scales, which is extremely useful in medical and biological signal processing [57]. In this study, we applied wavelet transform to convert time-domain signals into time-frequency domain signals, normalized them to a uniform size, and finally used deep learning models for learning and classification [58].

Data composition

Before using deep neural networks to learn and classify different specific events, it is necessary to extract or segment the specific FHR data. The average durations of BC, TC, ACC, and DEC events were calculated as 2,787, 5,196, 289, and 259 samples, respectively. Consequently, different extraction and segmentation methods were applied for each category. For BC and TC data, which represent bradycardia and tachycardia respectively, a 50 s window (200 samples at a 4 Hz sampling rate) is sufficient to reflect these physiological states. Therefore, a sliding window approach with a window size of 200 samples was used to segment these data. For ACC and DEC categories, the data were segmented based on the annotations and unified to a length of 200 samples.

After processing, we obtained 2192 DEC data segments, 795 ACC data segments, 3046 BC data segments, and 5103 TC data segments. To ensure data balance during classification, the same amount of data without specific events was selected as a comparison. Figure 2 illustrates the four types of specific data within a segment of FHR signals.

Figure 2:

The four types of specific data within a segment of FHR signals. (a) Shows a bradycardia event, (b) shows a tachycardia event, (c) depicts a deceleration event, and (d) displays an acceleration event. The top part represents FHR, and the bottom part represents UC.

Model fusion method

Following the feature extraction process, 110 features were calculated for each data segment. However, some features may not be as informative as expected, and there might be overlapping information among features. Therefore, feature selection algorithms are required to identify the most valuable and informative features. This step enhances computational efficiency, improves model performance, and increases model interpretability [59].

We utilized statistical testing methods for feature selection, specifically using the independent samples t-test to assess the differences between positive and negative sample data. Features were selected based on their p-values, retaining only those with p-values less than 0.05, indicating significant differences [60]. Through this process, we identified 62 significant features for ACC classification, 83 for BC classification, 90 for DEC classification, and 91 for TC classification.

In the four categories (ACC, BC, TC, DEC), the following feature numbers were found to exhibit significant differences: 1–5, 7, 9, 20–26, 28–30, 32–36, 39, 40, 52, 53, 62–65, 72–76, 78, 84, 86, 89, 90, 93, 95, 96, 99, 102–105. Among these, numbers 1 to 9 represent basic statistical features such as maximum value, minimum value, mean, median, standard deviation, skewness, and the number of peaks. Numbers 20 to 40 encompass features related to signal envelopes, including the number of peaks and troughs in the envelope width, as well as the maximum, minimum, mean, standard deviation, and kurtosis of both the upper and lower envelopes. Numbers 52 and 53 correspond to the maximum and minimum slope of the lower envelope trend line. Numbers 62 to 65 represent baseline statistical features, including maximum, minimum, mean, and median values. Finally, numbers 72 to 105 cover HRV parameters, including the maximum, minimum, mean, and standard deviation of RR intervals, low-frequency power, very low-frequency power, total power, and the ratio of low-frequency to high-frequency power. Figure 3 presents the box plot differences of four specific features between normal FHR and special events.

Figure 3:

The distribution of four selected features across basal FHR and special events (using ACC as the representative event) using box plots.

This article proposes a novel deep feature fusion network for evaluating specific FHR events. This network takes as input the extracted data features and the time-frequency domain FHR signals obtained through wavelet transform. We designed and integrated several deep learning models, including ResNet1D, ModifiedResNet, and MultiAttentionModel, for feature extraction and fusion from different data representations.

The ResNet1D model employs one-dimensional convolutional neural networks (1D-CNNs) to extract features from signals such as feature values, effectively capturing relevant information. The model comprises an initial convolutional layer, multiple residual blocks, and fully connected layers. Each residual block consists of two convolutional layers, their corresponding batch normalization layers, and nonlinear activation functions.

The ModifiedResNet model is derived from a pretrained ResNet34 model, which enhances feature extraction efficiency through the use of pretrained weights via transfer learning. The initial convolutional layer in this model has been modified to accommodate single-channel input with a kernel size of 7 and stride of 2. The final fully connected layer has been replaced with a layer tailored to the specific classification task, aligning the number of output neurons with the target classes.

Additionally, the MultiAttentionModel fuses the outputs of ResNet1D and ModifiedResNet using a multihead attention mechanism, which captures complex correlations between different features for improved classification accuracy. Finally, the classification task is completed through fully connected layers. The specific network model is illustrated in Figure 4.

Figure 4:

Multi-feature fusion network model.

Results

In this study, we proposed a deep feature fusion network that effectively integrates useful features from morphological, HRV, and nonlinear one-dimensional signals, as well as time-frequency domain two-dimensional signals. This integration is achieved through multi-model joint learning and the introduction of attention mechanisms, improving the accuracy and robustness of specific FHR event evaluation. The performance of the model was comprehensively evaluated using accuracy, sensitivity, specificity, positive predictive value (PPV), and F1 score on the validation set. The definitions and equations for these metrics are as follows:

where TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively.

The experimental results indicate that our method demonstrates high detection accuracy and stability in practical applications. Table 1 shows the network results, comparing the accuracy of using single features vs. fused features for classifying ACC, DEC, TC, and BC events. As observed in Table 1, the model using fused features significantly outperforms the single-feature model across all classification tasks. For ACC event classification, the accuracy with fused features reaches 96.2 %, which is 10.2 percentage points higher than the 86.0 % accuracy obtained with single features. Similarly, for DEC, TC, and BC events, the model with fused features demonstrates superior sensitivity, specificity, PPV, and F1 score. These results further validate the superiority of multimodal data fusion and deep learning models in complex signal classification tasks, indicating that combining multiple features can significantly enhance the classification performance and stability of the model.

Data composition

pAmong the four types of specific FHR events, deceleration (DEC) is particularly important, and we focus primarily on this specific event pattern. From the source files, we accurately extracted DEC data, and the next step is to accurately identify specific deceleration patterns within DEC. According to the annotation files, decelerations are categorized into early, late, variable, and prolonged, denoted as E, L, V, and P, respectively. From all source files, we obtained 995 E-type, 181 L-type, 1012 V-type, and 43 P-type data segments.

The data volumes for the four categories vary significantly, necessitating the use of data augmentation techniques to mitigate class imbalance [61], 62]. We employed the sliding window technique to augment the data. For L and P-type data, different strides were used to generate more samples, and the data within the window were resized to a fixed size to ensure consistent input data shapes. Subsequently, we applied wavelet transform to the CTG data for feature extraction. After these operations, we obtained 995 E-type, 1086 L-type, 1012 V-type, and 946 P-type data segments, providing a balanced and rich dataset for the subsequent training of deep learning models. Figure 5 illustrates the four types of deceleration-specific events, with the four graphs from left to right representing early, late, variable, and prolonged decelerations.

Figure 5:

The four types of deceleration-specific events, with (a) representing early decelerations, (b) representing late decelerations, (c) representing variable decelerations, and (d) representing prolonged decelerations. The top part represents FHR, and the bottom part represents UC.

Multimodal signal processing method

Identifying the four types of DEC requires using both FHR and UC modalities, primarily because the determination of different DEC types relies on the characteristics of UC data. The combination of these two modalities can accurately reflect different deceleration patterns [63]. We designed and implemented the ModifiedResNet model [64], which is based on the pretrained ResNet34. This model adjusts the input channels of the initial convolutional layer to accommodate dual-channel input data and modifies the final fully connected layer to align with the classification task. Specifically, the initial convolutional layer in the ModifiedResNet model uses a convolutional kernel size of 7 and a stride of 2, accepting both FHR and UC signals as inputs by concatenating them along the channel dimension. By leveraging transfer learning, we utilize the pretrained model’s weights to enhance feature extraction capabilities and further improve classification performance. The architecture of the network model is illustrated in Figure 6.

Figure 6:

Dual-modal deep neural network architecture.

Results

Experimental results demonstrate that the ModifiedResNet model, through the joint processing of FHR and UC signals and deep feature extraction, exhibits high classification accuracy and robustness in the task of specific FHR event evaluation. Table 2 presents the four-class classification results of the multimodal deep neural network. Additionally, since late deceleration and prolonged deceleration are closely associated with fetal distress, indicating potentially adverse fetal conditions, they warrant significant attention. As the most critical monitoring indicators, we calculated the accuracy of identifying late deceleration and prolonged deceleration from all DEC-specific events. These results are also reflected in Table 2.

In practical applications, the ModifiedResNet model can effectively identify and classify different deceleration patterns (E, L, V, P) in FHR signals. This research outcome offers robust support for real-time monitoring and evaluation of fetal health status in clinical settings, facilitating early detection and intervention for potential fetal distress, thereby reducing perinatal risks.