An experimental study of U-net variants on liver segmentation from CT scans

1.1 Research gaps, motivation, and contributions

1.2 Literature review

Several deep learning-based segmentation strategies are proposed to improve the liver segmentation task. Jiang et al. [49] proposed an attention hybrid connection network (AHCNet) architecture that aims to leverage the benefits of soft attention’s continuous weighting and hard attention’s explicit selection, while also utilizing long and short skip connections. The proposed model obtained a 0.945 DSC on the 3DIRCADb dataset. Jiang et al. [50] proposed a residual multi-scale-attention U-Net (RMS-U-Net) that incorporates residual connections to counter gradient vanishing and captures inter-channel relationships among features. Tested on the LiTS [51] and the 3DIRCADb datasets, it produced DSCs of 0.9552 and 0.9697 for liver segmentation, respectively.

Hille et al. [52] introduced a hybrid network named SWTR-U-Net, combining a pre-trained residual network, a few transformer blocks, and a U-Net-like decoder path. It obtained a DSC of 0.98 on the LiTS dataset for the liver. In all these works, the utilization of the attention mechanism increases the network’s capacity to record complex anatomical details and variations in the liver structure, producing segmentation results that are more accurate and resilient.

Li et al. [53] proposed a hybrid-variable structure, namely, the RDCTrans U-Net. The backbone comprises of ResNeXt50, which is basically a ResNeXt [54] with 50 layers. Dilated convolutions increase network depth, and transformers are used in down-sampling to increase accuracy. A DSC 0.9338 is obtained for the LiTS dataset. Jiang et al. [55] proposed a multi-dimensional hybrid network, namely the MDCF Net, which had the encoding part comprising CNN and CNNFormer [56]. A slice-by-slice novel feature stacking method is introduced to combine feature maps from encoder and decoder. A DSC of 0.968 is obtained for the segmentation of liver for the LiTS dataset.

Zhang et al. [57] proposed a star-shaped window transformer based on U-Net, namely the SWTRU, where the transformer is incorporated within the decoder section. A filtering feature integration mechanism for dimension reduction is utilized. A DSC of 0.972 is obtained for liver segmentation for the CHAOS + LiTS (CHLISC) dataset. Thereby, the usage of transformers helped to capture wide dependencies and pattern sequences within the image resulting in higher DSC.

Mulay et al. [58] presented a HED-Mask R-CNN approach in order to better identify edge maps from multimodal images. The holistically nested edge detection (HED) network was utilized to obtain an improved edge map from enhanced CT and MRI images, while the mask region-convolutional neural network (RCNN) is used to segment the liver from the improved edge maps. This architecture yields a DSC of 0.94 on CT and 0.91 on MRI images based on the CHAOS challenge dataset. Roy et al. [59] introduced a novel approach where a Mask-RCNN is used to segment the liver region, and then a maximally stable extremal regions network is utilized to identify the tumors in liver. The architecture correctly classified three categories of tumors with an accuracy of 0.878. These approaches based on RCNN are designed to concentrate on specific regions of interest (in our case, the liver) within the CT scans, thereby leading to more accurate segmentation results.

Chi et al. [60] proposed a novel architecture called X-Net for 2D intra-slice feature extraction of liver and tumors by incorporating an up-sampling branch (for identification of liver region) and a pyramid-like convolution network (for extracting inner-liver features) into the backbone dense U-Net. The method uses a joint objective function by combining Dice Loss and a contour-detection-based loss. The architecture achieves a DSC of 0.971 on liver and 0.843 on tumor based on the MICCAI 2017 LiTS Challenge dataset.

Manjunath et al. [61] presented a modified U-Net methodology comprising a 58-layer architecture for LiTS. The proposed model obtains a DSC of 0.9615 on liver and 0.8938 on tumor for the LiTS dataset and 0.9194 on liver and 0.6980 on tumor for the 3DIRCADb dataset. Khan et al. [62] presented a novel architecture called RMS-U-Net incorporating residual blocks and a multi-scale strategy to deal with inter-slice features with multi-channel input images. A combination of DSC and absolute volumetric difference is used as the loss function. The network shows a DSC of 0.9731 on 3DIRCADb, 0.9738 on LiTS, 0.9739 on SLiver07, and 0.9549 on CHAOS datasets for liver segmentation. In all these research works, the use of a U-Net-based approach involving a U-shaped architecture with skip connections enables a more efficient localization of the area of interest, leading to a better segmentation performance.

2.1 Datasets

For this experimental study related to liver segmentation, we have taken into consideration three publicly available datasets, namely, 3DIRCADb, CHAOS, and LiTS. We split each of the datasets into training, validation, and test sets (80% as the training data, 10% for validation data, and 20% as test data). Only the liver masks have been taken into consideration for the purpose of liver segmentation. A mask is essentially used to highlight a desired region of interest (RoI) within a particular image. The number of images for training and testing is shown in Table 1. In this table, we have merged the training and validation as one set.

2.2 Pre-processing

The CHAOS and 3DIRCADb datasets have images present in Digital Imaging and Communications in Medicine (DICOM) format. This causes a problem for visualizing the images. Thereby, we have converted all images into .jpg format. Furthermore, for both datasets, we have accumulated all CT images into one folder, and the liver masks into another folder. For the LiTS dataset, the images and masks are present in .nii format in the form of slices. They are extracted and also converted to .jpg format which were then splitted just like the other two datasets. Then, we remove all the images and masks where the mask has no RoI, that is, the masks are completely black. This significantly resolves the class imbalance problem.

Before feeding these CT scan images to the pipeline, we implement a windowing technique called HU Clipping [64–66] upon them. The HU is a measurement scale, used especially in CT scans, to express the radio density of various tissues in the body based on their attenuation of X-rays [67–69]. The intensity of X-rays varies as it traverses through matter. Different tissues have characteristic HU ranges, which help radiologists distinguish among various structures and detect abnormalities in a CT scan. The scale considers water as the reference point, assigning it a value of 0 HU. Air [70], which attenuates X-rays very weakly, is assigned a value of − 1,000 HU, whereas dense materials like bone tissues [71], which strongly attenuate X-rays, can have HU values even above +3,000.

For all the datasets, the window width is opened in the range of [ − 200 , + 200 ], spanning a total of 400 HU ( 200 − ( − 200 ) = 400 ). By using this clipping range, tissues with HU values between − 200 and + 200 are effectively highlighted on the CT image, while all other tissues are suppressed as they fall outside the selected window. Therefore, all irrelevant structures are removed, and our area of interest, i.e., the liver stands out from the surrounding viscera in the abdomen. A glimpse of the original CT scans in .jpg format versus the HU clipped images along with their corresponding liver masks for the said datasets is shown in Figure 3.

Figure 3

Visualization of the datasets – Column 1: Original CT scans, Column 2: HU-clipped CT images, and Column 3: Given liver masks. The images are examples from publicly available datasets (3DIRCADb, CHAOS, and LiTS) used in this study, where the first and second image is from 3DIRCADb, the third image is from CHAOS and the fourth image from LiTS.

2.3 Model architectures

In this section, we describe the architecture of Vanilla U-Net and its nine other variants that are used here for liver segmentation from CT scans. The pre-processed images are fed into these models for training.

2.3.1 Vanilla U-Net

The U-Net architecture introduced by Ronneberger et al. [27] is a CNN, designed particularly for semantic segmentation problems, wherein a given image needs to be segmented into different regions or classes. The architecture, as shown in Figure 4, consists of a contraction path called encoder and an expansive path called decoder, resembling a “U,” hence the name “U-Net.”

1. Encoder: It is the left part of the U-Net model that typically consists of a series of down-sampling blocks, each of which consists of two successive 3 × 3 convolutions along with a rectified linear unit (ReLU) activation unit and then a 2 × 2 max-pooling layer.

2. The convolution layers allow the network to learn more complex and abstract features, thereby increasing the number of feature channels, while the pooling layers help to reduce the spatial dimensions of the feature maps while retaining the most important information.

3. Decoder: The right side of the U-Net contains the decoder, which uses transposed convolutions (or deconvolutions) to gradually up-sample the feature maps, thereby increasing the spatial resolution while reducing the number of feature channels. Each decoder layer consists of an upsampling operation followed by a series of convolutional layers.

4. Skip connections: The novelty of U-Net lies in the usage of skip connections that link corresponding layers in the encoder and decoder paths. In the case of basic U-Net, skip connections are implemented by concatenating the feature maps from the encoder with the up-sampled feature maps in the decoder.

The final layer of the U-Net is a 1 × 1 convolutional layer followed by an activation function (sigmoid activation in our case for binary segmentation).

Figure 4

Diagrammatic representation of the U-Net model. Source: Created by the authors.

2.3.2 Attention U-Net

In order to capture increasingly complex picture properties, CNNs analyze localized information at each layer as they are built up. Rudimentary feature maps collect contextual information, focusing on the segmentation of conspicuous foreground objects. The combination of broad and detailed forecasts for an all-encompassing result is thus made possible by the use of skip connections to combine feature maps obtained at various scales. However, it might be difficult to successfully reduce false positives for small objects with large form fluctuations. Attention gates are used here. They are incorporated within the skip connections. Attention gates do not require an ROI to be cropped between network stages because they gradually reduce feature activation in unrelated background regions. Activation coefficients help to identify the important features and prune the irrelevant ones. The features transmitted over the skip connections are judiciously refined by them. The architecture of attention U-Net [40] is shown in Figure 5.

Figure 5

Diagrammatic representation of the Attention U-Net model created by authors based on [40].

2.3.4 U-Net 3+

The U-Net 3+ [42] architecture’s usage of skip connections and deep supervision result in the segmentation map’s position-awareness, and the border identification gets increased while efficiency with fewer parameters is maintained. The architecture of U-Net 3+ is shown in Figure 6. Both intra-connections among decoder sub-networks and the encoder–decoder connections are changed by full-scale skip connections in U-Net 3+. A fusion of feature maps from various scales, including smaller and same-scale maps from the encoder and larger-scale maps from the decoder, is made and incorporated into each decoder layer. Through this integration, the model can collect minute features and an extensive semantic context at all scales, resulting in a thorough knowledge of the data. For the implementation of deep supervision, the final layer of each decoder stage undergoes a sequence of operations: first passing through a standard 3 × 3 convolutional layer, then undergoing bilinear up-sampling, and finally being subjected to a sigmoid function [77,78].

Figure 6

Diagrammatic representation of the U-Net 3+ model based on [42].

2.3.5 R2U-Net

Recurrent residual convolutional neural network based on U-Net (R2U-Net) [43] combines the potential of the vanilla U-Net, recurrent-CNN (RCNN) and residual units. This architecture consists of typical encoder and decoder pathways as Vanilla U-Net but replaces the forward convolutional blocks with recurrent convolutional layers with residual units in both the encoder and the decoder. The outputs from the recurrent convolutional blocks are passed through a residual block for a more efficient and deeper feature accumulation. This architecture design, in spite of having lesser parameters in comparison to the traditional U-Net, ensures better feature extraction using the recurrent residual convolutional layers. The model’s efficiency in terms of the number of network parameters can be attributed to the fact that the recurrent operations do not consume additional parameters. The architecture of R2U-Net is shown in Figure 7.

Figure 7

Diagrammatic representation of the R2U-Net model created by authors based on [43].

2.3.6 U 2 -Net

The U 2 -Net architecture [44] is an extension of the U-Net architecture. The architecture of the U 2 -Net is shown in Figure 8. It introduces the concept of a two-level nested U-structure, namely the top and bottom levels. The bottom-level basic building block comprises a novel ReSidual U (RSU) block. The RSU block contains a symmetric encoder and decoder architecture that helps in capturing multi-scale feature information alongside local feature information. These basic RSU building blocks have been incorporated into the encoder blocks colored green on the diagram. The decoder blocks, colored pink, have a symmetric architecture corresponding to its encoder block and take as input the upsampled feature maps from the decoder path and the feature maps obtained from its corresponding encoder block and concatenates them. These decoder blocks would generate gray-colored saliency maps, which are concatenated and passed through a 1 × 1 convolution and sigmoid activation to generate the final output, S fuse .

Figure 8

Diagrammatic representation of U 2 -Net model created by authors based on [44].

2.3.7 U-Net++

U-Net++ [45] aims to enhance the segmentation performance of U-Net by incorporating nested skip pathways and multi-scale contextual information. U-Net++ retains the U-shaped architecture of the original U-Net, but the skip connections are extended to create nested skip pathways. Here, the encoder blocks and decoder blocks are connected through a series of nested densely connected convolutional blocks, which helps to bridge the semantic gap between the encoder and the decoder. This hierarchical decoder with nested skip connections enables the integration of features from various scales, enhancing detail preservation and capturing important contextual information. U-Net++ addresses scale variability and improves segmentation accuracy by hierarchically aggregating multi-scale information, making it suitable for diverse object sizes and scene complexities. The architecture of U-Net++ is shown in Figure 9.

Figure 9

Diagrammatic representation of U-Net++ model created by authors based on [45].

2.3.8 Res U-Net

Res U-Net [46], an extension of the U-Net architecture, enhances semantic segmentation tasks by introducing residual connections within encoder and decoder blocks. Inspired by ResNets, these connections facilitate better gradient flow during training, improving feature propagation throughout the network. This integration of residual connections allows Res U-Net to efficiently capture multi-scale contextual information, enabling accurate segmentation of objects with varying sizes and complexities. By addressing challenges such as gradient vanishing and feature representation, Res U-Net proves to be a robust model for semantic segmentation tasks across diverse datasets and scenarios. The architecture of the Res U-Net is shown in Figure 10.

Figure 10

Diagrammatic representation of the ResU-Net model created by authors based on [46].

2.3.9 Swin-UNet

The swin-UNet architecture [47] is a combination of sliding window-based transformer [79], a transformer-based architecture, and U-Net used for segmentation tasks. The architecture’s central component is the swin transformer, as shown in Figure 11. This hierarchical transformer model successfully captures contextual information, both local and global. With the hierarchical design, an image is divided into non-overlapping patches at various sizes, each of which is processed separately before self-attention mechanisms enable interactions between the patches. The self-attention mechanism of the Swin transformer is utilized to capture long-range dependencies and enhance feature representations. The architecture is made to take advantage of U-Net’s skip connections to preserve high-resolution details and Swin transformer’s attention techniques to capture global context. The mask is computed across different layers and compared with the concatenated mask for computing the final loss as depicted in the figure.

Figure 11

Diagrammatic representation of the Swin-UNet model created by authors based on [47].

3.1 System configuration

A Jupyter Notebook with a 12 GB NVIDIA Tesla T4 GPU that is made available through Google’s collaborative environment is used to run the entire set of experiments. Tensorflow, Keras, Matplotlib, Scikit, Numpy, and Pandas are the main open-source modules that are used in the tests to implement the U-Net variants in Python 3.

3.2 Evaluation metrics

Metrics for evaluation provide us with the ability to determine the predictive or learning potential. It tells us how well the system performs in predicting the proper class label. In regard to the segmentation task, we consider this as a pixel-wise two-class classification problem. For two labels, namely, “Positive” and “Negative,” we have the following:

• True positive ( T P ) refers to the number of accurately categorized positive pixels.

• False positive ( F P ) refers to the number of negative pixels that are mistakenly labeled as positive.

• False negative ( F N ) refers to those pixels that are mistakenly labeled as negative despite being positive.

• True negative ( T N ) refers to the number of accurately categorized negative pixels.

Dice score coefficient (DSC): DSC [50], also known as the Dice coefficient or Sørensen-Dice coefficient, is the similarity between the discovered region and the actual ground truth region. It entails calculating the ratio of the two regions’ intersections to the sum of each region’s individual areas. Higher values of the DSC suggest better overlap and accuracy and shed light on the degree of agreement between the anticipated and true regions.

Recall: Recall [80], often called the true positive rate or sensitivity, is a performance metric that measures a model’s or algorithm’s capacity to properly identify every pertinent occurrence present in a dataset. It is determined by dividing the total number of true positives (relevant cases that the model properly detected) by the total number of true positives and false negatives (relevant examples that the model missed). Recall measures the thoroughness of the model’s output, and a high number is desired to reduce the possibility of missing important occurrences.

Precision: Precision [80] measures the accuracy of a model to correctly identify instances as relevant. It is determined by dividing the total number of relevant instances by the sum of relevant examples and false positives. The model’s ability to prevent false positives and generate precise positive predictions is highlighted by precision.

F1 score: The F1 score [80] is a metric that balances both precision and recall. It is calculated as two times the product of precision and recall divided by their sum. It is the harmonic mean of accuracy and recall. When the trade-off between precision and recall needs to be taken into account, the F1 score offers a fair assessment of a model’s performance.

Accuracy: Accuracy [80] is a fundamental performance metric that calculates the ratio of accurately predicted cases (including true positives and true negatives) to all cases in the dataset. Although it provides a broad overview of a model’s accuracy, it might not be appropriate for datasets with imbalances, where one class considerably outnumbers the other. Accuracy can be deceiving in certain situations, so other metrics such as precision, recall, and F1 score are frequently favored.

IoU: IoU [81], commonly referred to as the Jaccard index, is a metric widely used in the field of computer vision and image segmentation. By dividing the area of their intersection by the area of their union, it calculates the amount of overlap between two sets, often bounding boxes or regions. In essence, by measuring how much two items or regions coincide in comparison to how much they diverge, the IoU quantifies the degree of agreement or resemblance between them. It is frequently used to analyze how well object detection and segmentation algorithms perform, determining how closely the results they anticipate match the actual data.

VOE: VOE [50] is another measure that is frequently used in the field of medical image analysis. Its objective is to assess the accuracy of segmentation results. By calculating the percentage of the total volume of these regions that are not shared by their intersection, VOE analyzes the discrepancy between a segmented area and its matching ground truth. In essence, it measures how much the segmented region and the real region are out of alignment or in error. VOE’s value equals ( 1 − IoU ) when calculated as the complement of the IoU. A lower VOE value indicates more segmentation accuracy, whereas a value of 0 indicates perfect matching.

RVD: RVD [50] is a measure that gauges how much a segmented region’s volume differs from the related ground truth. Finding the absolute difference between the segmented area’s volume and the real region’s volume, then dividing this difference by the real region’s volume, are the steps in this calculation. RVD throws light on how much the segmented region differs from the actual region in terms of volume. Higher values imply more significant volume discrepancies between the segmentation and the ground truth, whereas values closer to 0 suggest a small volume distinction.

3.3 Loss function

We have considered Dice Loss [82] as the loss function for this comparative study. It is a typical loss function in image segmentation problems. It gauges how differently the expected segmentation mask compares with the actual segmentation mask. When working with unbalanced datasets or attempting to prioritize precise object segmentation while penalizing false positives and false negatives, dice loss is especially helpful. The Dice Loss is defined as the complement of the Dice coefficient, aiming to be minimized during the training of a model:

In the context of neural network training, the goal is to minimize the Dice Loss, encouraging the model to produce segmentations that closely match the ground truth.

3.4 Hyperparameters

Table 2 shows the values of different hyperparameters that we used for this comparative study. It should be noted that we have used the backbone as VGG16 [83] for the Vanilla U-Net and Attention U-Net models. This is mainly because for these two models, the DSC values that we obtained were less without the backbone. A backbone implies the base architecture used to extract the features of an image. The VGG16 architecture has already been trained on the large-scale ImageNet dataset. Therefore, the learned features from ImageNet help to capture the inherent feature information from the images in the dataset. The NAdam optimizer [84] used essentially combines the properties of the Adam optimizer [85] and the Nesterov momentum [84,86,87]. This helps to achieve faster convergence, adapt to the learning rates for individual parameters, and handle sparse gradients.

3.5 Results

In this section, we present and discuss the findings, as well as thoroughly examine the assessment standards that are used in this study. Table 3 provides approximate parameter counts for the variants of U-Net architecture that are used in our experimental study. The trade-off between model complexity and computational resources must be taken into account when selecting U-Net variants because larger models require more memory and processing capacity for training and inference.

The loss curves for liver segmentation on 3DIRCADb, CHAOS, and LiTS datasets corresponding to all the U-Net variants are shown in Figure 12. Since early stopping is employed during the training process with a patience of 0.001 over the validation loss, we find that for most of the architectures, the loss curves do not span for the entire 70 epochs. We can observe that there is rarely any overfitting in the models from looking at their loss curves. Additionally, the fact that the training loss curve is decreasing with an acceptable steep slope, we can say that the models are easily able to learn the proper features reducing the loss in the process. Most of the loss curves are quite stable apart from the R2-UNet model. In the case of the R2U-Net model, we can see that there are several spikes in the validation loss curve for 3DIRCADb and LiTS datasets. However, it is performing better on the CHAOS dataset.

Figure 12

Loss curves corresponding to (a) Vanilla U-Net, (b) Attention U-Net, (c) V-Net, (d) U-Net 3+, (e) R2U-Net and (f) U²-Net, (g) U-Net++, (h) ResU-Net, (i) Swin-UNet, and (j) Trans-UNet for 3DIRCADb, CHAOS and LiTS datasets. The curves depict training and validation loss, highlighting model convergence, stability, and comparative learning dynamics for liver segmentation. Source: Created by the authors.

The observed differences in performance between the three datasets prompt further investigation into the underlying factors contributing to these variations. While the 3DIRCADb and LiTS datasets may pose greater challenges, possibly due to data irregularities or class imbalances, the CHAOS dataset appears to offer a more favorable environment for this model, potentially due to its data quality or distribution characteristics.

The experimental results for liver segmentation on 3DIRCADb, CHAOS, and LiTS datasets are shown in Tables 4, 5, and 6, respectively. Here, we analyze and interpret the experimental results obtained from our study on various U-Net variants for liver segmentation. On the basis of these results, we find that all variants of U-Net show reasonably good performance on liver segmentation in terms of DSCs. The obtained DSCs lie between 0.6304 (ResU-Net) and 0.9736 (Attention U-Net and V-Net) in the case of the 3DIRCADb dataset, 0.9056 (ResU-Net) and 0.9809 (Attention U-Net) in the case of the CHAOS dataset, and 0.5516 (ResU-Net) and 0.9545 (Vanilla U-Net in case of the LiTS dataset. Thus, it is evident that Attention U-Net is the best performing model, suggesting the fact that attention mechanisms could greatly benefit liver segmentation tasks that require precise organ delineation, as it effectively highlights relevant structures even in cases of low contrast with the surrounding viscera on abdominal CT scans. Table 7 throws light on a few analytic points and fallacies observed from the results so obtained.

However, when it comes to parameter size, Attention U-Net is quite a heavy architecture with about 28 million parameters. Notably, U-Net 3+ is the lightest of all architectures in our experiment with only about 7.8 million parameters but still shows a comparative performance of 0.9725 DSC in the case of the 3DIRCADb dataset and 0.9768 DSC in the case of the CHAOS dataset. The use of full-scale skip connections and full-scale deep supervision in U-Net 3+ results in faster feature accumulation in spite of fewer parameters, which makes it computationally economical.

We have shown the resulting metric scores for the test dataset in the form of bar graphs in Figure 13. We visualize the performance of the models in the form of bar charts specific to the datasets and evaluation metrics of interest. Here, we show accuracy, precision, recall, and F1 score for the three datasets. A few good segmented output masks are shown in Figure 14.

Figure 13

Performance comparison of the different U-Net models for liver segmentation in terms of: (a) Accuracy, (b) Precision, (c) Recall, and (d) F1 score. The metrics highlight the effectiveness of each model in accurately segmenting liver regions across the evaluated datasets. Source: Created by the authors.

Figure 14

Examples of successful predictions for liver segmentation: (a) Vanilla U-Net and (b) Attention U-Net on the 3DIRCADb dataset, (c) U-Net3+ and (d) U-Net++ on the CHAOS dataset, and (e) U²-Net and (f) ResU-Net on the LiTS dataset. These results demonstrate the models' ability to accurately segment liver regions across different datasets.

3.6 Qualitative analysis

In this section, we analyze the model performances in terms of some specific aspects. In Table 7, we consider four key points that are performance comparison, boundary accuracy, anatomical variations, and efficiency and convergence. From the experimental results, it is observed that the Attention U-Net model achieves the highest DSC as compared to the other U-Net variants showing higher segmentation accuracy. On the 3DIRCADb dataset, the V-Net model also achieves the same DSC score as the Attention U-Net. However, for the LiTS dataset, the Vanilla U-Net achieves the highest DSC, while the Attention U-Net yields nearly comparable results. Capturing intricate boundaries and producing smooth and coherent segmentation results on the datasets, the U 2 -Net model produces fairly high-quality segmentation results.

It is crucial to develop segmentation algorithms that are robust and adaptable to various anatomical variants. This involves not only accurate boundary delineation but also the ability to handle irregular shapes, tissue textures, and vascular variations. Among the other U-Net variants, the Attention U-Net model is advantageous in anatomical variations on both datasets because of its ability to focus on and capture relevant features within an image. The Vanilla U-Net and U-Net3+ model easily converge with fewer number of training epochs on the datasets that make them efficient to conserve computational resources and enable quicker deployment and adaptation to various practical scenarios.

3.7 Error case analysis

The consistently lower DSC observed for the Res U-Net model across all datasets in Tables 4 and 5 suggests its limited ability to capture important features, likely stemming from its relatively low number of parameters. The model’s diminished capacity to learn intricate patterns and structures within the data due to its sparse parameterization hampers its segmentation performance, indicating the need for increasing model complexity, to bolster feature extraction and improve segmentation accuracy. However, excessive model complexity can result in challenges such as increased computational demands, longer training times, and a higher risk of overfitting, necessitating a delicate balance between architecture weight and performance.

Despite having a heavy architecture, V-Net produces better DSCs in comparison to the Res U-Net. The V-Net is primarily designed for the segmentation of medical images. It captures contextual information more widely over the input image, which results in precise segmentation. Furthermore, the usage of 2 × 2 × 2 kernels with stride 2 instead of pooling ensures the learning of relevant features and capturing meaningful patterns without losing meaningful information. It also allows the incorporation of different activation functions, which unlike max-pooling does not have any learn-able parameters.

We have generated the segmented output masks for the test images, out of which only a few fail to generate the desired results. A few erroneous outputs generated have been shown in Figure 15. The weakly determined boundary points, a few extraneous-determined pixels, as well as unsegmented pixels, have been highlighted in this figure. The V-Net model uses 5 × 5 × 5 convolutions, which may be helpful in capturing local information, but it is not so efficient in capturing global information. For the models mentioned in Figure 15, they fuse different scale features through concatenation and upsampling which might lead to the deterioration of high-resolution features. Hence, at times, these models might fail to capture the precise location of the liver, especially the irregularities of the liver boundaries.

Figure 15

Examples of erroneous outputs for liver segmentation: (a) R2U-Net on the 3DIRCADb dataset, (b) U-Net++ on the CHAOS dataset, and (c) U²-Net on the LiTS dataset. These results highlight the limitations and challenges faced by the models in accurately segmenting certain liver regions in CT images.

3.8 Data visualization

In this section, we use Gradient-weighted Class Activation Mapping (GradCAM) and IoU Heatmaps as data visualization tools to graphically display some results obtained by various U-Net models.

GradCAM analysis: To provide visual explanations of the model predictions in this investigation, we have used GradCAM [88], which aims to generate a gradient-weighted class activation map. These graphic representations help to explain how neural networks make decisions. We have used GradCAM to generate visualization for the models Vanilla U-Net, Attention U-Net, V-Net, U-Net 3+, R2U-Net, U 2 -Net, Residual U-Net, and U-Net++ models. A CT scan image from the CHAOS dataset has been considered and the corresponding results are shown in Figure 16. Here, the models appear to focus on different segments of the input image. It is clear that the different models focus on distinct areas of the CT scans, suggesting that different models capture different but necessary information in their own unique way.

Figure 16

GradCAM visualizations for the 10 models used in this comparative study: (a) Original image, (b) Vanilla U-Net, (c) Attention U-Net, (d) V-Net, (e) U-Net 3+, (f) R2U-Net, (g) U²-Net, (h) U-Net++, (i) ResU-Net, (j) Swin-UNet, and (k) Trans-UNet. These visualizations highlight the regions of the input images that contributed most to the models' predictions, providing insights into their decision-making processes based on the 3DIRCADb dataset.

From Figure 16, the following things are observed:

1. Vanilla U-Net: GradCAM for Vanilla U-Net reveals that the model emphasizes specific liver structures during segmentation, demonstrating a strong alignment between its predictions and important anatomical features.

2. Attention U-Net: The GradCAM for the Attention U-Net indicates that this model places attention on critical liver regions, showing how it leverages its attention mechanism for accurate segmentation.

3. V-Net: V-Net’s GradCAM illustrates the regions of interest within the liver, showcasing the model’s ability to capture fine details and contours during the segmentation process.

4. U-Net 3+: The GradCAM for U-Net 3+ highlights areas within the liver, emphasizing the model’s capacity to incorporate multi-scale information for precise segmentation.

5. R2U-Net: R2U-Net’s GradCAM shows a broader focus compared to other models, revealing its segmentation strategy that considers a wider context within the liver region.

6. U 2 -Net: GradCAM for U 2 -Net displays the model’s attention to liver boundaries and internal structures, underlining its segmentation capabilities, especially for challenging cases.

7. U-Net++: U-Net++’s GradCAM highlights specific liver structures, indicating its proficiency in capturing complex anatomical features for accurate segmentation.

8. ResU-Net: ResU-Net’s GradCAM reveals the model’s focus on intricate liver textures and patterns, showcasing its ability to exploit residual connections for robust segmentation.

9. Swin-UNet: Swin-UNet’s GradCAM demonstrates the model’s attention to both global and local liver structures, highlighting its effectiveness in leveraging hierarchical representations for precise segmentation.

10. Trans-UNet: GradCAM for Trans-UNet illustrates the model’s adeptness in capturing spatial dependencies and long-range contextual information within the liver, indicating its capability to utilize transformer-based architectures for accurate segmentation.

IoU Heatmap analysis: When it comes to computer vision tasks like object identification, a model’s prediction accuracy is visually represented as an IoU heatmap. Brighter colors indicate higher overlap between expected and ground truth bounding boxes, emphasizing accurate predictions. Colors are assigned based on IoU values. As shown in Figure 17, the heatmap is an effective tool for evaluating and improving model performance.

1. Vanilla U-Net: The IoU heatmap for Vanilla U-Net shows high values corresponding to specific liver structures, indicating strong alignment between its predictions and important anatomical features during segmentation.

2. Attention U-Net: The IoU heatmap for Attention U-Net reveals elevated values in critical liver regions, demonstrating effective utilization of its attention mechanism for accurate segmentation.

3. V-Net: The IoU heatmap generated for V-Net illustrates heightened values in regions of interest within the liver, reflecting the model’s capability to capture fine details and contours during segmentation.

4. U-Net 3+: The IoU heatmap for U-Net 3+ emphasizes elevated values within the liver, showcasing the model’s proficiency in incorporating multi-scale information for precise segmentation.

5. R2U-Net: The IoU heatmap for R2U-Net indicates a broader focus compared to other models, revealing its segmentation strategy that considers a wider context within the liver region.

6. U 2 -Net: The IoU heatmap for U 2 -Net displays elevated values along the liver boundaries and internal structures, highlighting its segmentation capabilities, especially for challenging cases.

7. U-Net++: The IoU heatmap for U-Net++ highlights elevated values corresponding to specific liver structures, indicating its proficiency in capturing complex anatomical features for accurate segmentation.

8. ResU-Net: The IoU heatmap for ResU-Net reveals heightened values in intricate liver textures and patterns, showcasing its ability to exploit residual connections for robust segmentation.

9. Swin-UNet: The IoU heatmap for Swin-UNet demonstrates elevated values in both global and local liver structures, indicating its effectiveness in leveraging hierarchical representations for precise segmentation.

10. Trans-UNet: The IoU heatmap for Trans-UNet illustrates elevated values, highlighting the model’s adeptness in capturing spatial dependencies and long-range contextual information within the liver, indicating its capability to utilize transformer-based architectures for accurate segmentation.

Figure 17

IoU heatmaps for the 10 models used in this comparative study: (a) Original image, (b) Original mask, (c) Vanilla U-Net, (d) Attention U-Net, (e) V-Net, (f) U-Net 3+, (g) R2U-Net, (h) U²-Net, (i) U-Net++, (j) ResU-Net, (k) Swin-UNet, and (l) Trans-UNet. These heatmaps illustrate the Intersection over Union (IoU) performance of each model, highlighting their effectiveness in segmenting regions of interest compared to the ground truth based on the 3DIRCADb dataset.

3.9 Recommendations

Accurate liver segmentation from medical images is of paramount importance in the diagnosis and treatment of various liver diseases. In this article, we have presented a comprehensive comparative analysis of several liver segmentation approaches, with a specific focus on different U-Net variants. Based on the findings and insights derived from this study, we offer the following recommendations:

1. Adoption of U-Net variants: Our comparative analysis reveals that U-Net variants such as Vanilla U-Net, Attention U-Net, V-Net, U-Net 3+, R2U-Net, U 2 -Net, U-net++, ResU-Net, Swin-U-Net, and Trans-U-Net have demonstrated remarkable performance in liver segmentation. We recommend the adoption of these latest U-Net architectures in the field of medical image segmentation, especially for tasks involving liver segmentation.

2. Dataset consideration: The selection of appropriate datasets is crucial in evaluating the performance of liver segmentation algorithms. We recommend researchers and practitioners to consider the datasets like 3DIRCADb, CHAOS, and LiTS for benchmarking purposes, as these datasets provide diverse and challenging medical images that reflect real-world scenarios. Apart from these two, there is a wide variety of datasets that can be explored. A few among them are: PAIP2019 [89], and CholecSeg8k, etc. Comparative studies can also be conducted on them to check the performance of various models on these datasets.

3. Evaluation metrics: This study emphasizes the importance of using established evaluation measures to compare the results of the segmentation. We recommend the use of well-established metrics, such as DSC, IoU, VoE, RVD, etc., to ensure a rigorous and meaningful comparison among different segmentation models.

4. Visual interpretation: To enhance the interpretability of segmentation results, we recommend the use of visual tools like GradCAM, Activation Maps, IoU Heatmaps, etc. Such a tool can provide valuable insights into the performance of segmentation algorithms and facilitate a deeper understanding of the results. Other data visualization tools like t-distributed stochastic neighbor embedding (t-SNE) plots can be used in the future to know pixel-wise details segmentation results.

5. Further research: While U-Net variants have shown significant promise in liver segmentation, there is always room for improvement. We encourage further research into the development of novel architectures and techniques that can address specific challenges in liver segmentation, such as the segmentation of small structures and intricate details.

6. Clinical integration: Ultimately, the success of liver segmentation algorithms lies in their practical utility in the clinical setting. We recommend a close collaboration among researchers, medical professionals and healthcare institutions to ensure the seamless integration of advanced segmentation techniques into clinical practice. This collaboration can lead to better patient outcomes and more effective liver disease management.

Our comparative study provides valuable insights into liver segmentation techniques, with a focus on U-Net variants. By implementing the recommendations outlined above, researchers and practitioners can advance the field of liver segmentation, ultimately contributing to the improved diagnosis and treatment of liver diseases.

3.10 Challenges and future research directions

Despite significant developments, research gaps in liver and liver tumor segmentation still require thorough exploration. Accurate segmentation is crucial for diagnosis and treatment planning, but these computer-based methods are meant to assist, not directly plan, treatment. Consistently high segmentation accuracy is challenging due to anatomical variances, diverse tumor features, and changes in imaging modalities. To address these issues, specific algorithms are needed to improve patient care and medical decision-making. Key research gaps include the following:

1. Dataset limitations: The datasets used, namely 3DIRCADb, CHAOS, and LiTS, are limited in size and diversity, potentially, thereby impacting result generalizability.

2. Hyperparameter uniformity: The use of uniform hyperparameters across diverse models may not account for each model’s sensitivities to specific settings.

3. Threat to validation: Even with developments in CAD and deep learning-based models for medical imaging, medical experts’ approval and supervision are still crucial. Even with their great efficiency, automated systems can sometimes result in false positives or negatives that need to be validated by a specialist in order to guarantee correct diagnosis and patient safety. Furthermore, there are serious privacy and security issues when using patient data to train and validate deep learning models. Maintaining public confidence in these technologies and protecting sensitive patient data depend on compliance with regulations like GDPR and HIPAA.

4. Model-specific constraints: Specific model limitations, such as training time and robustness, should be acknowledged.

5. Generalizability: The study’s findings may not generalize well to other datasets or clinical scenarios, necessitating further exploration in future work.

Our research presents several interesting avenues for exploration in the domain of liver segmentation. However, there are future scopes in this domain that can give more insights to researchers.

1. Evaluation of latest U-Net variants: Assessing the performance of the latest U-Net variants could provide valuable insights for improving segmentation accuracy.

2. Exploration of different pre-trained encoder backbones: Experimenting with encoder backbones like DenseNet, ResNet, and MobileNet in U-Net architectures may enhance model robustness and efficiency.

3. Integration of recent models like ViT or Mamba: Exploring recent models such as ViT (Vision Transformer) or Mamba within U-Net frameworks could lead to novel advancements in liver segmentation.

4. Investigation of alternative imaging modalities: Studying other imaging modalities such as MRI or positron emission tomography scans may provide additional diagnostic insights.

5. Tumor detection and segmentation: Probing the task of tumor detection and segmentation, especially for HCC, could significantly impact liver cancer diagnosis and treatment planning.