Caries-segnet: multi-scale cascaded hybrid spatial channel attention encoder-decoder for semantic segmentation of dental caries

Conventional image processing for dental caries segmentation

Sornam et al. utilized 120 panoramic caries images to generate 1-rate by a linearly adaptive particle swarm-based optimization algorithm and assessed the classification performance with a backpropagation neural model but suffered from overfitting. Throughout the analysis, the developed model attained 99 % whereas, the error rate of Minimum Mean Square Error (MMSE) achieves 0.008 [18]. The dental caries diagnosis system was developed by extracting statistical features with filtered images by Laplacian filtering, morphological operations, and machine learning-based back propagation neural networks for classifying the features, but failed to consider the depth. Overall, the performance of the model has experimented with different measures like accuracy, false positive rate, Receiver-Operating Characteristic Curve (ROC), and Precision-Recall Curve (PRC). Here, the accuracy of the model was 97 % and the PRC rate was 0.987 [19]. In 2017 Prerna et al. experimented with panoramic images and devised a model for detecting caries automatically based on Radon, Discrete Cosine Transform along with Principal Component Analysis (PCA)to extract the desired information and it was classified using different machine learning-based classifiers. Here, the random forest classifier provided an accuracy of 86 % outperforming other classifiers [20].

Deep learning techniques for dental caries segmentation

Nevertheless, dental image segmentation using deep learning-based techniques was gradually evolving by incorporating various deep learning-based architectures. Choi et al. experimented with 475 Periapical and developed CNN architecture in conjunction with crown extraction algorithm images for boosting the performance of proximal dental caries detection which required a lot of preprocessing. Therefore, the developed model provided a better F1-max of 0.74 [21]. In 2017 Srivastava et al. proposed a Computer Aided Design (CAD)-based CNN model with more than 100 layers to detect cavities and their performance was higher than certified dentists based on overall F1 score but the sensitivity of enamel caries was low. The performance of the F1-score showed the efficient performance of the model. In accordance, 70 % has revealed a significant performance in the analysis of the score [22]. Casalegno et al. developed semantic segmentation of caries lesions at early stages with near-infrared illumination images and achieved a low Mean IOU score of 72.7 % [23]. Haghanifar et al. presented teeth segmentation with a genetic algorithm and caries detection workflow by capsule network utilizing 480 panoramic radiographs to attain caries. Here, the developed model has attained a recognition accuracy of 90.52 % [24]. Deep CNN-based U-Net architecture resembling a U shape is used for the segmentation of cavities as well as for structure segmentation in bitewing X-rays where the sensitivity of initial caries is very low. The different positive measures were evaluated to show the effective performance of the developed model. Henceforth, the F1-score, precision, and recall have outperformed the 64 , 63, and 65 % than the existing approaches [25]. HybridResneXt-RNN, a Meta-heuristic approach was introduced for segmenting and detecting caries [26]. U-Net architecture with 206 periapical radiographs outperforms Segnet and XNet in terms of all metrics for segmenting three types of dental abnormalities Caries, inter radicular radiolucency, and alveolar bone recession. In addition to these, the developed model proved efficient performance while validating with diverse performance measures. It has shown better analysis while computing with the existing approaches [27]. Although CariesNet’s deep learning-based U shape and full-scale axial attention mechanism focus on segmenting caries-affected with a dice value of 93.64 % they are not efficient for segmenting moderate caries regions from panoramic images [11]. Zhu et al. [28] have developed an effective deep-learning model called CariesNetto to differentiate caries from panoramic radiographs. Here, the high-quality panoramic radiograph dataset was utilized with the consideration of shallow, moderate, and deep carries. However, the U-shaped network has been adapted with the CariesNet to segment the caries from the panoramic images. Throughout the analysis, the developed model achieved better performance than the existing approaches. Ying et al. [29] have designed a deep network for segmenting the caries in the clinically gathered from the tooth X-ray images. Here, the skip connection has been applied from the U-shaped network. However, several methods were utilized to improve the features from the multi-scale. The diverse experimental validation was outperformed to improve the pixel classification for the caries. Moreover, the analysis has proved the superior performance rather than existing approaches. Qayyum et al. [30] have implemented a self-training-based method for detecting and segmenting dental caries efficiently. However, the augmentation techniques were adapted to provide better computational performance. Here, the experiments were validated based on the fully labeled dental radiographic dataset which contains 141 images. Throughout the analysis, the developed model has attained superior performance in terms of accuracy other than existing approaches.

Attention mechanisms

Attention mechanisms mimic human biological attention and become popular nowadays. It enables the baseline model like CNN by providing a certain type of interpolation along with their backpropagation mechanism. Recurrent Neural Network (RNN) based gated recursive CNNs with first attention mechanisms were developed in the discipline of Natural Language Processing (NLP) for learning sentences and their grammatical meanings [31]. Attention mechanisms were later introduced and popularly adopted during the past decade in computer vision for duplicating human visual systems and their effectiveness has been impressively demonstrated [32]. Deep learning models encapsulate numerous channels and each channel is responsible for producing a feature map to represent different objects with the notion of what to pay attention to. The Squeeze and Excitation (SE) attention mechanism envisions the channel-wise interdependencies with the squeeze and excitation module. In general, the error rate has been significantly reduced at a rate of 2.251 % [33]. Gated Channel Transformation (GCT) [34], andStyle Based Recalibration Modules [35] show improvements in channel-wise squeeze and excitation networks. The key notion behind the spatial attention mechanism is robust by learning the feature maps with the relevance of importance in the same layer and they focus on where to pay attention. Gather Excite Net (GENET) [36], Recurrent Models of Vision Attention RAM [37], and Spatial Transformer Networks (STN) [38] are some of the spatial attention mechanisms adopted in computer vision for providing attention to spatial features. While considering the analysis of the F1-score, the model has attained 92.9 %. It reveals the efficient performance of the model than the existing approaches. Attention mechanisms are infused with deep learning models for semantic segmentation by focusing more on the target features and overwhelming the irrelevant features. Attention U-Net is the first attempt to segment CT scans for locating the pancreas where a soft attention gate is infused with the CNN architecture for targeting the most relevant spatial features but fails to consider the interrelationship of features [17]. Breast ultrasounds are segmented with the aid of attention mechanisms along with the multi-scale features but they did not generalize well. Here, the different measures have been validated and executed to maximize the performance. Throughout the analysis, the developed model yielded 97 % accuracy and 98 % specificity [39]. USE-Net combines the essence of skip connections and SE modules into the U-Net [9] for obtaining the revamped channel-wise information without considering the spatial information while segmenting prostate gland and brain Magnetic Resonance Imaging (MRI) [40].

Although various methodologies discussed in the literature review using image processing, attention mechanisms, and deep learning techniques proves their performance for dental caries detection, they face the following issues.

1. Minimized accuracy: Due to the low contrast level in the image and the presence of noise in the dental images, it could degrade the image quality. Thus, it leads to minimize the accuracy performance in the dental caries segmentation process.

2. Feature loss: In conventional techniques, while accessing the irregular edges and small size of caries leads to inaccurate segmentation outcomes. These results in the conventional models are not efficient for extracting the relevant features that emerge feature loss.

3. Redundancy in training model parameters: Considering huge volume of images in the segmentation process, the traditional model shows redundancy in training the model parameters. This redundancy in the training process can easily affect the model’s performance. Also, it is not efficient to generalize the data in unseen data.

4. Needs more computational resources: The large size and complexity of images and datasets in the training process often require high computational resources otherwise, it merely maximizes the training time to detect the outcomes in the traditional models.

5. Overfitting problem: During the validation process, when a model fits too closely to the training data, overfitting issues are occurred. The emergence of overfitting issues can result in providing insufficient outcomes that affect the quality of the data.

These limitations from the conventional techniques are effectively resolved by developing a new technique to segment the dental caries effectively to suggest the appropriate treatment from the clinicians.

From the existing work, there is a need for a hybrid mechanism to combine channel and spatial attention for fusing both local and global features and producing a combined attention weight for enriching the semantic segmentation feature map and improving the quality of training. The proposed system overcomes the drawbacks of the existing architectures by reducing the redundancies in learning the parameters for segmenting caries-affected portions from X-ray images.

Overall architecture

The proposed multi-scale cascaded attention architecture is designed as depicted in (Figure 1). The encoder comprises four DFCR blocks arranged sequentially. The encoder extracts multi-scale features and produces feature maps S₁, S₂, S₃, and S₄ which are then fed into the cascaded structure of the decoder. The attention mechanisms incorporated in the decoder along with the cascaded mechanisms that are responsible for producing prediction feature maps at four stages of the decoder, which are then combined to generate an overall prediction map. Finally, binary segmentation is achieved by using the sigmoid activation function.

Figure 1:

Overall architecture of Caries-SegNet a. DFCR b. Upconv block c. conv block.

DFCR block

The proposed system includes a novel DFCR block that replaces the encoder blocks in the traditional U-Net architecture [9]. The DFCR block is illustrated in (Figure 1). This block is designed as an inspiration for the concept of residual. Here there are two identical blocks in a sequential fashion with 3×3 conv in which Batch Normalization and Rectified Linear Unit (ReLU) activation are included. There is another block in the skip connection with 3×3 convolutions Batch Normalization that also activates with a ReLU function. Here w₁, w₂, and w₃ are the weights produced by each of these convolutional blocks. These weights are summed up together to extract the shallow, middle layer, and deeper layer information. As these features are equally important additive features are extracted and their result obtained is comparable to using three different filters of sizes 1×1, 3×3, and 5×5. Hence, it is responsible for widening the receptive field for obtaining the multi-scale spatial features by reducing feature information loss and improving the generalization of the module. The pseudo-code of the Caries-SegNet Algorithm is shown in Algorithm 1.

Cascaded decoder

In the conventional U-Net [9] architecture, the features are just propagated from the encoder to the expanding path and the skip connections between them to enforce a featurefusion approach which leads to the accretion of feature maps with the same scale. Additionally, the contextual information at the higher levels in the decoder part is also being lost. Inspired by U-Net [9] and Attention U-Net [17] a cascaded decoder network with an attention mechanism and hierarchical feature aggregation is proposed to overcome this problem.

The cascaded decoder comprises the up-conv block for feature upsampling, attention gates for cascaded feature fusion, and the HSCA block for enhancing the feature map further in a robust manner. The decoder encapsulates four HSCA blocks to be combined with the features generated from four stages of the encoder in a pyramid fashion. The output features from the encoder through the skip connections are combined with the up-sampled features of the previous decoder for aggregating the multi-scale features. Then the output of the attention gate is fused with the previous layer’s up-sampled features. These concatenated features are processed by using the HSCA block, where the pixels are combined based on both spatial and channel domains and suppressing the background features. Finally, the output of each HSCA block in the decoder is sent to the prediction head, in which all the predictions are aggregated to produce an overall segmentation mask.

Attention gate

The encoder and the decoder features are fused with the attention gates as depicted in (Figure 2) for highlighting the foreground pixels along with their association with the adjacent pixels and eventually suppressing the irrelevant features. The attention gate generates the attention weights by modifying a small number of parameters without the requirement to train a different model. The attention gate produces an attention weight which is an interrelationship of the shallow layers and dense layers for producing both coarse and fine-grained predictions.

Figure 2:

Schematic diagram of attention Gate.

Term S _l represents the input from the skip connections and G _l represents the gating signal from the up-conv. Attention coefficients; βϵ[0,1] conserve only the activations relevant to the salient regions by locating the important visual regions along with interrelated features and trimming the irrelevant feature responses.

W _S ,W _G is the linear transformation obtained by the 1×1 convolution operation. The non-linearity of these linear transformations is produced by the swish activation function [39]. Swish activation is defined in Equation (1).

(1) f x = x × σ x

Here, the term σ defines the sigmoid activation function. However, the variable y₁ defines the feature map and z₁ shows the vector of the gating signal.For each pixel, S _l f(I,j) attention values are determined. The pixel-wise area to be focused is determined by the gating signal G_l.

(2) α i = σ 2 ω T σ 1 X y T y 1 + X z T z 1 + b x + b ω

(3) β i = σ Q A 1 S i 1 , G i , φ a  

Although swish activation functions have already been adopted in classification [41], it is the first attempt to incorporate the attention mechanism in semantic segmentation. Here, the bias terms are represented as b _x and b _ω .Swish activation, a non-monotonic function performs comparatively better than ReLUactivation and is considered a better alternative by converging smoothly and minimizes the loss function better than ReLU.

To fetch the output from the attention multiplication is done element-wise with the input feature maps and the attention coefficients are determined based on Equation (4).

(4) f   y = y . sigmoid  β . y = y 1 + e β y

Here, the term β is defined as a parameter that can be represented as predefined or learnable.

HSCA module

HSCA module as portrayed in (Figure 3) emphasizes blending the features from two primary axes by extracting significant features from the spatial and channel domains to produce aggregated features salient in both domains together. The HSCA module comprises CA(.) and SA(.) arranged in a sequential manner and a convolution block (conv) as in Equation (5).

(5) HSC A y = conv  ( SA  CA y

Here HSCA(.) is the Hybrid Spatial Channel Attention module and y denotes the input tensor. The working principle of the HSCA module is illustrated in (Figure 3). The pseudocode of the Hybrid Spatial Channel Attention is shown in Algorithm 2.

Figure 3:

Hybrid spatial channel Attention.

Algorithm 2:

Hybrid spatial channel attention.

Input: Feature Map of dimension Rwxhxc

Output: Feature map of dimension rwxhxcen

#Channel attentionit

1. Split and squeeze channels of input feature map with multi-scale kernels to produce feature maps

f 0 , f 1 , . . , f S − 1

2. #Concatenate multi-scale feature maps f

f = c o n c a t f 0 , f 1 , . . , f S − 1

3. #Generate a vector of attention weight yi using squeeze and excite blocken

Y i = S E W f i , i = 0 , 1 , 2 , … , s − 1

4. #Multi-scalechannel attention vectorit

Y = c o n c a t Y 0 , Y 1 , . . , Y S − 1

5. #Soft attention across various scales

ati=softmax(Yi) #channel-wise attention weighten

6. #Channel-wise attention weight multiplied by corresponding feature scaleen

Z i = a t i * f i   i = 1 , 2 , 3 , … s − 1

7. #Integration of multi-scale information

O = c o n c a t Y 0 , Y 1 , . . , Y S − 1

#Spatial attentionen

8. #Input feature map yro

y = c h a n n e l   max y + c h a n n e l   a v e r a g e y

9. #Spatial attentionit

S A y = s i g m o i d c o n v y   * y

#Hybrid spatial channel attentionen

10.# c o n v < c o n v o l u t i o n , S A < S p a t i a l   A t t e n t i o n , C A < C h a n n e l   A t t e n t i o n

H S C A y = c o n v ( S A C A y

11. Return output feature mapro

Channel attention (CA)

Channel-wise attention is a mechanism that enables the weight of each channel selectively based on their importance and provides a more informative output. A channel attention block is proposed for extracting long-range dependencies along with more granular, multi-scale spatial information. Here, the multi-scale pyramid convolution is being performed to integrate the channel-wise feature maps of the input image. Different scales of spatial information from channel-wise feature maps are obtained by squeezing the input tensor in the channel-wise dimension. Incorporating context features with neighbor scales in this way allows for greater accuracy. The sigmoid activation function enables the recalibration of the attention weights produced by the corresponding channels. Using this channel attention module, the complexity of the model is very low, and it integrates both local and global attention with long-range channel dependencies.

Split Squeeze and Concat (SSC)

This module is responsible for producing channel-wise multi-scale feature maps by ensuring the parallel execution of positional information from the input image at multiple scales. Multiple scale convolution kernels are utilized and channel-wise spatial information with varying scales is obtained by channel-wise squeezing the input tensor. Each feature map with different scales f_i has a channel of dimension c’=c/s.i=1, 2,.s-1, and they can learn multi-scale information and interact with cross-channel information locally. Group convolutions are employed in the convolution kernels to decrease the parameters and computational cost.

The group size and kernel size are related as in Equation (6).

(6) g = 2 k − 1 / 2

Where k denotes the kernel size, g represents the group size. A multi-scale feature map is generated by Equation (7)

(7) f i = Conv k j × k j , g j   x   j = 0 , 1 , 2 … s − 1

Here, the jth kernel size

(8) k j = 2 × j + 1 + 1

j t h group size

(9) g j = 2 k j − 1 / 2

(10) f j ∈ R C ´ × h × w

It represents different scale feature maps. Feature maps are concatenated for producing multi-scale featuremaps with Equation (11).

(11) f = c a t f 0 , f 1 , … , f s − 1  

Figure 3 illustrates the SSC module.

Where f∈R^ˆ(c×hxw) represents the multi-scale feature map. Multi-scale attention weight vectors are obtained by extracting the weights from feature maps based on channel-wise attention, which are pre-processed.

The vector of attention weight is represented in Equation (12).

(12) Y i = SEW f i   , i = 0 , 1 , 2 , … , s − 1

Here, the term SEW(.) defines the Squeeze Excite Weight module. Yi∈R^ˆ(C^´×1×1), thus it denotes the attention weight. Thus, attention weights with different scales are obtained from the input feature map. Then attention information is fused with cross-dimensional features without destroying the channel-wise original attention vector. The multi-scale channel attention vector is concatenated as shown in Equation (13).

(13) Y = Y 0 ⨁ Y 1 ⨁ … ⨁ Y s − 1

Y represents the multi-scale attention weight vector, Yi denotes the attention value, ⨁ is the concatenation operator. Soft attention across various scales for selecting the different scales of spatial information is provided in Equation (14).

(14) a t i = s o f t m a x Y i = exp Y i / ∑ i = 0 ⋀ s − 1 exp Y i

The softmax function is incorporated for producing the recalibrated weight a_ti, which includes local spatial information and channel-wise attention weight as in Equation (15). This results in the establishment of interplay between local and global channel attention.

(15) a t = a t 0 ⨁ a t 1 ⨁ … ⨁ a t s − 1

Where ati is the attention obtained after channel-wise interaction. Then multi-scale recalibrated channel-wise attention weight ati is multiplied by the corresponding feature scale f _i as shown in Equation (16)

(16) Z i = f i ⨂ a ti , i = 1 , 2 , 3 , … s − 1

Z _i denotes the feature map and ⨂ is the channel-wise multiplication. A concatenation operation is performed as compared to an addition operation to preserve feature information without destroying the original feature map. Thus, the output as in Equation (17) is an integration of multi-scale information and cross-channel interaction.

(17) O = concat Y 0 ⨁ Y 1 ⨁ … ⨁ Y s − 1

O represents the output, concat represents the concatenation operation, and Y _i represents the feature map.

Squeeze Excite Weight (SEW) block

The SE block comprises a squeeze and an exciting block for global feature encoding and extracting their channel-wise interrelationship. Let F∈R^ˆ(c×h×w) denote the input feature map, and w,c,andh are used for representing width, input channels, and height, respectively. Channel-wise information is obtained using global average pooling using the following equation (18).

(18) G c = H p q X z = 1 W × Y ∑ x = 1 W ∑ z = 1 Y X z x , z

Attention weight based on each channel is a calculation as shown in Equation (19).

(19) wc = σ W 2 ρ W 1   Gc   

Here, the term σ represents the sigmoid activation used for the excitation function, ρ represents the ReLU activation function, W1∈R^ˆ(C×C/r)andW2∈R^ˆ(c/r×C) are the fully connected layers.

Spatial attention (SA)

SA determines to focus in the feature map, which then strengthens the targeted features. The spatial attention SA(.) is represented by Equation (20).

(20) SA y = σ C Cm  y + Ca  y   * y

Here, Cm(.) is the maximum value in the channel domain.Ca(.) is the average value in the channel domain. C(.) denotes a 7×7 convolution with a padding of three for increasing the spatial context. σ(.) is the sigmoid activation function.

Hierarchical loss function and feature map aggregation

All four stages of the hierarchical encoder are assigned to an individual prediction head. An additive aggregation is used for calculating the final prediction map, as shown in Equation (23).

Terms a1,a2,a3,anda4 are the predictions head weights and p₁, p₂, p₃, and p₄ are the feature maps generated by the prediction heads. We have performed our experiments by assigning a value of one to all a₁, a₂, a₃, and a₄. We obtained the output of the final prediction for binary segmentation by applying a sigmoid activation function.

Loss functions for individual prediction heads are computed separately, and they are consolidated using Equation (24).

Dental caries dataset

Massive amounts of data play a key role in the Artificial Intelligence (AI)-enabled deep learning era. There is a lack of a sufficient number of dental X-rays used for training the model. Hence, we have collected 1,400 periapical dental X-rays and built a large dental caries data set to meet the data scarcity issues in working with deep learning models.

All the periapical X-rays were collected from a private dental clinic from July 2022 to April 2023, located in Chennai. The images used in this study were fully anonymized and did not contain any personally identifiable information such as the Patient’s name, gender, age, and gender. The study adhered to ethical guidelines and standards applicable to research involving anonymized data. This research involved the use of anonymized data and hence ethical clearance from an Institutional Review Board or ethics committee is exempted. All the images are acquired using SOPRO imaging software and stored in Joint Photographic Experts Group (JPG)format. Digital periapical X-rays are captured by the patient in a sitting position in wheelchair when normal room lighting condition. The images are acquired in two different resolutions 1,496×1,000 and 1,000×1,496. All the images are resized to 256×256 pixels. These images are annotated by three experienced dentists, and caries-affected regions are segmented from the X-rays and labeled using Gimp software (Figure 4). depicts the X-ray image and its corresponding ground truth mask for caries lesions.

Figure 4:

Sample image and mask from caries dataset.

Dental X-rays collected from the clinics are degraded by Gaussian noise. Wiener filter, which is a linear filtering technique, adapts itself to the local variance produces better quality images by preserving the edges in the images, and is utilized in the noise removal. Due to privacy issues and labeling expenses, gathering huge datasets for medical imaging is currently difficult. Hence it is essential to adopt data augmentation techniques so it is possible to significantly increase the volume of data. During training time augmentation techniques like horizontal flip, vertical flip, rotation 45°, and diagonal flip were employed to both the images and masks to significantly increase the size of the samples. The parameters of the Weiner filter are shown in Table 1.

Tufts dental dataset

We have also performed our experiments using another benchmark dataset. The images are taken from the Kaggle database by using the link https://www.kaggle.com/datasets/iftakharh/tufts-dental-datasetcustomized. Access date: 22-08-2024.Tufts dental dataset is a multimodal dataset that comprises 1,000 panoramic dental X-rays taken from Tufts Dental University [42]. Radiographic images in TIFF/JPEG format were collected from the patients who underwent radiographic procedures from January 2014 to December 2016. Radiographic images are acquired using OP100 Orthopantomograph and Plammeca Promax 2D (Henry Schein) radiographic units [42]. The images are annotated by dental experts and a fourth-year dental student after a thorough analysis of the radiographs. The images are made visually appealing by adopting the Contrast Limited Adaptive Histogram Equalization (CLAHE)method [43]. Although this is a multimodal dataset with teeth segmentation, abnormality segmentation, and eye tracker information in the experiments only teeth segmentation is used for comparative study.

Segmented image results

Figure 5 shows the resultant segmented image comparison on the developed Caries-SegNet-based segmentation model. The Caries-SegNet compared with other algorithms such as UNet [9], UNet++ [44], ResUNet [38], SEResUNet [45], AttentionUNet [17], DANet [46], and TransUNet [47]. Finally, the image results clearly show that the developed model provides the accurate and clear segmented images than other existing models.

Figure 5:

Resultant segmented images obtained from the Caries-SegNet and comparison with other algorithms.

Processing time of the developed model

The processing time of the developed model is computed and compared with diverse existing methods which are shown in Table 4. Certainly, the validation has proved that the developed model attains better performance than the existing methods.