A multiscale and dual-loss network for pulmonary nodule classification

2.1 Revisit of residual block

In this section, we first revisit the residual block, which serves as the fundamental building block of our architectures. The key innovation that the residual block introduces is the concept of skip connections, which facilitate gradient flow through the network and mitigate issues related to vanishing gradients. The mathematical formulation of a basic residual block can be expressed as follows:

where x represents the input to the residual block, ℱ ( x , { θ } ) denotes a series of operations parameterized by weights { θ } , and y is the output of the residual block. The addition operation in equation (1) allows for the identity mapping from input x , enabling a direct path for gradients during backpropagation. To elaborate further, each residual block typically consists of two or more convolutional layers along with batch normalization and ReLU activation. The common configuration within a residual block could be given as follows:

This structure provides not only depth but also enhances representational capacity while ensuring that valuable features are retained throughout multiple layers.

2.2 Multiscale feature weighted fusion

The process begins with inputting an image I ∈ R H × W × C , where H represents the height, W denotes the width, and C is the number of channels. By applying the ResNet network, this image undergoes multiple convolutional layers and activation functions to extract features at various scales and abstraction levels. At this stage, we obtain corresponding feature maps for i th layer, the output feature map from layer l i can be expressed as follows:

To reduce dimensionality while enhancing the expressive power of features, we apply a 1 × 1 convolutional layer to each extracted feature map. This not only allows us to reduce the channel dimensions while preserving spatial information but also enhances the model’s ability to capture nonlinear representations. This operation can be described by the following equation:

where W ( i ) represents the weights and b ( i ) is its corresponding bias term. Next, for all processed multiscale feature maps, we perform spatial resizing to unify their dimensions to a specified size H f , W f . This step is essential for effective combination and fusion of different scales without losing important information or details. The adjustment can be formulated as follows:

After resizing, we obtain a set of new feature maps expressed as follows:

Subsequently, at each pixel position ( h , w ) , we calculate the average value across different scales to construct a weight that provides insight into which scale contributes more significantly. This operation can be defined as follows:

Following this calculation, we build a weight matrix reflecting the importance of each scale through normalization:

Once these steps are completed, we acquire a weight matrix for every pixel that emphasizes important features. Next, these weighted feature maps are summation combined to derive a final fused representation defined by the following equation:

Finally, to achieve classification objectives, we employ a linear layer yielding class predictions according to:

where g ( ⋅ ) refers to softmax, and parameters W p , b p , represent weights and biases learned by the classifier.

2.3 Dual-loss strategy

In the binary classification task of pulmonary nodule malignancy, we first prepare a dataset consisting of pairs of pulmonary nodule images along with their corresponding labels. For each input image pair ( I 1 , I 2 ) , where I 1 and I 2 represent two pulmonary nodule images, if both belong to the same category (either both benign or both malignant), then the label is set to y = 0 ; if they belong to different categories (one benign and the other malignant), then the label is set to y = 1 . Next, we use the proposed ResNet as a feature extractor by loading pre-trained weights and removing the last fully connected layer, retaining only the part used for generating feature embeddings. During this process, for each input image I s , we obtain the feature representation through forward propagation:

where Z s is the output feature vector, and f denotes the feedforward process through ResNet. Subsequently, to optimize model performance, we first define Euclidean distance to measure the difference in feature embeddings between the two pulmonary nodule images:

Then we compute the overall contrastive loss L contrastive ( y , D ( Z 1 , Z 2 ) ) :

In this context, L c ( y , D ( Z 1 , Z 2 ) ) is the contrastive loss; when samples are from the same class (i.e., y = 0 ), we minimize their distance, whereas when samples are from different classes (i.e., y = 1 ), we desire that their distance exceeds some threshold m . Simultaneously, we use BCE loss to assess how well the network predicts given inputs. Corresponding to each category, the network produces a logits value that is transformed into probability via a sigmoid function:

where p ( y ∣ I s ) indicates the probability of malignancy given provided pulmonary nodule image I s and z s is its logits value after mapping; while function g ( I s ) maps inputs into output space via the neural network structure. Thus, the BCE loss can be defined as follows:

Finally, these two losses are combined into an overall loss function:

where L dual represents total loss including contrastive loss and binary cross-entropy loss. During training, paired data will be passed into model preferentially, followed by performing forward propagation in every epoch leading towards obtaining feature embedding accordingly. Full consideration regarding total loss would include comprehensive optimization schemes toward respective parameter updates throughout iterations.

3.1 Datasets

The LIDC-IDRI (Lung Image Database Consortium and Image Database Resource Initiative, https://www.cancerimagingarchive.net/collection/lidc-idri/) is a public database dedicated to the detection and classification of pulmonary nodules. It consists of CT scan images from 1,018 patients, with slice thicknesses varying from 0.6 to 5 mm. To ensure accuracy, each nodule’s diagnostic information – including location, diameter, malignancy classified on a scale of 1 to 5, calcification level, and spiculation – has been independently annotated by four radiologists. This research aims to develop a network that classifies nodules as benign or malignant based on a voting strategy; specifically, if at least half of the experts rate a nodule above 3, it will be classified as malignant; otherwise, it will be deemed benign. To simplify model complexity, only the center cross-section image of each nodule is used as input.

3.2 Implementation details

The classification network was developed in an experimental environment using Keras, leveraging the power of an Nvidia GTX 3090 GPU. In addition, we implemented a learning rate adaptation strategy known as ReduceLROnPlateau, which adjusts the learning rate based on validation performance metrics. To further enhance model generalization, we applied various data augmentation techniques such as random rotations, vertical and horizontal flips, and brightness adjustments. The experimental results are validated by cross-validation to ensure generalizability, where performance metrics are averaged over all folds.

3.3 Evaluation metrics

Accuracy measures the overall model performance in correctly classifying instances. It is calculated as the ratio of true positives (TP), which are correctly identified positive cases, and true negatives (TN), which are correctly identified negative cases, to the total number of instances assessed (including false positives (FP) and false negatives (FN)):

Sensitivity indicates the model’s ability to identify actual positive instances. It is defined as the proportion of true positives relative to all actual positives (TP plus FN):

Specificity assesses the model’s effectiveness in identifying negative instances. It expresses the proportion of TN out of all actual negatives (TN plus FP):

The area under receiver operating characteristic (ROC) curve (AUC) summarizes the model’s capability to discriminate between classes; higher values indicate better performance. AUC is evaluated by plotting the true positive rate against the false positive rate across various thresholds.

3.4 Evaluation of multiscale feature weighted fusion

This experiment evaluates the impact of MFWF on model performance. A comparison is made between the classification different evaluation metrics of the model employing the proposed MFWF and that without (w/o) MFWF as shown in Table 1. The results indicate an effective enhancement in the recognition of critical features across varying scales, suggesting that the integration of information from multiple scales allows for improved attention to essential characteristics relevant to accurate classification. This improvement highlights the effectiveness of our approach in addressing the limitations associated with single-scale methods, ultimately leading to better diagnostic capabilities in classifying pulmonary nodules.

3.5 Different fusion modes of MFWF

In the MFWF process, we examine various fusion techniques such as averaging, concatenation, and linear fusion alongside simple summation for integrating features from different layers. As shown in Figure 3, our experimental results demonstrate that summation consistently outperforms the other methods due to its ability to emphasize significant features captured in deeper layers while avoiding dimensionality issues present in concatenation. Unlike averaging, which dilutes key information by giving equal weight to all inputs, summation effectively enhances classification performance by retaining critical details. While linear blending shows promise in specific contexts through its adaptability, it does not achieve the robust performance exhibited by the summation method.

Figure 3

Different fusion modes of MFWF. a, b, c, d represents the averaging, concatenation, linear, and summation, respectively.

3.6 Impact of dual-loss training strategy

The experiments conducted to evaluate the benefits of combining L dual with L contrastive and L bce demonstrate enhanced generalization performance, as shown in Table 2. The dual-loss strategy ultimately achieves superior results. This improvement can be attributed to its ability to facilitate learning from both similarities and differences among paired samples. By integrating both types of information, the model gains a more nuanced understanding of the relationships between benign and malignant nodules, surpassing the performance of using a single loss function ( L bce or L contrastive ) alone.

3.7 The integration of various input sizes

In the experiment, image sizes of 64 × 64 , 96 × 96 , 128 × 128 , 256 × 256 , and 512 × 512 are utilized. As illustrated in Figure 4, the results show that the model performs best at an input size of 128 × 128 . This optimal performance stems from the ability to balance sufficient detail capture with manageable computational complexity. At this size, the model effectively retains important features necessary for classification while avoiding information loss seen in smaller sizes and excessive noise found in larger sizes. Ultimately, the choice of 128 × 128 enables efficient focus on significant patterns, leading to improved feature extraction and enhanced classification accuracy in deep learning models.

Figure 4

The integration of various input sizes.

3.8 Comparison with state-of-the-art models

To validate the proposed methods further, a comparative analysis is implemented against several state-of-the-art image classification models currently employed for malignancy detection in pulmonary nodules. Notably, since the sizes of each method in the original research is not the same, for a fair comparison, we reemploy those methods on the same data to validate the effectiveness of our proposed method. All the hyperparameters are obeyed the original settings. Here, we compare those methods as follow: Nibali et al. [21], Shen et al. [22], Zhang et al. [23], Wang et al. [24], Zhang et al. [25], Huo et al. [26], Ding et al. [27] (only using the single stream and modality), and Manzari et al. [28]. As shown in Table 3 and Figure 5, the results indicate that the proposed approach consistently outperforms existing methodologies across all evaluation metrics. Notably, we conduct paired t-tests to compare our proposed method with other existing approaches, and the p -values obtained from these comparisons are all below 0.05, thereby confirming the statistical significance of the reported improvements. Furthermore, this improved performance can be attributed to the effective integration of multiscale features employed during training, which enhances the model’s ability to discern subtle differences between benign and malignant nodules. In addition, the use of contrastive loss facilitates better learning from similar samples, contributing to an overall increase in diagnostic accuracy and reliability. These findings underscore the advancements achieved by our proposed methods in enhancing the classification capabilities for pulmonary nodule malignancy.

Figure 5

The ROC comparisons with state-of-the-art methods.

3.9 Different depths of ResNet

The influence of different depths of ResNet on performance is investigated through experiments examining various architectures. As shown in Figure 6, results indicate that as the depth of ResNet increases, performance initially improves due to enhanced capabilities in capturing complex features and learning intricate hierarchical representations. This improvement arises because deeper networks can learn more abstract representations, allowing them to model complex patterns in the data effectively. However, after reaching a certain depth, this performance stabilizes, suggesting diminishing returns regarding further depth. The saturation occurs because very deep networks may struggle with issues like overfitting and increased difficulty during training. Overall, while increasing depth enhances classification performance up to a point by leveraging these advanced capabilities, careful consideration is necessary regarding training dynamics and computational resources in deeper configurations.

Figure 6

Comparisons with different depths.