Classification of histopathological images for oral cancer in early stages using a deep learning approach

3.1 Data description

The dataset we used was published in 2023 by the Oral Diagnosis Project of the Federal University of Espírito Santo (NDB-UFES) [21]. It is composed of high-resolution 2,048 × 1,536 -pixel histopathological images of oral leukoplakia and OSCC from biopsies performed between 2010 and 2021 and stained with hematoxylin–eosin. A total of 237 images were obtained: 91 for OC, 89 for oral LW/D, and 57 for LW/oD. Figure 2 provides examples of images from these three classes. In addition, the dataset includes patches of the images with 512 × 512 pixels. There were 3,763 patches in total, of which 1,126 were of OC, 1,930 were of LW/D, and 707 were of LW/oD.

Figure 2

Examples of images from the three classes in the NDB-UFES dataset: (a) oral squamous cell carcinoma, (b) LW/D, and (c) LW/oD.

3.2 Preprocessing

Model generalization is a challenge when analyzing histological images due to differences between digital scanners and between their responses, which can affect their performance. To address this problem, some research on stain normalization [22] has suggested transferring color characteristics from one image to another using statistical analysis. The technique suggested in the study by Shen et al. [11] unifies two methods of analyzing histological images: stain normalization, in which an image is used as a template to alleviate stain differences, and stain augmentation, which enriches stain styles by simulating variations of those stains. With the objective of decreasing generalization errors, this method, called RandStainNA, applies these techniques to the problem of changing stain styles in the workable range and with several color spaces selected randomly. The colors in image I m Im are converted from RGB to LAB, and the pixel value can be written as [ l , a , b ] , where each channel in this image is represented by the average G m = [ g m ( l ) , g m ( a ) , g m ( b ) ] and the standard deviation S m = [ s m ( l ) , s m ( a ) , s m ( b ) ] . The resulting virtual template associated with image I m from Gaussian distribution F G and F S is represented by G t = [ g t ( l ) , g t ( a ) , g t ( b ) ] and S t = [ s t ( l ) , s t ( a ) , s t ( b ) ] )]. The formula that represents the image-wise normalization by the random virtual template is as follows:

The color space is then transferred again [ l ′ , a ′ , b ′ ] from the LAB to the RGB space. In this study, to increase the generalizability of the model and the number of training datasets, we used the RandStainNA technique with the color spaces in LAB. The number of datasets achieved after applying RandStainNA is shown in Table 2, and Figure 3 presents the steps of the RandStainNA technique when applied to the dataset.

Figure 3

Steps involved in performing random stain normalization and augmentation on an image from the dataset used in this study. Source: Created by the authors.

3.3 Transfer learning

In most studies, transfer learning using CNNs has been applied to extract features from medical images. The researchers in the study by Li et al. [23] used ResNet-50 to extract features from breast cancer histopathological images, while the study by Liu et al. [24] used ResNet-50 to predict the primary tumor areas in spinal metastases using magnetic resonance imaging. In this study, we used the pretrained VGG-19 and ResNet-50 models.

3.3.1 VGG-19

The VGG model is a CNN pre-trained model that was suggested in 2014 [25] by Simonyan et al. at Oxford University, UK, where the VGG was trained on ImageNet with 1.3 million images. The VGG-19 model contains 19 connected layers, which consist of fully and highly connected convolutional layers, and uses max pooling and the Softmax activation function. In this study, we used the same architecture with a simple modification in the classification block, which contains two linear layers and the ReLU activation function between them, followed by a dropout with a value of 0.5 (see Figure 4 for present the VGG-19 model architecture).

Figure 4

Graphic representation of the VGG-19 model architecture created by Simonyan et al. [25].

3.3.2 ResNet-50

ResNet-50 is a deep CNN [26] containing 50 deep layers. It is based on the ResNet architecture, which uses a residual block to add the output from one layer to the next, enabling skip connections and solving the vanishing gradient problem. The ResNet-50 model was trained on one million images from ImageNet. The architecture consists of many CLs followed by BatchNorm and ReLU activation to extract features. The data are subjected to max pooling, which reduces the spatial dimensions. It also contains two types of blocks, i.e., identity and convolutional blocks, to process and transform these features. Both blocks pass on the input and are processed through the CL, which adds the input to the output. In the convolutional blocks, a 1 × 1 CL is added prior to a 3 × 3 CL to decrease the number of filters. In the last of the ResNet-50 networks, the final classification is performed by the fully connected layers, which use Softmax as an activation function to determine the probability of the final class. Figure 5 presents the ResNet-50 model architecture.

Figure 5

Graphic representation of the ResNet-50 model architecture created by He et al. [26].

3.4 Data splitting and experiments

To evaluate the models, we divided the dataset into training and validation (5/6) and testing (1/6) portions and used standard forms of augmentation, including flipping, rotation, and range shifting. We experimented with various hyperparameters and settled on a 0.001 learning rate and fivefold cross-validation for all models. There were some differences in the settings for each model. With the VGG-19 model, we used an SGD optimizer with ten epochs. For the ResNet-50 model, we employed two SGD optimizers for the pretraining phase and Adam in the training on images using 15 epochs. For the ensemble model in the training phase, we used Adam optimization with five epochs. Table 3 displays the parameters of each model.

We performed the experiments for Task I using two methods to categorize images into three classes: LW/oD, LW/D, and OSCC. The steps for this task are explained in this section.

Task I: Preprocessing: we applied RandStainNA to the original images from all classes. Training: the transfer learning models were trained using the images after preprocessing, and fivefold cross-validation was applied during the training. Testing: the transfer learning models were tested using images unseen by the models, and the model’s performance was evaluated using the metrics of BCC, precision, recall, and AUC.

Task I with pretrained patches: Pretraining: the transfer learning models were trained using the patches of training images from the NDB-UFES dataset. We applied fivefold cross-validation during the training and then saved the models. Training: the models we saved after pretraining using patches were used to train the model on images, and RandStainNA was applied. We again applied fivefold cross-validation during the training. Testing: the transfer learning models were tested using images unseen by the models, and the performance was again evaluated based on BCC, precision, recall, and AUC.

3.5 Evaluation metrics

To evaluate the models’ performance on the training set, we used fivefold cross-validation to generate five trained models, and the average for the results of these five was computed after evaluation on the testing set. To assess the models’ performance, we used the four metrics: BCC, precision, recall, and AUC. BCC is used to deal with imbalanced datasets. As shown in equation (2), it is defined as the mean of both the recall true-positive rate and the specificity true-negative rate, as shown in equations (4) and (5), respectively. Precision reflects the percentage of images diagnosed, as shown in equation (3). Recall indicates the proportion of images that the model correctly diagnosed. Finally, we used the macro average of the AUC values:

4.1 Comparison with previous works

The performance of our framework can be compared with that of the methods used in the latest studies. The differences in performance between Task I and Task I with pretrained patches are shown in Table 6. Our framework achieved better results on Task I using both methods. To the best of our knowledge, only one study has used patches of images from the NDB-UFES dataset because it is new. Moreover, de Lima et al. [3] completed Task I using images and patches of images to classify the three classes. Their study achieved the best results using the PiT model for both procedures from Task I: the BCC was 63.70%, precision was 67.60%, recall was 65.60%, and AUC was 84.06% for Task I, while the BCC was 67.45%, precision was 69.20%, recall was 67.60%, and AUC was 81.88% for Task I with pretrained patches. The results using our framework demonstrated improved performance: the BCC was 66.66%, precision was 72.71%, recall was 71.79%, and AUC was 89.19% on Task I using the VGG-19 model, and the BCC was 73.33%, precision was 74.68%, recall was 74.35%, and AUC was 92.21% in Task I with pretrained patches using the ResNet-50 model.

4.2 Limitations

The proposed framework has proven effective in improving the results of classifications of OC in the early stages. However, our study has a few limitations that limit the learning ability of the models, including imbalanced and limited datasets and limited resources available to train the models. Regarding the issue of imbalanced and limited datasets, we used NDB-UFES, which contains histopathological images categorizable into three classes, namely, OSCC, LW/D, and LW/oD, of which LW/oD is a minority class by a difference of 32 samples. Additionally, the complexity of histopathology increases the difficulty of the learning process, particularly with respect to intra-disease heterogeneities in oral leukoplakia and the subtle differences between no dysplasia (LW/oD) and mild-to-moderate dysplasia (LW/D). This may have led to uncomprehensive data for each class in training and classification, which may have affected the accuracy of the outcomes. In addition, the process of training the models requires high-performance resources, particularly when dealing with the high resolution of histopathological images, which, in the NDB-UFES dataset, had a size of 2,048 × 1,536 pixels. This problem emerged when we applied the RandStainNA technique, wherein the number of images was increased by stain augmentation. This study used Colab Pro resources and an A100 processor with 83.5 gigabytes of random access memory. Therefore, limited resources were available to train the models.