Quantitative immunohistochemistry analysis of breast Ki67 based on artificial intelligence

2.1 Data collection and annotation

The data source for this study was the Hangzhou Obstetrics and Gynecology Hospital. A total of 400 breast Ki67 WSI pathology slides were obtained through scanning by KFBIO Scanner. The WSI pathology slides were divided into 8:2, with 320 slides used for model data annotation and 80 slides used for model inference. The data division is based on patients, meaning that the inference dataset used for testing is entirely independent of the annotation dataset. There are no common patients, and any WSI region comes from the annotation dataset used for training. The annotated dataset consists of two levels, which are the tile level and the WSI level. At the whole slide image level, we have explicitly annotated the specific values of ki67 positivity and drawn the curve graph of the tumor area. At the tile level, we have carefully annotated the contours of the cells and further specified the classification attributes of the cells, i.e., whether the cells belong to positive, negative, or uncertain categories. In contrast, the dataset used for inference only contains Ki67 percentage annotations. For the WSI to be annotated, pre-processed slicing was used to cut the slide into 1,536 × 1,536 pixel images. Then, the dataset for the tumor region segmentation model, tumor region nucleus detection model, and tumor region nucleus classification model were constructed, respectively. The dataset was divided in the ratio 7:2:1 (training:validation:test), while the training set was used for model training, the validation set was used to evaluate the effect of the model, and the test set was used for model inference. Specifically, the division of the annotated dataset is mixed based on tiles/patches. This implies that the training set, validation set, and test set in this portion may come from the same patient, but they cannot originate from the same WSI region. The data annotation quantities of each model involved in the AI-based breast Ki67 analysis system are shown in Figure 1b.

Figure 1

Dataset generation and distribution map of breast Ki67 analysis system. (a) The flow chart of the dataset label annotation: first annotating the tumor region mask (the delineation of the tumor area is indicated by an orange curve, whereas the purple area signifies the normal tissue region), then extracting the mask region patch image and annotating the cell region mask and cell category (the red curve precisely outlines positive cells, the blue curve outlines negative cells, and the black curve designates the boundaries of cells that are indistinct in nature) and (b) the data volume distribution of each model training, validation, testing (in order as follows: tumor region segmentation model data volume distribution, cell detection model data volume distribution, and cell classification model data volume distribution).

Following the specific process of data annotation shown in Figure 1a, the labeling team, which includes labeling physicians, pathologists, authoritative experts, and technicians, is well-versed in the operation of the labeling software. They are responsible for establishing labeling rules and ensuring a unified understanding of these rules. The physicians involved in annotation possess the qualifications to perform film reading tasks in the pathology department of a tertiary first-class hospital for over 5 years and are capable of issuing pathological diagnosis reports independently. In addition, a senior expert in this professional field is required to review, check, and correct missing or wrong labels in a timely manner. The data required for the tumor region segmentation framework mainly label the tumor region, while the data required for the nucleus detection framework in the tumor region mainly label the instance information of the nucleus in the labeled tumor region and the data required for the nucleus classification framework in the tumor region mainly label the category of the nucleus. In particular, due to the large amount of data required for the training of deep learning models, a semi-automatic annotation method was used to improve the efficiency of data calibration. Based on the existing tumor region segmentation model, nucleus detection model, and nucleus classification model, inference was performed on the unannotated data. The information on the inference results was obtained and transferred to the annotation format required by annotation software as the pre-annotation results. Then, the pre-annotation results were manually revised to modify the incorrect pre-annotation results. Finally, the experts reviewed the annotation results to ensure the quality of the annotation.

1. Informed consent: Informed consent has been obtained from all individuals included in this study.

2. Ethical approval: The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance with the tenets of the Helsinki Declaration, and has been approved by the Ethical Review Committee of Hangzhou Women’s Hospital ([2021]A-8-03).

2.2 Design of technical framework

For the quantitative analysis of Ki67 WSIs of breast cancer, we designed a technical pipeline based on AI technology to build a WSI tumor region rapid quantitative analysis system for breast Ki67 IHC images. The system framework is shown in Figure 2.

Figure 2

Overall framework of the WSI quantitative analysis system for breast Ki67 pathology image. ROI refers to the region of interest. This term is used to describe the parts of pathological images that are important for diagnosis and analysis. ROI patch refers to a small area segmented from the WSI image that contains important diagnosis information. Swin+K-Net, SCNet+DCN, and PS-ViT are all deep learning methods for specific tasks. (a) Tumor segment. (b) Nucleus recognition. (c) Nucleus detection. (d) Nucleus classification.

The assessment of Ki67 positive cells should be with the tumor regions. Our WSI tumor region rapid quantitative analysis system adopts deep learning image segmentation algorithm Swin+K-Net [26], for tumor region segmentation to automatically and rapidly identify the region where the tumor cells are located in the breast Ki67 pathology image. Then, the deep learning instance segmentation algorithm SCNet+DCN+LSJ is applied in this system to detect cells in the tumor region, obtaining the minimum bounding box (MBB) information of the cell location and cell mask region. Thus, it uses the deep learning target classification algorithm PS-ViT to classify the cells in the MBB to identify whether the cell is positive or negative. Finally, cell identification in tumor regions of the breast Ki67 pathology image is achieved and quantitative analysis is also performed to calculate the percentage of positive cells. Above all, the final result of the WSI of the breast Ki67 pathology image is determined.

2.3 Tumor region segmentation

Tumor region segmentation is a fundamental step of the quantitative breast Ki67 analysis system, which can segment the tumor regions, eliminating non-tumor regions and blank regions, thereby improving the rate of tumor cell identification.

We used a semantic segmentation method combining Swin and K-Net [26] for tumor region segmentation, considering the K-Net network achieved state of the art performance in the field of image segmentation. Different from FCN [27], PSPNet [28], and DeepLab [29,30], K-Net is comprised of a set of kernels responsible for generating semantic masks. Each kernel is tasked with a specific semantic category, as illustrated in the network framework shown in Figure 3a. For more details on the K-Net network structure, please refer to the “Tumor Region Segmentation” section in the supplementary material.

Figure 3

Framework diagram for each model of the breast ki67 analysis system. (a) The framework diagram of the K-Net algorithm for tumor region segmentation. (b) The framework diagram of SCNet algorithm for tumor region nucleus detection. (c) The framework diagram of PS-ViT for tumor region nucleus classification.

K-Net can adapt to different frameworks by making the semantic convolution kernel dynamic. We combined K-Net with Swin Transformer to apply tumor region segmentation on breast ki67 IHC pathology image and achieved excellent results, which well separated the main breast tumor regions. At present, in terms of diagnosis, the quantitative analysis of breast Ki67 pays more attention to the proportion of positive nucleus in the tumor region. Therefore, we first take the tumor region segmentation to extract the tumor region, and filter out the detection of nucleuses in the non-tumor region, which can not only significantly improve the efficiency of nucleus recognition in the WSI but also the accuracy of diagnosis can be improved. The way we adopt is much closer to the diagnosis of physician.

2.4 Nucleus detection in tumor region

Nucleus detection is another critical step of the quantitative breast Ki67 analysis system, which is performed on a segmented tumor region and focuses on detecting nucleuses within the tumor region. It is a prerequisite for the quantitative analysis of breast Ki67 to be achieved.

For the nucleus detection in breast Ki67 pathology image, we adopt a SCNet network combined with a variable convolutional DCN network and large-scale jitter (LSJ) data enhancement. The current cascade architecture has brought significant performance improvements in the instance segmentation, e.g., Cascade R-CNN [31], Mask R-CNN [32], and Hybrid Task Cascade [33] (HTC). However, the problem of the limitations of existing cascade methods for instance segmentation with mismatched distributions of training and inference samples has always affected the effectiveness of instance segmentation. SCNet [34] introduces a sample consistency network, as shown in Figure 3b to ensure that the training sample intersection over union (IoU) distribution is close to the IoU distribution at inference. The network combines feature relaying and overall contextual information to further enhance the interrelationship between the sub-tasks of classification, detection, and segmentation. In this way, the problem of the limitations of existing cascade methods, for instance, segmentation with mismatched distributions of training and inference samples is addressed. Compared to the powerful Cascade Mask R-CNN instance segmentation network, the accuracy of the SCNet network has been improved, while the speed has also been significantly improved. For more details on the SCNet network structure, please refer to the “Nucleus Detection in Tumor Region” section in the supplementary material.

We apply SCNet network combined with variable convolutional DCN network and multi-scale training and large-scale dithering data enhancement to detect nucleus in the tumor region of breast Ki67 IHC pathology image. Through training nucleus instance segmentation model, the nucleus MBB location information and nucleus segmentation mask information in the tumor region can be obtained. The location of nucleus in the tumor region of the breast Ki67 IHC pathology images has been well detected, providing an accurate reference for the quantitative analysis of Ki67 IHC pathology image diagnosis.

2.5 Nucleus classification in tumor region

Nucleus classification is the final essential step in our study. This is mainly used to distinguish the detected nucleuses categories and serves an important component of determining the proportion of positive nucleus in breast Ki67 quantitative analysis. For the classification model to differentiate positive and negative nucleus of breast Ki67 pathology image, we apply the PS-ViT [35] network which has achieved good performance. Transformer, a network with powerful global coding capabilities, has recently been widely applied to computer vision tasks. For example, ViT [36] used the transformer directly to solve an image classification task. To process two-dimension image data, ViT simply segmented the image and mapped it into a one-dimension sequence. This simple segmentation leaves the inherent structural information of the image lost, making it difficult for the network to focus on important object regions. To address this problem, PS-ViT proposes an iterative progressive sampling strategy to locate important regions, as shown in Figure 3c, where the sampling locations are updated at each iteration using global information. Therefore, it allows the network to gradually focus on the information of interest. The specific structure of the PS-ViT network can be found in the “Nucleus Classification in Tumor Region” section in the supplementary material.

We extracted the nucleus from the detected nucleus MBB in the tumor region, using the labeled category of the nucleus as ground truth. The distribution of the dataset is shown in Figure 4c. By using the PS-ViT network, we train a classification model, which is able to automatically distinguish whether the nucleuses detected in the tumor region of breast Ki67 pathology images are negative or positive. Thereby, it completed the accurate identification of positive and negative nucleus.

Figure 4

Comparison of the assessment metrics of each model of the breast Ki67 analysis system. (a) The assessment of the tumor region segmentation model. (b) The assessment of the tumor region nucleus detection model. (c) The assessment of the tumor region nucleus classification model.

2.6 Training implementation

The algorithm framework involved in the Ki67 breast quantitative analysis system is achieved based on Python 3.7 and Pytorch 1.8.0. For all training samples, data augmentations such as flips and rotations are used to increase data diversity. We train our models on a NVIDIA RTX 3090 *4 GPU with 24GB memory. The input image size of the tumor region segmentation and tumor region nucleus detection are set as 512 × 512, while the input image size of tumor region nucleus classification is 48 × 48. For more training details, please refer to the “Training Implementation” section in the supplementary material.

2.7 Full image inference and prediction

AI-based studies of Ki67 breast pathology images are mainly carried out on WSI, whose resolution scale are often very large, often reaching the 100,000 × 100,000 level, making it difficult for machines to achieve direct inference. We used a sliding window to crop the whole image into patches for WSI inference. Then, the patch results were stitched and mapped to the full image to achieve WSI inference.

Specific process is as follows: First, set a window sliding block with the size of 1,536 × 1,536 and the stride of 300 for the WSI to obtain the patches. Then, applied the deep learning tumor region segmentation model to obtain the tumor region in the patch image. Next detected the nucleus by using the deep learning instance segmentation model to get the location information MBB and the nucleus mask in the tumor region. After that, classified the nucleus in MBB through the nucleus classification model and identified the nucleus categories. Finally, the identification results of nucleuses in the tumor region of patch were spliced and mapped to the original image. During the splicing process, non maximum suppression was used to suppress the repeated prediction of the target in the overlapping region. Based on the above steps, the nucleus identification results of the WSI were obtained. The number of positive nucleus and negative nucleus in the tumor region of the WSI, as well as the proportion of positive nucleus, was counted. Ultimately, the AI-based diagnosis results of the breast Ki67 pathology images were completed.

2.8 Evaluation metrics

Our breast Ki67 quantitative analysis system includes three deep learning frameworks in different fields. In order to evaluate the effect of each deep learning model, we respectively evaluated it on the corresponding model validation set. Each model adopts the corresponding evaluation method. Besides, to verify the clinical effect of our breast Ki67 quantitative analysis system, we evaluate the effect on the clinical data and compared with the results of the physicians. The relevant evaluation methods are described as follows.

1. The evaluation method of the tumor region segmentation model was based on mDice (an ensemble similarity measure function), which is the Dice coefficient that is usually used to calculate the similarity of two samples (with values ranging from [0, 1]).

   (1) Dice = 2 ∣ X ∩ Y ∣ ∣ X ∣ + ∣ Y ∣ ,

   where X and Y represent the set of predicted labels and true labels, respectively. |X∩Y| is the true intersection of the labeled pixel and the predicted pixel, |X| and |Y| are the number of true values in the predicted pixel and the labeled pixel, respectively. The calculation of mDice is to calculate Dice of all categories, and then take the average value.

2. The nucleus instance segmentation model was evaluated by adopting the mAP index, which is commonly used in the field of instance segmentation. To calculate mAP, first the instance segmentation Precision and Recall need to be calculated. Precision is the proportion of samples with positive predictions that are actually positive, which can be thought of as the ability of the model to find the correct data. Recall refers to the proportion of the samples predicted positive while it is actually positive in the total positive samples. Recall can be regarded as the ability of the model to detect target data in the dataset. The calculation formula is as follows:

   (2) Precision = TP TP + FP ,

   (3) Recall = TP TP + FN .

   TP (True Positive) means the prediction is positive and the truth is positive, and the prediction is correct. FP (False Positive) means the prediction is negative and the truth is positive, and the prediction is wrong. FN (False Negative) means the prediction is negative and the truth is positive, and the prediction is wrong.

   Second, a concept in the field of object detection need to be illustrated, IoU, which is the ratio of intersection and merge, indicating the degree of overlap between the prediction box and the original annotation box. The ideal situation is the complete overlap, where the ratio is 1. IoU is used to measure the accuracy of the prediction box. In general, the ratio of IoU above 0.5 is considered as a good result.

   Next we draw the Precision and Recall curve, which is the P–R curve. The Precision and Recall of each predict result are calculated and the curve is drawn according to the relationship. As shown in Figure 5.

   Finally, on the basis of the Precision–Recall curve, a numerical metric can be obtained by calculating the average of the Precision values corresponding to each Recall value, used to measure how well the trained model is able to detect the category of interest. The calculation formula is as follows:

   (4) AP = ∑ i = 1 n − 1 ( r i + 1 − r i ) P interp ( r i + 1 ) ,

   where n represents the number of samples. r ₁, r ₂, …, r _n are the recall values corresponding to the first interpolation of the precision interpolation segment in ascending order. P _interp(r _i+1) is the maximum value of the sample after taking the r _i+1th point.

   AP value is calculated for only one category. Once the AP is obtained, mAP is calculated for all categories and then averaged. mAP measures how well the trained model is at detecting all categories. Calculation formula is as follows:

   (5) m AP = ∑ i = 1 k AP i K .

3. The nucleus classification model was evaluated by adopting Accuracy, ROC curve, and AUC values, which are commonly used in the field of image classification.

   Accuracy: the ratio of correctly classified samples to the total number of samples

   (6) Accuracy = n correct n total .

   ROC curve: vertical axis is the true positive rate (TPR) and the horizontal axis is the false positive rate (FPR). It is plotted by sorting the samples according to the prediction results of the classification model, and the samples are predicted as positive examples one by one in this order. Each time the TPR and FPR values are calculated and plotted with their horizontal and vertical coordinates, respectively, to obtain the “ROC curve.” AUC can be obtained by summing the areas of each part under the ROC curve. AUC represents the area under the ROC curve, which takes a floating-point value between 0 and 1.

4. For clinical validation, we adopted ICC as the evaluation method, i.e., the test consistency of AI diagnosis evaluation results and doctor diagnosis evaluation results.

Figure 5

Visualized inference results of breast Ki67 analysis system. (a) The visualization of the inference results of tumor segment and nucleus detection in patch. (b) The visualization of the inference of nucleus recognition in patch. (c) The visualization of the WSI inference of the breast Ki67 case.

The ICC was used to evaluate the consistency or reliability of the results of different measures or evaluators for the same quantity.

where σ T 2 is the variation in the measure, σ B 2 is the difference caused by systematic error, and σ E 2 is the difference caused by random error. Larger ICC means less variation due to systematic and random errors. The ICC value is between 0 and 1. It is generally accepted that ICC greater than 0.75 is better in terms of consistency, 0.40–0.75 is average and less than 0.40 is worse.

3.1 Model evaluation

The distribution of the labeled training and validation sets used for each model is shown in Figure 1b. In order to verify the effect of each training model, we validated the trained models on the corresponding labeled validation set. The model training effectiveness was evaluated on the validation set according to the evaluation metrics. The tumor region segmentation model was evaluated based on the mDice coefficient, the tumor region nucleus detection model based on the mAP metric, and the nucleus classification model based on the accuracy metric.

In addition, we compared the results of the individual algorithm of the Ki67 breast quantitative analysis system with those algorithms of SOTA performance in the related field. Specifically, we compared the K-Net+Swin segmentation algorithm that we used with the Segformer [37] algorithm and PSPNet+U-Net algorithm for the tumor region segmentation algorithm, which have advanced performance. For the tumor region nucleus detection algorithm, we compared the SCNet+DCN+LSJ detection algorithm that we used with the HTC algorithm, Cascade_Mask_RCNN algorithm, and Mask_RCNN algorithm, which have advanced performance. For the tumor region nucleus classification algorithm, we compared the PS-ViT classification algorithm that we used with the mainstream ResNet50 algorithm and Vgg16_bn algorithm. Results are shown in Figure 4.

In Figure 4a, we present the results comparison of the tumor region segmentation models. We comprehensively evaluated the average Dice coefficient and the inference speed frame per second. It can be observed that the PSPNet+U-Net algorithm based on a pure convolutional neural network achieved a higher inference speed but with the lowest accuracy of 82.09. In contrast, the Segformer algorithm based on the Transformer achieved a higher average Dice coefficient (83.37) but with a significant decrease in inference speed. The proposed K-Net+Swin segmentation algorithm, combining the windowed Swin Transformer algorithm with a convolutional neural network, maintained a high inference speed, surpassing the pure Transformer-based Segformer, and achieved the best average Dice coefficient of 84.8.

Figure 4b illustrates the results comparison of the tumor region cell detection models using the P–R curve evaluated at IoU 0.5. Cascade_Mask_RCNN and Mask_RCNN algorithms performed similarly and less effectively, indicating that a simple cascade algorithm cannot address the sample consistency issue. To address this problem, the designed HTC algorithm and SCNet+DCN+LSJ algorithm achieved high performance, with SCNet+DCN+LSJ showing the best results.

In Figure 4c, the results comparison of the tumor region cell classification models is presented using ROC curves and AUC for evaluation. It can be observed that the ROC curve of the PS-ViT classification algorithm completely enveloped those of the ResNet50 and VGG16_bn algorithms. This suggests that the PS-ViT algorithm has an absolute advantage in distinguishing positive cells, as reflected in the AUC score, with PS-ViT having an AUC of 0.9666, higher than ResNet50 (0.9574) and VGG16_bn (0.9331).

3.2 Model inference

We inferred each model on the test set and performed breast Ki67 patch-level nucleus recognition and made inference visualization. In addition, we also performed full-image inference visualization on WSI images of breast Ki67. The abovementioned visualized inference results are shown in Figure 5.

3.3 Results of clinical validation

We validated our designed breast Ki67 quantitative analysis system comparing the results with the diagnostic results of clinical pathologists. Clinical validation data retrospectively selected a total of 80 patients with invasive breast cancer who underwent Ki-67 IHC examination in Hangzhou Obstetrics and Gynecology Hospital. The selection criteria for these patient data were that modified radical breast cancer specimens and histologically diagnosed patients with IDC. First, the 80 pathology slides were diagnosed by two experts, including a senior pathologist and a junior pathologist, with the junior pathologist having under 5 years of experience in pathology and the senior pathologist having up 10 years of experience in pathology. Then, we input these 80 Ki67 pathology slides into our AI-based breast Ki67 quantitative analysis system to get inference results. Finally, we performed a human and machine comparison of the diagnostic consistency. The results showed that the ICC of diagnosis result between the breast Ki67 quantitative analysis system and pathologists reached 0.964 (95% CI: 0.948–0.976), in which, the ICC of the breast Ki67 quantitative analysis system and senior physician reached 0.974 (95% CI: 0.960–0.984), and the ICC of the breast Ki67 quantitative analysis system and junior physician reached 0.944 (95% CI: 0.913–0.964). The results of the clinical validation are shown in Figure 6. The diagnosis results of our system are mostly in between the results of the senior and junior physicians, and were much closer to those of the senior physicians.

Figure 6

Comparison of clinical validation results and consistency of AI results of the Ki67 quantitative breast analysis system in cases.

We compared our designed breast Ki67 quantitative analysis system with QuPath v0.4.4 software. QuPath is an innovative bioimage analysis tool for digital pathology and whole-slide image analysis, offering comprehensive tumor identification and high-throughput biomarker assessment tools. Analyzing breast cancer WSI with QuPath software, as shown in Figure 7, we observed that QuPath excelled in accurately detecting normal cells (outlined in green), both in quantity and morphological contour. However, concerning positive cells (outlined in red), QuPath missed a considerable number of visibly positive cells, which were not difficult to distinguish. Additionally, QuPath often failed to accurately outline the edges of identified positive cells. As shown in Figure 6, the diagnostic consistency test results comparing QuPath’s inference results with expert assessments revealed an ICC of 0.658 (95% CI: 0.466–0.781) for diagnostic consistency with senior physicians and an ICC of 0.664 (95% CI: 0.476–0.784) with junior physicians. It is noteworthy that QuPath’s diagnostic results are generally lower than the scores provided by experts, directly related to the presence of numerous false negatives in QuPath software.

Figure 7

Comparison of QuPath detection results. There are 172 positive cells in the raw image. Only 76 positive cells are detected by QuPath, while our method is able to accurately identify 154 positive cells. Compared to the QuPath, our method performs twice as well. The detection accuracy of our method is 89.5%.

As an AI-assisted system, the ultimate goal of the breast Ki67 quantitative analysis system is to provide diagnostic assistance to professional doctors rather than replace them in the diagnostic process. Therefore, assessing whether doctors can improve diagnostic efficiency with the assistance of the AI system is a crucial factor in evaluating AI-assisted systems. We conducted a time comparison between the breast Ki67 quantitative analysis system, doctors diagnosing individually, and doctors diagnosing with the assistance of the AI system, and the results are shown in Figure 8. From the figure, it can be observed that doctors typically require approximately 22 min to diagnose a case. When doctors face uncertain situations or need to make conclusions at the threshold boundaries between negative and positive diagnosis, they tend to be more cautious, resulting in significant time and effort consumption. In contrast, AI diagnosis takes only about 3 min, with the inference time mainly influenced by the size of WSI and the number of cells within them. When doctors diagnose with the assistance of the AI system, the diagnostic time is reduced to approximately 5 min. This is because the AI system almost never misses positive cells, and doctors only need to focus on analyzing cells that are challenging to judge based on AI suggestions, ruling out false positives, and providing a comprehensive assessment based on their professional knowledge. This time comparison indicates that AI-assisted systems can significantly improve doctors’ diagnostic efficiency, reduce diagnostic time, allow doctors to address challenging situations in a more target-oriented manner, and maintain control over the overall diagnostic process.

Figure 8

Time efficiency comparison between Doctors, Doctors+AI, and AI. We used three different markers to represent different types of diagnostic times. The horizontal axis represents the three diagnostic ways, i.e., Dr, AI, and AI+Dr. Among them, “Dr.” stands for the time required for a doctor to diagnose alone; “AI+Dr” refers to the time spent by a doctor using our provided analysis system to assist in diagnosis; while “AI” is specifically used to represent the time, it takes for our analysis system to complete the analysis alone.