Improving endoscopic smoke detection with semi-supervised noisy student models

Introduction

A surgeon has many tasks which require manual intervention and focused attention. Automation of such tasks can alleviate the burden on surgeons and allow their attention to stay on more important topics. One such task is manual smoke evacuation. Correct handling of surgical smoke consists of multiple steps and would profit highly from automation. Surgical smoke is a byproduct of coagulation tools like the high-frequency electrosurgical unit or coagulating shears. It diminishes the quality of the endoscopic vision with increasing intensity and must be removed from the body. Smoke evacuation must be accompanied by an increase of gas flow into the body to keep the pressure in the cavity constant. Additionally the evacuation is best handled by a specific device to avoid health risks to the surgical staff [3]. Aside from this intraoperative uses of smoke detection, there also exist various postoperative applications for recording analysis. Large amounts of smoke in a recording may indicate surgical errors due to the deteriorated quality of the endoscopic vision. The automatic localization of smoke in recordings can also help to navigate through the vast amounts of recordings that hospitals have to store. A large amount of research on automatic smoke detection focuses on the more general fire and smoke detection in outdoor videos. [4] applies convolutional neural networks (CNNs) for fire classification. The work in [5] uses the Faster R-CNN object detection model to determine the location of smoke and fire in images. The authors also try to overcome the lack of data with synthetic smoke images. In the surgical domain few works exist on the subject of automatic smoke detection. Manually extracted features of laparoscopic recordings lead to very good results with a SVM classifier in [6]. This work evaluates smoke detection on only four recordings and reports metrics on a testset randomly sampled from all data. Another SVM based algorithm is used in [7] with good results. This work also suffers from the very limited dataset of only 76 extracted clips with less than 5,000 frames in total. The work in [8] evaluates a histogram method based on colorspace features as well as a pretrained GoogleNet CNN on a non-public dataset with 30,000 images. The same authors published smoke annotations for parts of the cholec80 dataset [9] together with new evaluations in [1]. To overcome the lack of available training data for surgical data science in endoscopy several works evaluated the use of unlabeled recordings for surgical workflow analysis [10] or instrument segmentation [11]. The work in [2] utilizes large amounts of unlabeled data with a pseudo-labeling approach to improve a noisy student model. A teacher model is trained in a supervised way and used to create pseudo labels for the unlabeled data.

This work evaluates the use of pseudo-labeled data to improve the state of the art in smoke detection with CNNs. Student models are learned iteratively with increased capacity to achieve knowledge expansion. Input and model noise is applied to the student model increasing the range of learnable invariants. This algorithm is used iteratively to get the most out of the available data.

Materials

This work uses the following datasets: The cholec80 dataset [9] with the public smoke annotations from [1] referred to as dataset I, the part of cholec80 without annotations as dataset II and a non-public unlabeled dataset showing similar laparoscopic procedures designated dataset III. The available smoke annotations have been created on a subset of the cholec80 dataset extracted at 25 frames per second. The roughly 100,000 annotated frames of this dataset show minimal variation in parts due to this high framerate. Extracting test and validation datasets with random sampling would therefore lead to information leakage and unrealistic high metrics. This makes it necessary to choose the validation and test subsets from different videos to show a more realistic generalization error. Sixty videos are used for the training set, and 10 videos each for validation and test. This results in 75,622 images for training and the rest for validation and test. For the unlabeled part of the cholec80 dataset pseudo labels are created at one frame per second. This gives 159,440 additional images. Pseudo labels for the non-public dataset are created in the same way for 257,611 images.

A rectangular area is cropped from the center of the cholec80 images to reduce the influence of the dark circle as used in [1]. The images of both datasets are cropped to the same aspect ratio, resized to square images with size 256 by 256 pixels. Finally data augmentations are applied and the image data is normalized to the range [−1, 1].

Methods

We compare the two best performing methods, Saturation Peak Analysis (SPA) and classification with a CNN, from [1] which represent the state of the art with different noisy student models.

Results and discussion

The top two rows of Table 1 show the metrics for the state of the art, taken from [1]. SPA in the third line shows the results on the testset for our own reimplementation of this algorithm: results are very near to the original, higher precision suggests a reduced number of false positives, the lower recall an increased number of false negatives. Row four shows the testset metrics for the teacher model (D121). It shows an increase in all metrics compared to GoogleNet (GLN RGB) due to the modernized neural network architecture.

To evaluate the noisy student algorithm the datasets were used as follows. Dataset I, the part of the cholec80 dataset with public smoke annotations is used to train the teacher model. The student models were trained with datasets I, II and III. Dataset II, the unlabeled part of cholec80 with pseudo labels. And dataset III, the non-public dataset with pseudo-labels.

Table 2 shows student models, of both sizes, trained on two of the three datasets: dataset I and II or dataset I and III. It can be seen that using any one of both pseudo-labeled datasets improves performance compared to the teacher model by approximately the same margin. The numbers also show, that larger student models trained on two of the datasets perform slightly worse than a same-sized student.

Finally Table 3 shows the results for student models trained on all three datasets. Utilising more pseudo-labeled data shows an improvement for the larger models (D169) over the small students (D121) by a small margin. The ratio of pseudo-labeled samples is further increased for the models in the last two rows. The D121 model (penultimate row) shows almost no change in metrics after this change of ratios, except a small substitution of false negative and false positive samples. This leads to an increase of precision and a decrease of the recall metric. The D169 model (last row) on the other hand shows distinct improvements. Accuracy, precision and f1 metrics show an increase, with only recall being in the same range as the others or slightly lower than the first model. A nearly constant recall metric is an indication that the number of false negatives for this best model stays approximately the same. Meaning that the recognition of the most difficult examples is not improved by much. Whereas the growth of the other metrics points to a decrease in the number of false positives. In other words, the reduction of false positives suggests less overfitting, meaning better generalization.

Figure 1 depicts results for the best model. False negatives in the first row containing low amounts of smoke, making it hard to recognize even for the human eye. The second row shows smoke-free images that were wrongly determined to contain smoke. The image on the left shows the typical color distribution of laparoscopic images, the classifier might have been fooled by the white discoloration on the tool dominating the bottom right. The image on the right with a specimen retrieval bag and slightly blurred vision is most likely misclassified due to these greyish features. The last row depicts two correctly classified smoke images. The amount of smoke is high enough that the human eye can distinguish the smoke even in this small image size.

Figure 1:

Examples of model predictions. The rows contain from top to bottom: false negative, false positive and in the third row true positive smoke detections.

Conclusion

A method for utilising unlabeled endoscopic recordings to improve smoke detection was proposed. In a first step the state of the art is improved upon by modernizing the employed model architecture. The results show that further improvements can be reached by utilising increasing amounts of pseudolabeled data and by scaling the network size accordingly. The prediction of less false positives also shows that the robustness of the classifier is improved. While the results are promising there are still many false negatives in the predictions. Performance on these harder examples, showing only small amounts of smoke when it is emerging or during the evacuation might be improved by using more training data with such edge cases.