Improving the cleaning quality of tube lumen instruments by imaging analysis and deep learning techniques

TLI image acquisition

The sample data analyzed in this study were obtained from a central sterile supply department. Surgical instruments received between June 2021 and March 2022 were used and processed for manual or machine cleaning. Among these instruments, two common surgical TLIs, namely, suction tips and surgical forceps, were selected for the study because of their frequent use in the clinic. To perform channel inspection of the TLIs, an electronic endoscope model MAG-NJ400 was used. The field acquisition is shown in Figure 2(a). Due to the slender structure of TLIs, it is not possible to use the conventional camera spindle acquisition method perpendicular to the inspection surface [24], nor is it possible to design a multiangle acquisition method with multiple cameras arranged in a circular pattern [25].

Figure 2:

TLIs images acquisition strategy. (a) On-site photos of actual data acquisition. (b) Schematic diagram of the data acquisition method for axial feeds.

To solve this problem, an alternative image acquisition method was used. Figure 2(b) shows a schematic of the acquisition. We used a 0.3 megapixel CMOS camera sensor with a 120° field of view. At the front of the optical path, there was a thin LED exit, positioned off-center from the lens. The resolution of the acquired image was 640 × 480 pixels. The sensor optical path was extended inside the TLIs to perform axial image acquisition in the direction of the spindle by manually controlling the feed rate. During the inspection process, the inspector needed to focus on whether there were still foreign objects left inside and recorded the results of the inspection.

The acquisition of large-scale image data of TLI defects is very time-consuming. In this study, approximately 501 images were collected through imaging of TLIs that had been thoroughly cleaned after surgery to construct a research dataset. Despite the relatively small size of the dataset, we emphasize that these images are highly representative in clinical practice, representing typical defects and different coverage areas. During the data collection process, we relied on medical device cleaning quality monitoring technicians to label the image categories. These technicians have a long history of practical experience in medical device cleaning quality inspection. The image set is categorized into qualified and defective based on the type of data. A “qualified” image is completely free of residue and scratches inside the TLI after the cleaning process. In other words, a “qualified” image should be visually completely clean and spotless. “Defective” images include two main categories: blood residue and surface scratches, which represent poor cleaning quality. These defects are clearly visible in the image. In addition, we further classified the two previously categorized image types into two groups: the middle part and the end of the TLIs, based on the internal location of the images within the TLIs. According to the different combinations of image types and location information, the dataset consists of four types: middle-qualified TLIs (79 images), end-qualified TLIs (106 images), middle-defects TLIs (110 images) and end-defects TLIs (206 images). Among the middle-defect TLIs, 33 images show surface scratches, while the remaining 77 images display blood residue. Notably, the surface scratches are primarily concentrated in the middle section of the TLIs.

Data preprocessing

Due to the low resolution of imaging devices, image shaking during acquisition, internal structure irregularity and uneven aging inside TLIs, the TLIs images acquired usually show low contrast and unclear detailed information. In addition, the present TLI image dataset is relatively limited in size, which introduces some challenges to the training of deep learning models.

To address these problems, we used image processing methods. First, image enhancement was used, which tends to improve the quality and contrast of the image to present the detailed information of the image more clearly. Second, we introduced an image slicing analysis method, that can slice an image into multiple small pieces (subimages) to increase the size of the defect data. The sliced subimages retained all the details and reduced the computational load while meeting the model input size requirements. Finally, the threshold filtering method was used to exclude invalid data containing locally highlighted regions, thus improving the quality of the dataset.

Image enhancement

In this study, contrast-limited adaptive histogram equalization (CLAHE) was used for image enhancement [26]. This method utilizes the inherent ability of the human eye to perceive image contrast and brightness. The basic principle was to divide the image into several small image blocks and apply histogram equalization to each block, applying contrast limits to prevent noise. To further remove artifacts from the boundaries of each block, bilinear interpolation was applied after equalization. In the image enhancement method applied in this paper, the TLI images were first converted to the LAB color space. Contrast-limited adaptive histogram equalization was then applied to the luminance (L) component. To perform the equalization, the image was divided into 16 × 16 pixel blocks and a contrast limiting factor of 0.005 was applied to control the contrast within these blocks. The enhanced L component was then combined with the original A and B components to produce a new LAB image. Finally, the enhanced LAB image was converted back to the RGB color space to produce enhanced TLI images.

TLI image slicing

To further improve the completeness of the image analysis and to effectively address image boundary cases, we adopted an overlapping image slicing strategy. This strategy involved slicing the TLI images into a series of subimages with overlapping parts to ensure the continuity and completeness of the information. Specifically, we set the overlap ratio to 0.5, while the sliced image size was set to 224 × 224 pixels. By using Eqs. (1) and (2), a single original TLI image can be sliced into a set S = I 1 , I 2 , ⋯ , I h _ s e g × w _ s e g containing h_seg × w_seg subimages.

where seg_size is the size of the segmented image, overlap_ratio is the overlap ratio, ⌈x⌉ is x rounded up, height and wide are the height and width of the original image, respectively.

Subimages threshold filtering

For the set S of subimages that result from slicing, the total number of gray values in the subimages that exceeded a set threshold T (set to 208 in this paper) was calculated according to Eqs. (3) and (4). Next, by using Eq. (5), we calculated the ratio of the number of pixels that exceeded the threshold (N) to the total number of pixels (H*W). If this proportion exceeded the set threshold P (20 % in this paper), the subimage was labeled as invalid data and deleted from the set S.

where I _indicator (x) is the indicator function, I_{i, j} is the pixel value of image I at position (i, j), N is the sum of the number of pixels above threshold T, P is the proportion of pixels above the threshold T, and H and W are the height and width of the sliced image, respectively.

Transfer learning with ResNet for TLI image classification

Traditional CNN models are trained by randomly initializing weights and iteratively updating them using a backpropagation algorithm until convergence or stopping conditions are met. This training procedure is usually time-consuming, and to achieve satisfactory performance, a large number of training samples are required to ensure the reliability and generalizability of the model. At present, the available datasets are limited for TLI images, which makes traditional CNN models prone to overfitting.

Transfer learning has become a frequently utilized approach to solve this issue. Transfer learning methods include instance-based, feature-based, and shared parameter-based transfers [27]. Instance-based transfer requires that two tasks have similar instance distributions, and feature-based transfer depends on sharing similar low-level features. However, it is unwise to directly apply the pretrained model on ImageNet to the acquired TLI image dataset. Because of the differences in classes between TLI images and natural images, both transfer learning approaches may not be directly applicable to TLI images classification. In contrast, the transfer method based on shared parameters can better address this problem better. Local detail information of images, such as edges, textures and colors, is captured by low-level features, which are shared among different tasks. By sharing low-level features, the model training is accelerated, and the model performance is improved. The high-level features can be fine-tuned or adjusted for specific target tasks to better fit the characteristics of the TLI image dataset.

ResNet is a deep convolutional neural network model that shows excellent performance among many CNNs [28]. The unique residual block structure and the design of residual connections are the key factors in its success. By using residual connections, ResNet successfully solves the problems of gradient disappearance and gradient explosion. The connection enables the model to better capture the abstract features in the image, thus significantly improving the classification performance. Additionally, residual connectivity makes the network easier to optimize and train. ResNet provides several versions of the model, such as ResNet-101, ResNet-50, and ResNet-18 [29]. Each model version has a specific residual block structure that is suitable for feature representation of different classification tasks. In this study, we chose the ResNet model pretrained on the ImageNet dataset as a starting point and transferred it to the TLI image classification task. This transfer learning strategy could reduce the risk of overfitting, improve model performance, and accelerate the training of models.

Attention mechanism model

In this study, two attention mechanisms, namely the squeeze-and-excitation network (SE Net) [30] and concurrent spatial and channel squeeze & excitation (scSE) [31], are used separately to optimize the baseline model for TLI image classification. The SE Net attention mechanism has several advantages, including fewer parameters and easy embedding. It dynamically adjusts the importance of each channel by learning the relationships between channels, thus enhancing the focus on key features. On the other hand, the scSE attention mechanism represents an enhanced version of SE Net, which combines spatial attention and channel attention to capture the correlation between channels and spatial locations more completely in the feature map. The introduction of these attention modules in the baseline model is intended to evaluate the impact of different attention mechanisms on the classification accuracy of TLI images and to improve the distinction between qualified and defective TLIs.

SE Net is an attention mechanism that adaptively learns the importance of each channel. It consists of two main steps: squeeze and excitation. The squeeze step entails a spatial compression of the feature map by global average pooling, which compresses the dimension of the original feature dimension of the original feature map from H × W × C to 1 × 1 × C. This step enables the feature map to perceive a wider range of global information and helps to extract important channel features. To enhance model generalizability, excitation is applied to compress and reduce the number of feature map channels through two fully connected layers to obtain the correlation between channels and to calculate the importance weight of each channel for the overall feature representation.

scSE adopts a parallel spatial attention and channel attention mechanism. The channel attention part of scSE in this paper retains the SE Net structure for capturing correlations between different channels in the feature map. Spatial attention is performed using a 1 × 1 convolution of the channel dimension of the feature map, which is compressed from the original H × W × C dimension to the H × W × 1 dimension, and then the importance weight of each spatial location is obtained by a stimulus operation. Figure 3 illustrates the network structure of scSE.

Figure 3:

Concurrent spatial and channel squeeze & excitation structure diagram.

Performance evaluation metrics

Using the model to classify TLI images, we focused on model’s ability to accurately identify TLIs with defects. The accuracy, recall, precision, and F2 score were used as evaluation metrics to assess the performance of the mode. In addition, to reduce the potential risk of missed detections, we used the F2 score, which emphasizes recall more than the F1 score. Considering that TLI image classification is a multicategory problem, we calculated the precision, recall, and F2 score for each category and then averaged them as shown in Eqs. (6)–(9). This process helps to comprehensively evaluate the performance of the model in the TLI image classification.

where N denotes the number of classes and is set to 4; β is set to 2; TP _i denotes the number of true positives in the i-th class; TN _i denotes the number of true negatives in the i-th class; FP _i denotes the number of false positives in the ith class; and FN _i denotes the number of false negatives in the i-th class. Recall _i denotes the recall rate of the i-th class, and precision _i denotes the precision rate of the i-th class.

Data preprocessing

Comparisons are made between the various image enhancement outcomes of CLAHE, HE, and contrast adjustment methods. As shown in Figure 4(a), the CLAHE method yields better results in processing the TLI images. It can effectively enhance the details of the image by expanding the distribution of the gray histogram, which can improve the visual effect and make defects more visible. In contrast, HE is limited in its effectiveness in processing local areas that are too bright or too dark, thus easily leading to a loss of detail. Contrast adjustment methods result in changes to image brightness and color. Moreover, the contrast adjustment method yields insignificant results when applied to low-contrast areas.

Figure 4:

Image preprocessing process and results.

Image slicing and threshold filtering were implemented on all the images after image enhancement. For example, one of the images was segmented into multiple small blocks, as shown in Figure 4(b-2). Then, the highlighted regions that were not relevant to TLI image detection were removed by threshold filtering, as shown in Figure 4(b-3). Finally, the sliced images were separated. Namely, the nondefective sliced images were distinguished from the defective sliced images, as shown in Figure 4(b-4).

The results obtained after applying image slicing and threshold filtering are shown in Figure 4(b). As seen from the sliced images, each slice contained local information about a specific region, which improved the quality of the dataset. Importantly, the sliced images completely captured the defective parts, thus increasing the diversity of the data. Moreover, the sliced image size met the input requirements of the ResNet model and helped to reduce the computational load of the model.

After image enhancement, slicing, and thresholding, we obtained a TLI image dataset with four categories: 2,633 images of End-defect TLIs, 1,291 images of Middle-defect TLIs, 2,078 images of End-qualified TLIs, and 1,992 images of Middle-qualified TLIs.

Selected baseline model

Considering the large difference between TLI images and the ImageNet dataset, low-level features can share basic features while high-level features are more abstract and more important for the classification of TLI images. Therefore, two different strategies were used in transfer learning to adapt the TLI image classification task and determine the best baseline model.

The first approach: ResNet model with all the convolutional layers frozen (FA-ResNet). In this approach, a ResNet model pretrained on the ImageNet dataset was used, and the weights of all the convolutional layers were frozen. Then the original fully connected layers and classification layers in the model were removed. A new fully connected layer was added with the number of output nodes set to 4, and a classification layer with 4 node output nodes was added. This approach allowed the model output to be mapped to the 4 classes and the class probabilities to be calculated using the softmax function. Using this fine-tuning approach, the last three layers of the model were trained, including the newly added fully connected layer, the softmax layer, and the weights of the previous layer.

The second approach involves the use of a ResNet model with fine-tuned partial convolutional layer weights (FT-ResNet). Based on the first model, we unfroze part of the convolutional layers of the ResNet model and allowed them to participate in training. However, as the number of unfrozen layers increased, the learned generic features were destroyed, leading to overfitting in the TLI image classification task. Therefore, the strategy of fine-tuning only the last ResNet convolutional block was chosen for this study.

Two different transfer learning approaches were used for the three models: ResNet18, ResNet50 and ResNet101. The performance of the models was evaluated on the TLI image classification task to determine the better baseline model. To ensure the stability of the model’s recognition results, we employed 5-fold cross-validation. For each fold, data augmentation techniques, including random horizontal and vertical flipping, were applied to the training set to increase the diversity of the data. These augmentation methods effectively prevented the neural network from overfitting and avoiding memorization of training image details, thereby enhancing the model’s generalizability. It is important to note that the augmented images were only used in the training set and did not appear in the test set. The model was trained using the Adam optimization algorithm with 20 epochs, an initial learning rate of 0.003, and a batch size of 64.

As shown in Figure 5, FA-ResNet18 outperforms FA-ResNet50 and FA-ResNet101 in classifying the End-qualified and Middle-defect categories, demonstrating stronger recognition capabilities in these areas. In contrast, the deeper model FA-ResNet50 excels in classifying the End-defect and Middle-qualified categories. FT-ResNet50 outperforms FT-ResNet18 and FT-ResNet101 in the End-qualified and Middle-defect categories, while FT-ResNet101 performs better in classifying the End-defect and Middle-qualified categories compared to the other models. However, all models still exhibit noticeable misclassifications between the End-defect and End-qualified categories, indicating that there is room for further improvement in this classification task.

Figure 5:

Confusion matrices of baseline models after cross-validation. (a) FA-ResNet18. (b) FA-ResNet50. (c) FA-ResNet101. (d) FT-ResNet18. (e) FT-ResNet50. (f) FT-ResNet101.

To quantitatively evaluate model performance, Table 1 presents a comparison of the classification results based on various metrics. When all convolutional layers were frozen, the transfer learning classification performance of ResNet18 and ResNet50 exceeded 95 %, outperforming ResNet101, which achieved 93.8 %. After fine-tuning some convolutional layers, both ResNet50 and ResNet101 surpassed 95 %, outperforming ResNet18, which achieved 94.3 %. However, training the ResNet101 model took significantly more time. A comparison of the two transfer learning approaches shows that, considering both model performance and training time, the FA-ResNet18 and FT-ResNet50 models provided better performance in TLI image classification. Furthermore, the classification results of both models are close to the human eye’s recognition accuracy for natural images (94.9 %) [28].

In addition, we also evaluated the performance of smaller models, including GoogleNet, MobileNet, and EfficientNet, in TLI image transfer learning. According to Appendix Table 1, EfficientNet demonstrated the best classification performance, with all evaluation metrics exceeding 96 %. MobileNet followed, closely matching ResNet50’s performance but exhibiting higher variability, as indicated by its larger standard deviation. GoogleNet demonstrated slightly lower performance, with classification metrics below 95 % and the highest standard deviation among the models. It is important to note that EfficientNet requires significantly longer training times compared to the other models. Considering both model performance and training efficiency, we selected FA-ResNet18 and FT-ResNet50 as the baseline models, as they provide a better balance between performance and computational efficiency.

Optimizing the baseline model with attention mechanisms

After determining the baseline models, we introduced the SE and scSE attention mechanisms to optimize the model. Initially, all convolutional layers of the baseline model were frozen. Then, the fully connected and classification layers were removed, and the SE and scSE attention modules were appended after the final convolutional block of ResNet, resulting in four improved models: FA-ResNet18-SE, FA-ResNet18-scSE, FT-ResNet50-SE, and FT-ResNet50-scSE. This approach aimed to retain the baseline model’s overall structure and feature extraction capabilities while enhancing the model’s ability to focus on key regions in TLI images. All comparative experiments were conducted using 5-fold cross-validation to accurately assess the impact of incorporating the SE and scSE attention mechanisms on the performance of each baseline model.

Figure 6 displays the overall confusion matrices of models with attention mechanisms after cross-validation. Among these models, FA-ResNet18-scSE outperforms the others in the End-defect category, while the results for the End-qualified and Middle-qualified categories are quite similar across the models. In the Middle-defect category, FA-ResNet18 with attention performs better than FT-ResNet50 with attention.

Figure 6:

Confusion matrices of models with attention mechanisms after cross-validation. (a) FA-ResNet18-SE. (b) FA-ResNet18-scSE. (c) FT-ResNet50-SE. (d) FT-ResNet50-scSE.

The performance comparison of models with attention mechanisms, based on 5-fold cross-validation, is shown in Table 2. FA-ResNet18-scSE achieves the highest accuracy at 98.34 %, slightly surpassing FA-ResNet18-SE, which achieves 98.20 %. Both FT-ResNet50-SE (98.12 %) and FT-ResNet50-scSE (98.17 %) show similar performance, with slightly lower accuracy compared to FA-ResNet18-scSE. In terms of macro recall, FA-ResNet18-scSE leads with 98.31 %, followed by FA-ResNet18-SE at 98.28 %, FT-ResNet50-scSE at 98.15 %, and FT-ResNet50-SE at 98.14 %. Similarly, FA-ResNet18-scSE also shows the highest macro precision at 98.35 %, with FA-ResNet18-SE at 98.21 %, FT-ResNet50-scSE at 98.19 %, and FT-ResNet50-SE at 98.15 %. Finally, for the macro F2-score, FA-ResNet18-scSE remains the top performer with 98.32 %, while FA-ResNet18-SE, FT-ResNet50-SE, and FT-ResNet50-scSE follow with scores of 98.26 %, 98.14 %, and 98.16 %, respectively.

Overall, in the TLI image classification task, models with attention mechanisms exhibit strong performance across all metrics, with those incorporating the scSE mechanism showing a notable improvement. FA-ResNet18-scSE consistently outperforms the other models in accuracy, macro recall, macro precision, and macro F2-score, demonstrating superior classification capability. While FT-ResNet50-SE and FT-ResNet50-scSE perform similarly, their results are slightly lower compared to FA-ResNet18-scSE. The results show that models with attention mechanisms achieve detection levels higher than those of human eyes. The optimized model can identify defective TLIs more accurately, providing more reliable support for TLI cleaning quality inspection.

We further studied the effect of image quality on classification performance by inputting the enhanced and original images into the FA-ResNet18-scSE model. The final classification results are in Appendix Table 2.

The results show that the enhanced image leads to better model classification performance, which emphasizes that image enhancement has a positive effect on improving the model classification performance. On the other hand, downsampling of the original image is required when using CNN models for classification, which may result in loss of image details. In contrast, the method using image slicing is able to retain more pixel and detail information from the original image while meeting the CNN input requirements. It further improves the classification performance on TLI images and overcomes the problem of information loss caused by downsampling.