Deep learning assisted intraoperative instrument cleaning station for robotic scrub nurse systems

1 Introduction

In the past few years, the application of robotic scrub nurse systems (RSN) has been investigated in the scientific community [1] due to the ongoing shortage of qualified operating room (OR) assistants [2] and the drive for automation. Robotic scrub nurses assist surgeons in the selection and delivery of surgical instruments. In this regard, such robotic systems should fulfill all indirect and direct forms of surgical assistance that are currently provided by human OR assistants. This includes not only the provision of required instruments and the reception of instruments and surgical specimens that are no longer required, but also the intraoperative cleaning of instruments in order to ensure a complication-free surgical workflow. Current RSNs such as the well-known Gestonurse [3] and the 3D gaze-guided [4] systems do not address the possibility of intraoperative cleaning. To provide a fully supportive RSN system, we are developing the Situation Aware Sterile Handling Arm for the OR (SASHA-OR), a robotic platform that is context-sensitively integrated into the surgical workflow of minimally invasive interventions and can make autonomous decisions about the delivery and cleaning of laparoscopic instruments.

The cleaning aspect in particular has gained importance with the increasing use of electrosurgical techniques, which have become indispensable during minimally invasive procedures due to significant advantages compared to conventional cutting techniques using a scalpel, including time benefits [5] as well as minimization of patient blood loss [6]. Electrosurgery or high-frequency surgery refers to the application of high-frequency electric current to cut tissue (electrotomy) or seal bleeding vessels (electrocoagulation) [7]. Generally, a distinction is made between these two types of application. In electrotomy, when tissue is cut with electric current, the current heats the tissue very quickly so that the cell wall explodes due to the resulting pressure. In contrast, in coagulation, the current heats the tissue more slowly, so that the extracellular and intracellular fluids evaporate. As a result, the tissue shrinks and open blood vessels are sealed [5]. However, both monopolar and bipolar application techniques result in adhesions and contamination on the laparoscopic instrument end effectors [8].

Sterile OR assistants accordingly perform intraoperative cleaning procedures to roughly remove residual blood and tissue that can interfere with the work flow of a surgery [5]. Inadequate removal of organic contaminants increases the risk of intraoperative complications. For example, adhesions of blood to the shaft of instruments lead to increased resistance during insertion into the trocar shaft [9]. When removing a used instrument, the sterile OR assistant cleans it manually by hand with a suitable textile such as an abdominal linen. Alternatively, cleaning can be done either by wiping with a soft, disposable cloth soaked in cleaning fluid [10] or by using a sterile, lint-free sponge moistened with water [11]. This process is illustrated in Figure 1.

Figure 1:

Sterile OR assistant cleans a laparoscopic instrument intraoperatively with a disposable cloth soaked in cleaning fluid.

To extend RSN systems by the possibility of intraoperative cleaning, we present a deep learning assisted cleaning station for robotic scrub nurse systems. The cleaning station provides autonomous intraoperative cleaning of laparoscopic instruments during minimally invasive procedures. In addition, the system uses deep learning methods to decide on the need for intraoperative cleaning to preserve instrument functions.

2 Methods

2.1 Workflow integration

For autonomous decision-making on the need for intraoperative cleaning, a camera mounted to the robot arm is used. In the first step, the camera detects the tip or end effector of the instrument after it is placed back on the instrument tray by the surgeon. The system then uses the image section of the instrument tip to decide whether contamination by residual blood or tissue is present and the instrument should be cleaned. If the system decides that cleaning is required, the RSN system picks up the laparoscopic instrument at the shaft and moves to the cleaning station attached to the robotic platform. At the cleaning station, the instrument is immersed in the cleaning container and cleaned. After the cleaning process, the robotic handling arm places the instrument back on the instrument tray.

2.1.1 Autonomous cleaning decision

The computational pipeline of the autonomous cleaning decision can be divided into the detection of the instrument tip and classification of its contamination. The proposed system, illustrated in Figure 2, is implemented in Python and deployed with the NVIDIA Clara AGX Dev Kit.

Figure 2:

Computational pipeline of the two-stage algorithm.

2.1.1.1 Tip detection

For instrument tip detection, we use the YOLOv5 architecture [12]. To create a dataset for training and evaluation of the proposed method, 3694 images of laparoscopic instruments were acquired from different heights of the instrument tray used in our experimental setup. The images were captured with an Intel RealSense D435i depth camera. The laparoscopic instruments included those predestined for contamination during minimally invasive procedures such as the scissors, probe excision forceps, grasping forceps or the bipolar grasping forceps. All images were annotated with bounding boxes on the instrument tip using the annotation tool CVAT [13]. The generated dataset was shuffled and split into a training (70 %) validation (20 %) and test (10 %) dataset. We trained the YOLOv5m model for a total of 30 epochs using the training set with a batch size of 16 and an initial learning rate of 0.01, which was reduced to 0.001 after 30 epochs. The hyperparameters, including bounding box size, batch size, and learning rate schedule were tuned based on the validation set. To avoid overfitting and improve overall performance, data augmentation was conducted during online training. It did not affect the number of available training images, but randomly changed the available images in each epoch.

2.1.1.2 Contamination level

While the YOLOv5 network focuses only on instrument tip detection, we use a simple Convolutional Neural Network (CNN) at the second stage to decide on the need of cleaning an instrument. For this purpose, the region of interest generated by the YOLOv5 network is resized to the size of 128 × 128 pixels. The CNN deciding on the need for cleaning consists of two convolutional layers [5 × 5 conv, 8], a max pooling layer [5 × 5], two convolutional layers [3 × 3 conv, 16], a max pooling layer [3 × 3] and three fully connected layers. Each layer is activated by the RELU function as shown in Figure 3. For training the CNN, we acquired 3376 images of above mentioned laparoscopic instruments in clean and contaminated conditions. Similar to the tip detection pipeline, the generated dataset was split and shuffled into a training, validation, and test dataset. The CNN was trained for a total of 15 epochs using the training set with a batch size of 16 and an initial learning rate of 0.01. We also augment and normalize the training dataset to avoid overfitting.

Figure 3:

Convolutional neural network for contamination level detection.

The tip detection provides a potential bounding box of the instrument tip for each frame at time t and its output probability p _TD,t . The contamination level detection provides the class c _CL,t (clean = 0, contaminated = 1) and its output probability p _CL,t . For robust detection and as a starting point for manipulation of an instrument s _t , we aggregate the normalized information of the last N = 24 frames:

(1) s t = ∑ t N c C L , t ⋅ p C L , t ⋅ p T D , t N

If s _t exceeds the value of 0.5, the cleaning task is initiated.

2.1.2 Cleaning station

Once the cleaning task is initiated, the robotic scrub nurse system picks up the contaminated laparoscopic instrument at the shaft and moves it to the cleaning station. The cleaning station, which is shown in Figure 4, consists of an embedded cleaning container, a motion unit, an inductive proximity sensor and a power unit. The cleaning container is equipped with nylon cleaning brushes and a cleaning fluid and is 3D printed using Fused Deposition Modeling. This allows the container to be replaced after each operation to meet the required hygiene standards. The motion unit consists of a servo motor mounted on a linear rail, a stationary nut attached to the enclosure, as well as a spindle upon which the cleaning container is mounted. When the contaminated instrument is inserted into the cleaning container, the proximity sensor detects the metallic instrument tip and triggers the container motion sequence. The servo motor rotates the spindle, which in turn causes a simultaneous rotation and vertical translation of the cleaning container due to the stationary nut.

Figure 4:

Hardware integration of cleaning station. (a) Cleaning station with enclosure attached to the robotic platform. (b) Actuator technology of the cleaning station.

3 Results

3.2 Configuration and durability tests

The aim of the configuration tests was to determine the ideal combination of system parameters for an optimal system performance. The results of the tests, shown in Figure 5a and b, indicate that the combination of configuration A with a cleaning interval of t _iv = 3 s resulted in a minimal total cleaning time of t _clean,1 = 9 s in contamination scenario S ₁ and t _clean,2 = 12 s in S ₂. Examining the cleaning intervals shows that a longer cleaning interval of t _iv = 3 s yielded better cleaning results per interval than a shorter interval of t _iv = 2 s, resulting in fewer overall required cleaning intervals. Comparing the different combinations of cleaning brush b _c and cleaning fluid f _c reveals that the hard brushes were able to remove contamination more effectively than the soft brushes. This is especially apparent in S ₂, where the soft brushes failed to clean the instrument to a sufficient degree, regardless of the cleaning fluid or cleaning interval. Moreover, the use of NaCl as a cleaning fluid appears to improve the system performance compared to H₂O, as the configurations using NaCl performed better than the ones using H₂O in all examined scenarios.

Figure 5:

Results of configuration tests. (a) Contamination scenario S₁. (b) Contamination scenario S₂.

Using the optimal configuration and cleaning interval obtained through the configuration tests, the aim of the subsequent durability tests was to evaluate the durability of a cleaning container under repeated use. For this purpose, we measured the performance of the system in configuration A with a cleaning interval of t _iv = 3 s. The results of the durability tests show that the evaluated cleaning container was capable of withstanding n = 5 consecutive cleaning procedures when removing blood residue, and n = 4 consecutive procedures when removing coagulated coarse tissue residue.

4 Discussion

The results of the computational pipeline of autonomous contamination level detection show that it is sufficient for fast and robust operation during robotic assisted minimally invasive procedures. The two stage algorithm provides accurate detection of the instrument tip as well as the degree of contamination. The decision-making rapidity of the algorithm on the need of cleaning can be considered comparable to that of a human OR assistant. Nevertheless, the system uses a supervised learning approach, which may lead to non-perfect predictions in unknown data. A possible alternative method could be an unsupervised learning approach, being trained to detect anomalies to provide a better degree of generalization.

The configuration tests show that our proposed cleaning system is capable of removing contaminations such as coagulated coarse tissue residue and blood residue without any problems in a configuration using hard brushes and NaCl as cleaning fluid, with a cleaning interval of t _iv = 3 s.

The durability tests show that the cleaning function is in principle guaranteed for the duration of a surgical intervention. However, for a more valid statement regarding durability, the tests should be reproduced with a larger number of test runs. Furthermore, the failure condition used in the durability tests could be reconsidered, as a hard limit after a single unsuccessful cleaning procedure does not necessarily represent the process performed during a manual intraoperative cleaning task.

In the course of our experimental evaluation, we additionally identified other limitations of our system and the corresponding test setup. One main limitation is presented by the fact that we only implemented one cleaning method using moving brushes, while it would be interesting to investigate other modalities such as ultrasound baths or pressurized jets.

A challenge during the configuration tests was the repeatability of initial instrument contamination levels, as it was impossible to reproduce the exact contamination pattern for each test run. While this factor may have influenced the results of the cleaning tests, we chose to use this method of evaluation regardless, as in a real-world scenario no two instrument contaminations are identical either.

5 Conclusions

In this paper, we presented our deep learning assisted intraoperative instrument cleaning station for robotic scrub nurse systems. We conclude that the system ensures fast and robust detection of the contamination level of laparoscopic instrument tips via a two-stage deep learning algorithm. Using a modular cleaning station, our robotic scrub nurse can clean contaminated instruments by simply immersing the instrument into the autonomously actuated cleaning container. While the performance of the computational pipeline was comparable to that of a human OR assistant, we identified some constraints of the mechanical system offering further potential for improvement. Overall, the proposed system supports the general practicability of robotic scrub nurses for minimally invasive interventions.