Band 10, Heft 1 - Proceedings of the CURAC 2024 Annual Meeting, Leipzig, September 12–14, 2024

The level of digitalization and automation, including robotic systems, is increasing in modern operating theaters. However, the integration of such systems within an already complex and limited environment creates challenges, such as collaboration with dynamic robots and thus potential collisions. Yet, the placements of robotic systems largely rely on empirical guidelines provided by manufacturers, which do not account for the specific requirements of operating theater layouts or individual surgical procedures. In this work, we propose a framework for optimization of robotic set-ups in the operating theater to address these limitations and to enhance the overall workflow efficiency. By utilizing a simulation-based approach, we aim to determine the optimal placement of each surgical equipment within the constrained operating theater environment. In this study, we set the stage for future efforts and demonstrate the effectiveness of our approach by determining the optimal placement of a LBR iiwa 7 R800 as a surgical assistant for an abdominal surgery. Overall this work lays the groundwork for future research that will accommodate more complex scenarios and is a crucial first step towards a holistic workflow optimization in the operating theater.

During maxillofacial surgery, the precise placement of surgical tools is crucial for accurate implant placement. Particularly if multiple implants are needed, e.g., after cancerous bone removal, visual landmarks might not be obtainable. To this end, we propose a vision-based tracking approach with deep learning. Our markerless tracking approach is based on video streams from two cameras for tracking a mandible phantom. We study, to what extent vision-based localization using deep learning is feasible. For real-time 6D pose estimation we propose a Siamese network with a ResNet-18 subnetwork. We acquire a large training dataset with a robot and evaluate the tracking accuracy on partially occluded images. Thereby, we mimic visual information that is accessible during clinical interventions. We report a mean position error of 1.33 ± 1.14mm and a rotation error of 0.86 ± 0.71 deg for partially occluded images. Overall we present a promising tracking approach that is marker-free and robust toward image artifacts.

Force feedback is currently missing in most surgical robot systems, even though it would be essential for many surgical tasks, such as palpating tissue. In this study, we used a previously developed robotic endoscope based on series elastic actuation and tested the feasibility of using it for force feedback with a pilot study. The participants had a lower test time (−27%) and lower applied forces (−23%) with force feedback enabled. While further studies are needed for validation in realistic surgical scenarios, these results suggest improved safety and efficiency in telemanipulated surgeries.

Knowledge of contact forces between instruments and blood vessels during endovascular interventions like thrombectomies can help make these interventions safer and faster. We show that it is generally feasible to derive the contact forces directly from the intraoperative fluoroscopy imaging system which can make the costly integration of additional sensors into catheters and guidewires dispensable. Our study is limited to stationary normal loading and planar deflections. In our loading scenario of crossing the carotid siphon with a guidewire, the magnitude of contact forces can be detected up to an average deviation of 4.5 % for an imagebased pose measurement accuracy of the guidewire of 1 mm.

Limited or absent haptic feedback is reported as a factor hindering the continued adoption of surgical robots. This article presents a proof of concept for vibrotactile feedback integrated into a continuum robot to explore whether such feedback improves spatial perception in surgical settings. The robot is equipped with a capacitive sensor for noncontact endoscope localization, enabling spatial awareness of the robot’s tool center point (TCP) within the surgical environment. The data from the sensor is processed and transmitted to a bracelet worn by the user, which generates vibrotactile feedback. The bracelet contains four vibration motors providing tactile cues for navigation and positioning of the robot’s TCP. All subsystems are integrated into a unified system to deliver vibrotactile feedback to the user. When the user maneuvers the TCP of the robot near an object, they receive vibrotactile feedback via the bracelet. Thereby, the intensity of vibration increases as the TCP approaches the object, and the direction of the obstacle is mapped on the bracelet. Initial functional tests were performed and prove the functionality of the proposed system.

For the visualization of pulmonary ventilation with Electrical Impedance Tomography (EIT), most devices rely on generalized reconstruction models. Yet, fixed thorax dimensions, predetermined electrode locations, and standard lung shapes lead to multiple sources of EIT imaging distortion. The following work compares reconstructions of a library model, a practical model based on landmark fitting, and a detailed model based on manual segmentation. CT images of five pigs were segmented into torso surface, electrode positions, and lung borders. Practical models were created with reduced data, registration, and fitting. Detailed models were created by using the complete segmentation data. After EIT reconstruction and tidal image calculation, the overlap of CT lung segmentation and ventilated voxel was calculated. Compared with the results of a standardized model, the practical model reached an average ventilation/segmentation-overlap difference of an additional 12 %, and the detailed model achieved an average difference of 10 %. In conclusion, the shift of reconstructed impedance towards the segmented lung region is similar in both models when compared to the standard model, while the practical model requires a considerably less amount of information.

Mitral valve regurgitation is one of the most common heart valve diseases that can occur when the structural composition of the mitral valve is affected. Mitral valve repair, typically performed in a minimally invasive setting, is a complex surgery that aims to reinstate the structural integrity of the valve and thereby restore normal valve function. The current practice lacks quantification of geometrical changes and interactive 3D visualization that could enhance the understanding of pathomorphological changes and impact of surgical correction. This work presents functionalities from smartMVR, a quantitative tool to measure, compare, and analyse changes in mitral valve geometry from data captured on-demand using an RGB-D camera. The feasibility of the approach is demonstrated on a silicone replica of a pathological mitral valve that is iteratively corrected using common surgical techniques.

Liver vessel segmentation in computed tomography represents a highly challenging task due to the imbalanced distribution within the liver parenchyma, the small and branched vessels with decreased image contrast to surrounding tissue and in general, due to the scarcity of highresolution and -contrast images, which hampers the efficient training of deep learning-based approaches. This study applies two state-of-the-art networks, the fully convolutional nnUnet and the transformer-based VT-Unet to three publicly available datasets, 3DIRCADb, one task of the Medical Segmentation Decathlon (MSD) and the more recent LiVS dataset. The nnUnet achieved Dice scores of 0.761, 0.714, and 0.696 on the 3DIRCADb, LiVS, and MSD datasets, respectively. In contrast, the experiments with the VT-UNet resulted in Dice scores of 0.795, 0.713, and 0.610. These findings indicates good accordance of the performance of the nnUnet and the transformer-based VT-Unet, with differences regarding individual datasets. Both network variants show competitive performances regarding the current state-of-the-art, yet the need for large-scale and high-quality datasets becomes evident to further enhance the accuracy of liver vessel segmentation.

Generating synthetic contrast-enhanced liver MRI scans from native MRI images can serve to mitigate the issue of sparse contrast-enhanced image datasets while concurrently circumventing the time-consuming and costly process of administering contrast agents during image acquisition. In this study, we conducted three experiments using paired image-to-image translation techniques. Native T1 sequences showing the abdominal liver region served as the input, while contrast-enhanced T1 sequences were the target. The data preprocessing methods and image boundaries were varied for the individual experiments in addition to the implementation of a 5-fold cross-validation for the top-performing approach. Focusing on liver regions achieved the best results with a mean absolute error (MAE) of 0.0837 ± 0.0068, a mean squared error (MSE) of 0.0128 ± 0.0023 and a peak signal-to-noise ratio (PSNR) of 18.99 ± 0.81 dB. Our findings serve as a proof-of-concept, demonstrating the feasibility of generating contrast-enhanced MRI images. However, the current state necessitates further enhancements for effectively addressing the challenge posed by limited dataset sizes with difficult anatomical circumstances as well as MR imaging-related heterogenous tissue contrasts.

This work presents a novel approach for detecting underlying pulsatile structures with laparoscopic tools by using proximal vibroacoustic sensing. This new method could detect pulsations at a depth of up to 1.5 cm. The outcome of this work indicates the potential of identifying deep-seated pulsatile structures during robot-assisted laparoscopy, where haptic feedback is highly limited, to help surgeons prevent vascular damage that may lead to injuries.

Robotic assistance systems are being used more and more frequently in the operating room, with the goal to support surgeons and to automate parts of a procedure. The laparoscopic cholecystectomy is one of the most common procedures in Germany.We aim to automate the assistant grasp task in this procedure. To achieve this goal, first the grasp points on the gallbladder need to be determined. In this work, we therefore present a statistical shape model fitting to the gallbladder for grasp point determination. Gallbladder and liver point clouds are utilized as inputs. A registration algorithm is used to fit the shape model to the gallbladder mesh. The process is evaluated on three different datasets achieving a successful grasping point identification of 90% for artificially created gallbladders, 100% for our silicon phantom model, and 90% for ex-vivo organs.

Automatic speech recognition (ASR), powered by deep learning techniques, is crucial for enhancing humancomputer interaction. However, its full potential remains unrealized in diverse real-world environments, with challenges such as dialects, accents, and domain-specific jargon, particularly in fields like surgery, persisting. Here, we investigate the potential of large language models (LLMs) as error correction modules for ASR.We leverage Whisper-medium or ASRLibriSpeech for speech recognition, and GPT-3.5 or GPT-4 for error correction.We employ various prompting methods, from zero-shot to few-shot with leading questions and sample medical terms to correct wrong transcriptions. Results, measured by word error rate (WER), reveal Whisper’s superior transcription accuracy over ASR-LibriSpeech, with a WER of 11.93% compared to 32.09%. GPT-3.5, with the few-shot with medical terms prompting method, further enhances performance, achieving a 64.29% and 37.83% WER-reduction for Whisper and ASR-LibriSpeech, respectively. Additionally, Whisper exhibits faster execution speed. Substituting GPT-3.5 with GPT- 4 further enhances transcription accuracy. Despite some few challenges, our approach demonstrates the potential of leveraging domain-specific knowledge through LLM prompting for accurate transcription, particularly in sophisticated domains like surgery.

This study developed and evaluated a prototype for augmented reality (AR) interaction among multiple users sharing a three-dimensional hologram within a shared physical space. A systems requirements analysis was conducted to determine necessary interactions, followed by an iterative implementation plan with testing at each step. The application for the HoloLens 2 (HL2) headset was developed using Unity for 3D design, C# for programming, and Photon for multiplayer capabilities. The prototype was evaluated in a user study utilizing custom and NASA TLX questionnaires, comparing participants' self-reported task success with recorded data to assess the accuracy of selfassessments. The prototype effectively facilitated multi-user interactions within a shared environment, achieving acceptable synchronization for two users. Most participants successfully completed tasks, with TLX scores indicating low cognitive and mechanical load during AR task execution. This study demonstrates AR's viability for displaying and manipulating complex three-dimensional structures within a collaborative medical context among multiple individuals in the same physical space.

In hospital operating theatres, inadvertent violations of behavioural rules by novices may increase risk. Virtual reality (VR) can provide a training environment to ease the burden for healthcare professionals. Our work develops and tests a VR training program for correct operating room (OR) etiquette. The VR simulator incorporates realistic 3D models as well as corresponding character and object interactions. The simulator provides a comprehensive interactive educational game covering hand hygiene and disinfection, surgical attire and OR behaviour. 13 clinical trainees and experts assessed acceptance and perceived benefit of the simulator. This evaluation shows promise in supporting clinical staff education. Future developments aim to enhance effectiveness, including the integration of hand tracking and a more detailed selection of interactive tasks.

Augmented reality navigation (ARN) enhances localization and operation by overlaying virtual guides onto realworld surgical fields. The study introduces a laser path anchor (LPA), a novel tool for ARN in surgery aimed at mitigating depth perception challenges. The LPA anchors the virtual planned path onto the physical laser beam, ensuring accurate puncture from any perspective. The tool consists of a rack, a primary laser emitter, an indicating laser emitter, and an AR marker with intrinsic and extrinsic properties validated through calibration and testing. Intrinsic validation involved measuring the parallelism between the indicating and primary laser using Cartesian graph screens. Extrinsic validation assessed the alignment and shortest distance between the virtual path and primary lasers. Results showed an angle deviation of 0.08° between the indicating and primary laser and 1.44° ± 0.47° between the virtual path and primary lasers, with a shortest distance of 7.60 mm ± 2.50 mm. The usability test demonstrated the LPA’s effectiveness in guiding needle insertions without altering the perspective or distracting the surgeon. Despite some limitations, the LPA enhances the user’s perception of linear objects and eliminates the need for perspective adjustments during puncture, warranting further development.

Prostate cancer is a commonly diagnosed cancerous disease among men world-wide. Even with modern technology such as multi-parametric magnetic resonance tomography and guided biopsies, the process for diagnosing prostate cancer remains time consuming and requires highly trained professionals. In this paper, different convolutional neural networks (CNN) are evaluated on their abilities to reliably classify whether an MRI sequence contains malignant lesions. Implementations of a ResNet, a ConvNet and a ConvNeXt for 3D image data are trained and evaluated. The models are trained using different data augmentation techniques, learning rates, and optimizers. The data is taken from a private dataset, provided by Cantonal Hospital Aarau. The best result was achieved with a ResNet3D, yielding an average precision score of 0.4583 and AUC ROC score of 0.6214.

A deformation of the hard palate can occur in spinal muscular atrophy and leads to problems with feeding and swallowing in early childhood. An objective analysis of the palatal changes is therefore desirable for early treatment initiation. In this study, we investigate a deep learning approach to automatically detect deformation in endoscopic images which were collected in a prospective in-vivo study on 33 infants. Ratings of five different experts were used to quantify the deformation and to train our network. We investigate different network architectures and data set splits and achieve classification performances of up to 0.85 ± 0.05 when distinguishing between normal and deformation using the Efficient- Net architecture. This combination of endoscopic imaging and deep learning offers a first approach for the objective assessment of palatal changes.

We present a novel tool designed for the semiautomatic detection and assessment of prostate cancer during recurrence diagnostics using PET-CT with prostate-specific membrane antigen (PSMA). The tool automatically generates a structured report, which is enhanced by 3D images and ensures the reproducibility of the findings. Moreover, the area of the findings is automatically determined and can be corrected manually. Image data from 50 patients following radical prostatectomy was used. Our approach and its user interface were qualitatively evaluated by a nuclear medicine specialist. The segmentation analysis revealed a requirement for further automation. The resulting report was deemed highly beneficial, particularly in terms of the resulting reproducibility and the presentation of 3D images depicting the findings. The presented tool has great potential for much-needed support in everyday clinical practice, as it facilitates the communication of the diagnostic results to colleagues and patients and thus prevents misunderstandings.

We present a framework with two options for selecting a subgroup of training cases for intracranial aneurysm (IA) treatment. Option one, general training, describes training cases that represent a variety of different IAs that represent the diversity of real world cases. This can be achieved via instance selection (IS), which reduces size of a dataset by eliminating redundancies via outlier removal and clustering. Option two, a specific training scenario, describes training that is specialized based on IA features of a specific case, for which we present the novel reverse instance selection (RIS), which introduces similarity to the specific case to the IS methodology. We evaluated our IS and RIS by comparing them to subsets selected based on similarity (SIM) and random sampling (RndS). RIS outperformed SIM and RndS in 79% of experiments. IS outperformed RndS in 33% of experiments. In both scenarios, we observed that our approaches, which balance the weaknesses of SIM and RndS, perform best for small subset sizes close to the database cluster size. Our IS and RIS are flexible in regards to the underlying machine learning and weighting of metrics for evaluation, thus providing a way to select a representative and diverse subset not just for IAs, but also for different kinds of data.

The acquisition of two-dimensional X-ray images (of non-human objects) is quite expensive, primarily due to the costly flat-panel or line detectors. Reducing these costs could open up new possibilities in healthcare applications, such as inspecting limb prostheses for cracks and defects or safety-related material testing of helmets and child seats. In this contribution, two different setups of low-cost mono-pixel X-ray scanners are described: a small scanner with a working volume of 20 x 20 x 20 cm 3 and a large system with a volume of 150 x 230 x 110 cm 3 . Using these novel mono-pixel X-ray scanning systems, various objects - ranging from an X-ray phantom and a human skull phantom to a high-end limb prosthesis with integrated electronics - were scanned and investigated. Compared to conventional X-ray systems, the mono-pixel systems can depict these objects without the distortion problems of cone beam detectors. Although the scanning procedure is time-consuming, the results are quite promising. Mono-pixel X-ray-scanning is a novel modality capable of scanning largescale objects with low-dose radiation. While it is currently not suitable for human applications due to the long scanning times, it can still be used for the non-destructive testing of large-scale medical devices, such as e.g. arm and leg prosthesis.

Lower back pain (LBP) is a common symptom that can be linked to the degeneration of intervertebral discs or injuries to vertebrae. Accurate segmentation of lumbar spinal structures is vital for image-based surgery planning but can be resource intensive. Recent studies have shown that U-Net models can be used reliably to segment vertebrae and intervertebral discs from medical imaging data. This study investigated the potential of combining multiple planar MRI images to segment these structures automatically. The study used MRI data from 83 patients with back pain of the lower spine, with ground truth segmentations performed by a spine surgery specialist using Mimics software. The segmentation models were trained using a specialized patient batching method and a FiLMed U-Net model, which applies a Feature-wise Linear Modulation layer to improve mask generation. The models were trained using a 10-fold cross validation study, and the dice loss was used as the objective function. The study found that both patient batching and FiLM layers improve results by around 5% compared to a baseline U-Net. The average IoU scores of the patient batching method were 0.896 for vertebrae and 0.876 intervertebral disc segmentation. The FiLMed U-Net model achieved average IoU scores of 0.844 and 0.814 and against a baseline of 0.837 and 0.774, respectively. A web-based tool was developed to enable medical personnel to assess segmentation quality visually. Overall, the study indicates that U-Nets show better segmentation potential when using imaging data with complete 3D information of one patient. Future studies should focus on increasing the dataset size and exploring other 3D model architectures to improve segmentation performance.

The interaction between the surgical instrument and tissue is a yet little-exploited source of information. Vibroacoustic signals resulting from an interaction can be analyzed to extract relevant information related to the tissue. This work presents a summary of feasibility studies with synthetic-, animal-, and ex-vivo tissue specimen. A standard palpation probe is equipped with a vibration measurement system. A manual and robot-assisted setup is used for data acquisition. Simple features derived from the signals are used in a subsequent classification step using kNN and SVM. The obtained results show the capability to differentiate various tissues, such as subchondral bone with an accuracy of 98.44%. This demonstrates the potential of the approach to provide tissue related information based on instrument interactions.

Minimally Invasive Robotic Surgery (MIRS) has emerged as a transformative approach in surgical practice, offering reduced patient trauma and enhanced precision. However, challenges persist, including the loss of tactile feedback for surgeons. This study explores the application of machine learning algorithms, specifically variational autoencoders, in vibro-acoustic (VA) signal analysis to address this issue. Our comparative analysis evaluates the potential of supervised learning in surgical data analysis, contributing to advancements in surgical technology. Despite achieving an accuracy of 81%, our results indicate opportunities for further refinement, considering the superior accuracies reported in previous studies. This research underscores the importance of innovative approaches in medical data analysis for optimizing patient care in minimally invasive surgery.

The increased clinical relevance of image-guided procedures, particularly of interventional magnetic resonance imaging (iMRI), highlights the need for advanced devices and robotics to optimize those procedures. To enable a realistic integration of robotics into the clinical workflow, robotic-patient interfaces (RPI) are required to ensure both functionality and usability. However, current concepts have issues with patient accessibility and safety, user handling, or performing interventions on versatile body regions. This work presents a novel silicone-based RPI to flexibly mount the ’Micropositioning Robotics for Image-Guided Surgery’ (μRIGS) system on differently shaped body regions through vacuum, regulated with an electronic miniature pump. Maximum holding forces of 60N at −0.1 bar and 66N at −0.2 bar relative vacuum pressure were reached depending on the applied human body area and tensile force angles. MRI with the integrated metasurface indicated up to 200% signal enhancement, enabling improved tissue contrast within the first 20mm in depth. The multifunctional design supported the incorporation of the sterile iMRI workflow concept. This RPI enables a realistic integration of future technologies such as miniature robots, tracking markers, and instrument holders into complex interventional workflows. Further long-term studies are needed to evaluate the effects of vacuum application lasting for 1-2 hours on different skin types.

The operating room demands well-functioning workflows and optimized communication. Based on the formalization of specific surgical interventions via BPMN, intraoperative checklists can automatically be generated to support the surgical team by the provision of relevant clinical information and situation-dependent assistive functions. While focusing on the surgeon, other actors have not yet been considered. The aim of this work is therefore the extension of the intraoperative checklist to enable role-specific support and improved communication between the actors. The BPMN approach is adapted to depict role-specific surgical steps and clinical information to realize the collaborative, multi-role usage of the checklist. The proof of concept demonstrates the automatic generation of intraoperative checklists which provide relevant information depending on the role. An initial evaluation of the checklist based on the cochlea implantation use case shows promising results.