Volume 6, Issue 1 - Proceedings of the CURAC 2020 Annual Meeting, Hamburg, September 17-19, 2020

Tracking and localizing objects is a central problem in computer-assisted surgery. Optical coherence tomography (OCT) can be employed as an optical tracking system, due to its high spatial and temporal resolution. Recently, 3D convolutional neural networks (CNNs) have shown promising performance for pose estimation of a marker object using single volumetric OCT images. While this approach relied on spatial information only, OCT allows for a temporal stream of OCT image volumes capturing the motion of an object at high volumes rates. In this work, we systematically extend 3D CNNs to 4D spatio-temporal CNNs to evaluate the impact of additional temporal information for marker object tracking. Across various architectures, our results demonstrate that using a stream of OCT volumes and employing 4D spatio-temporal convolutions leads to a 30% lower mean absolute error compared to single volume processing with 3D CNNs.

Surgical tool presence detection in laparoscopic videos is a challenging problem that plays a critical role in developing context-aware systems in operating rooms (ORs). In this work, we propose a deep learning-based approach for detecting surgical tools in laparoscopic images using a convolutional neural network (CNN) in combination with two long short-term memory (LSTM) models. A pre-trained CNN model was trained to learn visual features from images. Then, LSTM was employed to include temporal information through a video clip of neighbour frames. Finally, the second LSTM was utilized to model temporal dependencies across the whole surgical video. Experimental evaluation has been conducted with the Cholec80 dataset to validate our approach. Results show that the most notable improvement is achieved after employing the two-stage LSTM model, and the proposed approach achieved better or similar performance compared with state-of-the-art methods.

Ultrasound compounding techniques offer the possibility to enlarge the otherwise limited field of view of ultrasound. However, existing works mainly rely on larger ultrasound sensors. In this work, we attach electromagnetic (EM) tracking sensors to small tubular echo probes, namely an intracardiac echocardiographic (ICE) probe and a transesophageal echocardiographic (TEE) transducer. The EM tracking allows, when synchronized to the ultrasound, localization of the probes in either 5 DOF (Degrees of Freedom) or 6 DOF without line-of-sight requirement. For computation of the references between the two systems, we developed a novel customized 3D-printable phantom, which is especially convenient for tubular probes that acquire images laterally. Calibration with the phantom and 3D volume reconstruction was conducted in the Plus Toolkit. The volume reconstructor uses the captured position and orientation information to fuse 2D ultrasound slices into a compounded volume. Mean calibration error is below 2.5 mm for ICE and TEE. An accuracy evaluation of the 3D reconstruction using an object of known geometry revealed that tracking with 5 DOF provides unsatisfactory results, while the combination of 6 DOF and TEE achieved a mean absolute difference of 3.08 mm. Our calibration phantom fCal-Echo1.0 is openly available at http://perk-software.cs.queensu.ca/plus/doc/nightly/modelcatalog/.

In endoscopy, depth estimation is a task that potentially helps in quantifying visual information for better scene understanding. A plethora of depth estimation algorithms have been proposed in the computer vision community. The endoscopic domain however, differs from the typical depth estimation scenario due to differences in the setup and nature of the scene. Furthermore, it is unfeasible to obtain ground truth depth information owing to an unsuitable detection range of off-the-shelf depth sensors and difficulties in setting up a depth-sensor in a surgical environment. In this paper, an existing self-supervised approach, called Monodepth [1], from the field of autonomous driving is applied to a novel dataset of stereo-endoscopic images from reconstructive mitral valve surgery. While it is already known that endoscopic scenes are more challenging than outdoor driving scenes, the paper performs experiments to quantify the comparison, and describe the domain gap and challenges involved in the transfer of these methods.

Checklists are a valuable tool to ensure process quality and quality of care. To ensure proper integration in clinical processes, it would be desirable to generate checklists directly from formal process descriptions. Those checklists could also be used for user interaction in context-aware surgical assist systems. We built a tool to automatically convert Business Process Model and Notation (BPMN) process models to checklists displayed as HTML websites. Gateways representing decisions are mapped to checklist items that trigger dynamic content loading based on the placed checkmark. The usability of the resulting system was positively evaluated regarding comprehensibility and end-user friendliness.

In this study, we propose a method for marker detection in X-ray fluoroscopy sequences based on adaptive thresholding and classification. Adaptive thresholding yields multiple marker candidates. To remove non-marker areas, 24 specific features are extracted from each extracted patch and four supervised classifiers are trained to differentiate non-marker areas from marker areas. Quantitative evaluation was carried out to assess different classifier performance by calculating accuracy, sensitivity, specificity and precision. SVM outperforms other classifiers based on the mean value for accuracy, specificity and precision with 81.56, 91.94 and 84.21%, respectively.

The treatment of cerebro- and cardiovascular diseases requires complex and challenging navigation of a catheter. Previous attempts to automate catheter navigation lack the ability to be generalizable. Methods of Deep Reinforcement Learning show promising results and may be the key to automate catheter navigation through the tortuous vascular tree. This work investigates Deep Reinforcement Learning for guidewire manipulation in a complex and rigid vascular model in 2D. The neural network trained by Deep Deterministic Policy Gradients with Hindsight Experience Replay performs well on the low-level control task, however the high-level control of the path planning must be improved further.

Cardiac diseases manifest in a multitude of interconnected changes in morphology and dynamics. Radiomics approaches are a promising technique to analyze such changes directly from image data. We propose novel features to specifically describe moving cardiac structures, and an interactive 4D visualization method to explore such data. Prototypical tests with an open data set containing different diseases show that our approach can be a fast and useful tool for the 4D analysis of heterogeneous cohort data.

Risk classes defined by MDR and FDA for state-of-the-art surgical robots based on their intended use are not suitable as indicators for their hazard potential. While there is a lack of safety regulation for an increasing degree of automation as well as the degree of invasiveness into the patient’s body, adverse events have increased in the last decade. Thus, an outright identification of hazards as part of the risk analysis over the complete development process and life cycle of a surgical robot is crucial, especially when introducing new technologies. For this reason, we present a comprehensive approach for hazard identification in early phases of development. With this multi-perspective approach, the number of hazards identified can be increased. Furthermore, a generic catalogue of hazards for surgical robots has been established by categorising the results. The catalogue serves as a data pool for risk analyses and holds the potential to reduce hazards through safety measures already in the design process before becoming risks for the patient.

Previous research reported catheter pose-dependent virtual angioscopy images for endovascular aortic repair (EVAR) (phantom studies) without any validation with video images. The goal of our study focused on conducting this validation using a video graphics array (VGA) camera. The spatial relationship between the coordinate system of the virtual camera and the VGA camera was computed with a Hand-Eye calibration so that both cameras produced similar images. A re-projection error of 3.18 pixels for the virtual camera and 2.14 pixels for the VGA camera was obtained with a designed three-dimensional (3D) printed chessboard. Similar images of the vessel (3D printed aorta) were acquired with both cameras except for the different depth. Virtual angioscopy images provide information from inside the vessel that may facilitate the understanding of the tip position of the endovascular tools while performing EVAR.

Dysphagia, the difficulty in swallowing, is one of the most common and, at the same time, most heterogeneous symptom of the upper digestive tract. Due to its lifetime prevalence of about 5%, every 19th person is affected on average, especially with increasing age. Dysphagia occurs in both benign and malignant diseases of the esophagus and the oropharyngeal tract as well as in neuromuscular diseases. Even dysphagia caused by benign diseases can lead to significantly reduced quality of life. The diagnostics of the actual underlying disease in patients with dysphagia is commonly conducted using a combination of endoscopy, esophageal manometry, functional assessments and radiologic means, e.g. X-ray-fluoroscopy. As these examinations are typically performed in sequential order, it remains to the physicians to combine the relevant information from each modality to form a conclusion. We argue that this is neither an intuitive, nor a standardized form of presenting the findings to the physician. To address this, we propose a novel approach for fusing time-synchronized manometric and X-ray data into a single view to provide a more comprehensive visualization method as a novel means for diagnosing dysphagia.

Injuries to the biliary tree during surgical, endoscopic or invasive radiological diagnostic or therapeutic procedures involving the pancreas, liver or organs of the upper gastrointestinal tract give rise to the need to develop a method for clear discrimination of biliary anatomy from surrounding tissue. Hyperspectral imaging (HSI) is an emerging optical technique in disease diagnosis and image-guided surgery with inherent advantages of being a non-contact, non-invasive, and non-ionizing technique. HSI can produce quantitative diagnostic information about tissue pathology, morphology, and chemical composition. HSI was applied in human liver transplantation and compared to porcine model operations to assess the capability of discriminating biliary anatomy from surrounding biological tissue. Absorbance spectra measured from bile ducts, gall bladder, and liver show a dependence on tissue composition and bile concentration, with agreement between human and porcine datasets. The bile pigment biliverdin and structural proteins collagen and elastin were identified as contributors to the bile duct and gall bladder absorbance spectra.

Tracking of surgical instruments is an essential step towards the modernization of the surgical workflow by a comprehensive surgical landscape guidance system (COMPASS). Real-time tracking of a laparoscopic camera used in minimally-invasive surgery is required for applications in surgical workflow documentation, machine learning, image-localization, and intra-operative visualization. In our approach, an inertial measurement unit (IMU) assists the tool tracking in situations when no line-of-sight is available for infrared (IR) based tracking of the laparoscopic camera. The novelty of this approach lies in the localization method adjusted for the laparoscopic visceral surgery, particularly when the line-of-sight is lost. It is based on IMU tracking and the positioning of the trocar entry point. The trocar entry point is the remote center of motion (RCM), reducing degrees of freedom. We developed a method to tackle localization and a real-time tool for position and orientation estimation. The main error sources are given and evaluated in a test scenario. It reveals that for small changes in penetration length (e.g., pivoting), the IMU’s accuracy determines the error.

Bilateral control of teleoperated robots still poses a challenge, especially if environment properties vary over a large degree. Most currently available systems do not provide force feedback and consequently surgeons still have to estimate contact forces predominantly visually. During drilling or milling in bone surgery, visual estimation is virtually impossible due to hardly any deformations. However, the force progression contains important complimentary information for the surgeon. Therefore, a concept for a force-reflecting controller for drilling or milling during teleoperated bone surgery was developed and tested on a one degree of freedom (DOF) test setup. First, the desired behavior and control architectures were derived based on the context of bone surgery. The resulting controller combines three control architectures in a switching controller, depending on the tool actuation and environment properties. Experimental results with a 1-DOF test setup showed the desired control and switching behavior, while remaining stable. Therefore, the developed control concept seems promising for teleoperated bone surgery.

Deep learning methods produce promising results when applied to a wide range of medical imaging tasks, including segmentation of artery lumen in computed tomography angiography (CTA) data. However, to perform sufficiently, neural networks have to be trained on large amounts of high quality annotated data. In the realm of medical imaging, annotations are not only quite scarce but also often not entirely reliable. To tackle both challenges, we developed a two-step approach for generating realistic synthetic CTA data for the purpose of data augmentation. In the first step moderately realistic images are generated in a purely numerical fashion. In the second step these images are improved by applying neural domain adaptation. We evaluated the impact of synthetic data on lumen segmentation via convolutional neural networks (CNNs) by comparing resulting performances. Improvements of up to 5% in terms of Dice coefficient and 20% for Hausdorff distance represent a proof of concept that the proposed augmentation procedure can be used to enhance deep learning-based segmentation for artery lumen in CTA images.

Semantic segmentation of organs and tissue types is an important sub-problem in image based scene understanding for laparoscopic surgery and is a prerequisite for context-aware assistance and cognitive robotics. Deep Learning (DL) approaches are prominently applied to segmentation and tracking of laparoscopic instruments. This work compares different combinations of neural networks, loss functions, and training strategies in their application to semantic segmentation of different organs and tissue types in human laparoscopic images in order to investigate their applicability as components in cognitive systems. TernausNet-11 trained on Soft-Jaccard loss with a pretrained, trainable encoder performs best in regard to segmentation quality (78.31% mean Intersection over Union [IoU]) and inference time (28.07 ms) on a single GTX 1070 GPU.

Minimally invasive surgery is increasingly utilized for mitral valve repair and replacement. The intervention is performed with an endoscopic field of view on the arrested heart. Extracting the necessary information from the live endoscopic video stream is challenging due to the moving camera position, the high variability of defects, and occlusion of structures by instruments. During such minimally invasive interventions there is no time to segment regions of interest manually. We propose a real-time-capable deep-learning-based approach to detect and segment the relevant anatomical structures and instruments. For the universal deployment of the proposed solution, we evaluate them on pixel accuracy as well as distance measurements of the detected contours. The U-Net, Google’s DeepLab v3, and the Obelisk-Net models are cross-validated, with DeepLab showing superior results in pixel accuracy and distance measurements.

Photodynamic diagnostic (PDD) is an optical enhancement option for the endoscope to support the detection of cancer, for example in the bladder. In real application PDD efficiency suffers due to the complex accumulation of the photosensitizing drug inside the tumor and the associated processes of heme syntheses to create the fluorescent components needed. To optimize the diagnostic outcome of PDD it would be helpful to predict the optimal time for diagnosis based on measurable precursors. In a previous cell study, we proposed a new filter fluorometer to image the accumulation of the precursors Coproporphyrin III (CP-III) and Uroporphyrin III (UP-III) that metabolize to Protoporphyrin IX (PP-IX) later. This accumulation process can be used to predict the optimal time slot for diagnostic imaging. Therefore, a new filter system was designed to distinguish between CP-III and PP-IX. In this work we tested this filter system in combination with a standard PDD endoscopic imaging system. Goal of this study was to prove the technical feasibility in a non-patient setup to prepare a later clinical study.

Telemedicine has become a valuable asset in emergency responses for assisting paramedics in decision making and first contact treatment. Paramedics in unfamiliar environments or time-critical situations often encounter complications for which they require external advice. Modern ambulance vehicles are equipped with microphones, cameras, and vital sensors, which allow experts to remotely join the local team. However, the visual channels are rarely used since the statically installed cameras only allow broad views at the patient. They neither allow a close-up view nor a dynamic viewpoint controlled by the remote expert. In this paper, we present EyeRobot , a concept which enables dynamic viewpoints for telepresence using the intuitive control of the user’s head motion. In particular, EyeRobot utilizes the 6 degrees of freedom pose estimation capabilities of modern head-mounted displays and applies them in real-time to the pose of a robot arm. A stereo-camera, installed on the end-effector of the robot arm, serves as the eyes of the remote expert at the local site. We put forward an implementation of EyeRobot and present the results of our pilot study which indicates its intuitive control.

Fluoroscopy-guided percutaneous biopsy interventions are mostly performed with traditional free-hand technique. The practical experience of the surgeon influences the duration of the intervention and the radiation exposure for patients and him-/herself. Especially when the placement of heavy and long instruments in double oblique angles is required, manual techniques reach their technical limitations very fast. The system presented herein automatizes the needle positioning using only two 2D scans while the robotic platform guides the intervention. These two images were used to plan the needle pathway and to estimate the pose of the robot using a custom-made end-effector with embedded registration fiducials. The estimated pose was subsequently used to transfer the planed needle path to the robot’s coordinate system and finally to compute the movement parameters in order to align the robot with this plan. To evaluate the system, two phantoms with 11 different targets on it were developed. The targets were punctured, and the application accuracy was measured quantitatively. The solution achieved sub-millimetric accuracy for needle placement (min. 0.23, max. 1.04 in mm). Our approach combines the advantages of fluoroscopic imaging and ensures automatic needle alignment with highly reduced X-ray radiation. The proposed system shows promising potential to be a guidance platform that is easy to combine with available fluoroscopic imaging systems and provides valuable help to the physician in more difficult interventions.

Direct microlaryngoscopy is a frequently performed procedure in otorhinolaryngology, whereby considerable force is exerted on the patients’ teeth, as well as oropharyngeal and laryngeal structures. Especially in cases of a challenging exposure of the anterior commissure, the cervical spine needs to be brought into a hyperextended position, which is not possible in every patient. Therefore, the conflict between the straight rigid microlaryngoscope and the curvature of the oropharyngeal corridor frequently results in tissue trauma with the consequence of teeth fracture, hematoma or nerve injury. We have developed the s-MAC system, a hyper-angulated video laryngoscope with flexible instruments for transoral laryngeal surgery, which so far shows high feasibility in preclinical studies. Due to its curved shape it may exert less force on teeth and supraglottis as conventional direct microlaryngoscopy. We quantified the effects of these two surgical systems using flexible pressure sensors in two different scenarios: a mobile, hyperextended and an immobile cervical spine of the dummy, to simulate a challenging airway. We could demonstrate a 21% (40% with immobilized cervical spine) reduction of the average peak force acting on the maxillary incisors and a 55% (65% with immobilized cervical spine) reduction of the average intraoperative force on the supraglottis. The developed prototype applied therefore significantly less force on upper front teeth and supraglottis as compared to conventional direct microlaryngoscopy – especially in the case of an immobilized cervical spine.

Minimally invasive robotic surgery offer benefits such as reduced physical trauma, faster recovery and lesser pain for the patient. For these procedures, visual and haptic feedback to the surgeon is crucial when operating surgical tools without line-of-sight with a robot. External force sensors are biased by friction at the tool shaft and thereby cannot estimate forces between tool tip and tissue. As an alternative, vision-based force estimation was proposed. Here, interaction forces are directly learned from deformation observed by an external imaging system. Recently, an approach based on optical coherence tomography and deep learning has shown promising results. However, most experiments are performed on ex-vivo tissue. In this work, we demonstrate that models trained on dead tissue do not perform well in in vivo data. We performed multiple experiments on a human tumor xenograft mouse model, both on in vivo , perfused tissue and dead tissue. We compared two deep learning models in different training scenarios. Training on perfused, in vivo data improved model performance by 24% for in vivo force estimation.

During thermal ablation in a target tissue the information about temperature is crucial for decision making of successful therapy. An observable temporal and spatial temperature propagation would give a visual feedback of irreversible cell damage of the target tissue. Potential temperature features in ultrasound (US) B-Mode image sequences during radiofrequency (RF) ablation in ex-vivo porcine liver were found and analysed. These features could help to detect the transition between reversible and irreversible damage of the ablated target tissue. Experimental RF ablations of ex-vivo porcine liver were imaged with US B-Mode imaging and image sequences were recorded. Temperature was simultaneously measured within the liver tissue around a bipolar RF needle electrode. In the B-Mode images, regions of interest (ROIs) around the centre of the measurement spots were analysed in post-processing using average gray-level (AVGL) compared against temperature. The pole of maximum energy level in the time-frequency domain of the AVGL changes was investigated in relation to the measured temperatures. Frequency shifts of the pole were observed which could be related to transitions between the states of tissue damage.

Robot-assisted minimally-invasive surgery is increasingly used in clinical practice. Force feedback offers potential to develop haptic feedback for surgery systems. Forces can be estimated in a vision-based way by capturing deformation observed in 2D-image sequences with deep learning models. Variations in tissue appearance and mechanical properties likely influence force estimation methods’ generalization. In this work, we study the generalization capabilities of different spatial and spatio-temporal deep learning methods across different tissue samples. We acquire several data-sets using a clinical laparoscope and use both purely spatial and also spatio-temporal deep learning models. The results of this work show that generalization across different tissues is challenging. Nevertheless, we demonstrate that using spatio-temporal data instead of individual frames is valuable for force estimation. In particular, processing spatial and temporal data separately by a combination of a ResNet and GRU architecture shows promising results with a mean absolute error of 15.450 compared to 19.744 mN of a purely spatial CNN.

A common treatment for malignant hepatic tumors is radiofrequency thermal ablation (RFA); however this procedure is often exhausting and error-prone, when performed only with hand held devices. In this work, we present a new concept for a computer- and mechatronic-arm-assisted treatment of hepatic tumor with RFA. Challenging factors of state-of-the-art RFA procedures are tackled by utilizing a prototype, which combines state-of-the-art ultrasound image guidance with a mechatronic support arm system. Physicians can image and examine the human abdomen using an ultrasound capturing device which is guided by the mechatronic support arm. The mechatronic arm allows to track the ultrasound transducer via joint position encoders, so that the recorded images’ exact positions are acquired and the patient is automatically registered during imaging. Consequently, physicians can use the acquired ultrasound images to navigate in the imaged abdomen area. The established prototype enables preplanning and semi-automatic alignment of trajectories with sufficient accuracy. In addition, the arm system supports the usually lengthy, exhausting and error-prone holding of the needle during the ablation process. Based on the results of accuracy examination and expert assessments by surgeons, the established concept could be proven to be advantageous for the considered use case.

Laparoscopic surgery consists of many tasks that have to be handled by the surgeon and the operating room personnel. Recognition of situations where action is required enables automatic handling by the integrated OR or notifying the surgical team with a visual reminder. As a byproduct of some surgical actions, electrosurgical smoke needs to be evacuated to keep the vision clear for the surgeon. Building on the success of convolutional neural networks (CNNs) for image classification, we utilize them for image based detection of surgical smoke. As a baseline we provide results for an image classifier trained on the publicly available smoke annotions of the Cholec80 dataset. We extend this evaluation with a self-training approach using teacher and student models. A teacher model is created with the labeled dataset and used to create pseudo labels. Multiple datasets with pseudo labels are then used to improve robustness and accuracy of a noisy student model. The experimental evaluation shows a performance benefit when utilizing increasing amounts of pseudo-labeled data. The state of the art with a classification accuracy of 0.71 can be improved to an accuracy of 0.85. Surgical data science often has to cope with minimal amounts of labeled data. This work proposes a method to utilize unlabeled data from the same domain. The good performance in standard metrics also shows the suitability for clinical use.

Patient tracking is an essential part in a surgical augmented reality system for correct hologram to patient registration. Augmented reality can support a surgeon with visual assistance to navigate more precisely during neurosurgical interventions. In this work, a system for patient tracking based on infrared markers is proposed. These markers are widely used in medical applications and meet the special medical requirements such as sterilizability. A tracking accuracy of 0.76 mm is achieved when using the near field reflectivity and depth sensor of the HoloLens. On the HoloLens a performance of 55–60 fps is reached, which grants a sufficiently stable placement of the holograms in the operating room.

The aim of this work was the development of an isolated heart setup to delineate the interactions between intraventricular flow features, hemodynamic parameters and mechano-energetics after certain mitral valve therapies. Five porcine hearts were explanted and prepared for (i) edge-to-edge mitral valve repair, (ii) implantation of a rotatable biscupid mechanical valve prosthesis. Flow structures were visualized using echocardiography while hemodynamics was recorded in terms of pressures, flow rates and ventricular volume. Hemodynamic and cardiac mechano-energetics implied a marginal effect (<5%) of alternating leaflet orientation on ventricular pre-load and stroke work. After edge-to-edge repair, substantial variations in flow structures were observed. Beside promoting profound insights into fundamental physiologic mechanisms, the setup may be used for validation of computer aided therapy planning tools.

C-arms are medical devices widely used for image-guided minimally invasive endovascular procedures. This technology requires considerable experience for the physicians to position the C-arm to obtain X-ray images of the endovascular tools. In addition, this image-guided therapy is based on two-dimensional images which lack depth information. The purpose of this study was to develop a system that controls the C-arm movements based on the previous position of the tip of a guide wire and the vessel information, and also displays the estimated tip position (specifically, the virtual line that would join the X-ray source and the projected tip in the flat-panel detector) on an augmented reality device (HoloLens). A phantom study was conducted to evaluate the system using intraoperative cone-beam computed tomography scans to obtain the reference tip position. The mean distance between the tip position (ground truth) and the virtual three-dimensional line was 1.18 mm. The proposed system was able to control the C-arm movements based on the position of the tip of the guide wire. The visualization on HoloLens also allowed a more intuitive understanding of the position of the endovascular tool related to the patient’s anatomy during the intervention.

Although robotic radiosurgery offers a flexible arrangement of treatment beams, generating treatment plans is computationally challenging and a time consuming process for the planner. Furthermore, different clinical goals have to be considered during planning and generally different sets of beams correspond to different clinical goals. Typically, candidate beams sampled from a randomized heuristic form the basis for treatment planning. We propose a new approach to generate candidate beams based on deep learning using radiological features as well as the desired constraints. We demonstrate that candidate beams generated for specific clinical goals can improve treatment plan quality. Furthermore, we compare two approaches to include information about constraints in the prediction. Our results show that CNN generated beams can improve treatment plan quality for different clinical goals, increasing coverage from 91.2 to 96.8% for 3,000 candidate beams on average. When including the clinical goal in the training, coverage is improved by 1.1% points.

Forensic autopsies include a thorough examination of the corpse to detect the source or alleged manner of death as well as to estimate the time since death. However, a full autopsy may be not feasible due to limited time, cost or ethical objections by relatives. Hence, we propose an automated minimal invasive needle biopsy system with a robotic arm, which does not require any online calibrations during a procedure. The proposed system can be easily integrated into the workflow of a forensic biopsy since the robot can be flexibly positioned relative to the corpse. With our proposed system, we performed needle insertions into wax phantoms and livers of two corpses and achieved an accuracy of 4.34 ± 1.27 mm and 10.81 ± 4.44 mm respectively.

The basic concept of Diagnostic Laparoscopy (DL) is old but not old-fashioned. – Before the rise of tomography and ultrasound there was just the direct look into the abdomen and onto the affected organ available. As open surgery comes along with trauma, blood loss and infection, every effort have been made to improve the access strategies. Finally, due to innovation in optics, video technology and computer science, the look into the abdomen through a “keyhole” is a standard procedure today. – In this review we give an overview of history, implications and cost-effectiveness of DL, attempting an extrapolation of its future role.

The first choice in diagnostic imaging for patients suffering from peripheral arterial disease (PAD) is 2D ultrasound (US). However, for a proper imaging process, a skilled and experienced sonographer is required. Additionally, it is a highly user-dependent operation. A robotized US system that autonomously scans the peripheral arteries has the potential to overcome these limitations. In this work, we extend a previously proposed system by a hierarchical image analysis pipeline based on convolutional neural networks (CNNs) in order to control the robot. The system was evaluated by checking its feasibility to keep the vessel lumen of a leg phantom within the US image while scanning along the artery. In 100% of the images acquired during the scan process the whole vessel lumen was visible. While defining an insensitivity margin of 2.74 mm, the mean absolute distance between vessel center and the horizontal image center line was 2.47 mm and 3.90 mm for an easy and complex scenario, respectively. In conclusion, this system presents the basis for fully automatized peripheral artery imaging in humans using a radiation-free approach.

Objectives Ultrasound is a widely used imaging technology that allows for fast diagnosis of a broad range of illnesses and injuries of the musculoskeletal system. However, interpreting ultrasound images remains a challenging task that requires expert knowledge and years of training for each exam. One crucial step for the long-term goal of automatic diagnosis is pixel wise semantic segmentation. Methods In this work, several state-of-the-art semantic segmentation networks were trained on a new dataset of manually annotated ultrasound images depicting the distal femur. Results PSP-Net achieved the best overall performance with an average surface distance error (SDE) of 0.64 mm. Conclusions We recommend the PSP-Net architecture for semantic segmentation of bone surfaces.

We present a mesh generation algorithm which is able to produce smooth meshes from point clouds derived from histological slices. In this work, the shrinking tube mesh generation is used on histologic images depicting pathologic vessels. Our mesh generation is modeled after the behaviour of a shrinking tube. A start shape is fitted iteratively to the point cloud. The presented algorithm was successfully used to generate meshes of the inner and outer contour from vessels in histologic images. While histologic slices have a high in-plane resolution, the large slice distance and deformations during tissue deformations are challenging for 3D model generation.

This work aims to demonstrate the feasibility that haptic information can be acquired from a da Vinci robotic tool using audio sensing according to sensor placement requirements in a real clinical scenario. For that, two potential audio sensor locations were studied using an experimental setup for performing, in a repeatable way, interactions of a da Vinci forceps with three different tissues. The obtained audio signals were assessed in terms of their resulting signal-to-noise-ratio (SNR) and their capability to distinguish between different tissues. A spectral energy distribution analysis using Discrete Wavelet Transformation was performed to extract signal signatures from the tested tissues. Results show that a high SNR was obtained in most of the audio recordings acquired from both studied positions. Additionally, evident spectral energy-related patterns could be extracted from the audio signals allowing us to distinguish between different palpated tissues.

Automatic recognition of surgical phases is an important component for developing an intra-operative context-aware system. Prior work in this area focuses on recognizing short-term tool usage patterns within surgical phases. However, the difference between intra- and inter-phase tool usage patterns has not been investigated for automatic phase recognition. We developed a Recurrent Neural Network (RNN), in particular a state-preserving Long Short Term Memory (LSTM) architecture to utilize the long-term evolution of tool usage within complete surgical procedures. For fully automatic tool presence detection from surgical video frames, a Convolutional Neural Network (CNN) based architecture namely ZIBNet is employed. Our proposed approach outperformed EndoNet by 8.1% on overall precision for phase detection tasks and 12.5% on meanAP for tool recognition tasks.

Ultrasound (US) imaging, in contrast to other image guidance techniques, offers the distinct advantage of providing volumetric image data in real-time (4D) without using ionizing radiation. The goal of this study was to perform the first quantitative comparison of three different 4D US systems with fast matrix array probes and real-time data streaming regarding their target tracking accuracy and system latency. Sinusoidal motion of varying amplitudes and frequencies was used to simulate breathing motion with a robotic arm and a static US phantom. US volumes and robot positions were acquired online and stored for retrospective analysis. A template matching approach was used for target localization in the US data. Target motion measured in US was compared to the reference trajectory performed by the robot to determine localization accuracy and system latency. Using the robotic setup, all investigated 4D US systems could detect a moving target with sub-millimeter accuracy. However, especially high system latency increased tracking errors substantially and should be compensated with prediction algorithms for respiratory motion compensation.

Intraoperative brain deformation, so-called brain shift, affects the applicability of preoperative magnetic resonance imaging (MRI) data to assist the procedures of intraoperative ultrasound (iUS) guidance during neurosurgery. This paper proposes a deep learning-based approach for fast and accurate deformable registration of preoperative MRI to iUS images to correct brain shift. Based on the architecture of 3D convolutional neural networks, the proposed deep MRI-iUS registration method has been successfully tested and evaluated on the retrospective evaluation of cerebral tumors (RESECT) dataset. This study showed that our proposed method outperforms other registration methods in previous studies with an average mean squared error (MSE) of 85. Moreover, this method can register three 3D MRI-US pair in less than a second, improving the expected outcomes of brain surgery.

Patient handover is an important part for information transfer between medical professionals in a clinical setting. Yet, in current medical education, these conversations are only trained sparsely, since they are costly to perform as they take place in offsite courses and are led by experts over several days. Virtual reality (VR)-based training courses could increase the availability of training, by eliminating travel costs and reducing the time-commitment of the teaching experts. This work presents a VR prototype of a multi-user training and examination application for patient handover. To ensure a similar interaction quality to its current real world counterpart, this work used omni-directional video recordings to create a realistic setting and compared different projection methods. A pilot study highlighted distinct use-cases of sphere and mesh projections to visualize the recordings. The results suggest enhanced spatial presence relating to the usage of omni-directional videos in VR-applications.

In this paper we present results to the problem of an adequate and compact symbolic representation of morphological features of anatomical structures that serve as surgical landmarks for automated assistance in endoscopic surgery using the General Formal Ontology (GFO) as a formal framework. For this purpose, we employed a translation from this first-order logic representation to a more compact description logic based formalism with the associated benefits, such as the availability of decidable reasoning procedures, for the purpose of automated landmark recognition in a hybrid symbolic/subsymbolic AI approach.

We present a novel approach to simulate deformation in aortic valve replacement scenarios with applications in operation planning and batch domain creation for large computational fluid dynamics studies of the aortic arch.

Dizziness is one of the most common symptoms in medicine. For differentiation of peripheral or central origin of the vertigo, history and clinical examination with detection of a nystagmus is essential. The aim of this study was to detect horizontal vestibular nystagmus utilizing a webcam. In the feasibility study, caloric induced vestibular nystagmus was recorded with conventional video-nystagmography and webcam. Analysis of recorded data was performed with a developed software which used computer vision techniques. A designed algorithm detected nystagmus existence and their direction. The software was evaluated by an expert-rated video-nystagmography. Webcam-based vestibular nystagmus detection is possible. Currently, a clinical application is not approved. Further software improvements are necessary to increase its accuracy.

An auditory brainstem implant (ABI) attempts to restore hearing at patients with bilateral damaged hearing nerve. The optimal placement of the ABI is challenging even with available auditory measures on the brainstem. We present a visual guidance system that aims to assist during intraoperative ABI placement. As a starting point, a surgical probe is navigated and intuitively visualized in the microscope oculars. The system is developed using modular and agile software design techniques. In a usability study, the participants were able to detect invisible targets marked in a phantom image with a millimetric precision. To the best of our knowledge, this is the first time that this kind of visual guidance is presented. In the future, the system will be expanded with surgical instruments used for the ABI placement.

We present a tool for automatic segmentation of wall enhancement of intracranial aneurysms in black blood MRI. The results of the automatic segmentation with several configurations is compared to manual expert segmentations. While the manual segmentation includes some voxels of lower intensity not present in the automatic segmentation, overall the volume of the automatic segmentation is higher.