Skip to main content
Article Open Access

Surgical navigation with QR codes

Marker detection and pose estimation of QR code markers for surgical navigation
  • EMAIL logo , and
Published/Copyright: September 30, 2016

Abstract

The presented work is an alternative to established measurement systems in surgical navigation. The system is based on camera based tracking of QR code markers. The application uses a single video camera, integrated in a surgical lamp, that captures the QR markers attached to surgical instruments and to the patient.

1 Introduction

Surgical navigation systems are used to visualize the position of medical instruments relative to the patient on three dimensional computer tomography (CT) or magnetic resonance tomography (MRT) patient data during surgery. With the help of this assistance, the surgeon can navigate accurately to the operating field and perform the operation without harming sensitive structures around that region. For localizing the patient and the instruments two established measurement principles exist: optical tracking [1] and electromagnetic tracking [2]. Optical tracking systems use an external stereoscopic camera that measures the position of reflector spheres arrayed on the instruments and the patient. That way the relative transformation between the patient and the instruments is determined. The main disadvantages are the required space in the operating room and the restriction of the medical personnel in accessing the patient, as the camera’s line of sight must not be blocked. Electromagnetic tracking systems use a magnetic field to measure the position of a sensing coil on the instrument or the patient. The key disadvantages are the higher inaccuracy compared to optical tracking and its susceptibility to electrically conductive and ferromagnetic materials.

The developed system based on QR codes is a novel approach for the tracking procedure for surgical navigation systems. The proposed system uses a video camera and flat markers for localization [3].

2 System description

A single high resolution camera that is integrated in a surgical light is used to measure the position and orientation of QR codes attached to the patient and surgical instruments to track. The system is highly integrated, the line of sight problem is minimized and the QR code markers are intuitively in the field of view of the camera. A special feature is that QR codes of different sizes can be flexibly used for various instruments since the QR code size, that is required to determine the position, and transformation from marker to tip can be stored in the QR code. A further benefit of using a QR code as a marker for pose estimation, is that the QR code structure allows to detect many feature points that can be used for precise detection. Figure 1 shows a system setup with a patient marker and an instrument marker.

Figure 1 System description of the surgical navigation measurement system based on QR codes. A high resolution camera integrated in the surgical light acquires images of QR codes attached to a pointer instrument and the patient. The transformation between the QR codes and the camera is computed by detecting the marker position and orientation in the images and reading the QR size that is encoded in the QR codes.
Figure 1

System description of the surgical navigation measurement system based on QR codes. A high resolution camera integrated in the surgical light acquires images of QR codes attached to a pointer instrument and the patient. The transformation between the QR codes and the camera is computed by detecting the marker position and orientation in the images and reading the QR size that is encoded in the QR codes.

2.1 Image processing algorithms

Surgical navigation requires a highly accurate estimation of the patient and instrument pose. In this work, this approach is done by detecting geometrical features and computing the homography transformation between the QR code and the camera.

The work flow of image processing is shown in Figure 2. First of all, the region of interest is detected for every QR code in the acquired images. Then each detected QR code is decoded, i.e. the information that is saved in the QR code is extracted. As input for the pose estimation, the code structure is detected with high precision, i.e. corresponding feature points are computed. Finally the position and orientation is estimated.

Figure 2 Work flow of the image processing algorithms.
Figure 2

Work flow of the image processing algorithms.

2.1.1 Region of interest (ROI) detection

QR codes are characterized by three finder patterns (FIPs) [4]. The FIPs are located in three corners of the QR code. Each finder pattern consists of three contours that can be detected, i.e. i, j and k (see Figure 3). To achieve a robust recognition of the FIPs of each QR code, three geometrical features are considered: hierarchy, concentricity and perimeter proportionality.

Figure 3 Finder pattern (FIP) structure.
Figure 3

Finder pattern (FIP) structure.

  • Hierarchy: Contour i encloses contour j and k

  • Concentricity: The three contours have the same or nearly the same center of mass

  • Perimeter proportionality: The contours have the following specific length ratio:

(1)Perimeteri=75Perimeterj
(2)Perimeterj=53Perimeterk

when the above conditions are fulfilled, a FIP is unequivocally detected.

To establish the desired ROIs it is necessary to group the detected FIPs. This means, that in a frame with multiple markers, the detected FIPs have to be assigned to the corresponding QR code in order to create a ROI of each QR code. The assignment can be considered as a classification problem. To perform the classification, the following features are considered for each FIP: perimeter, area and distance to the origin. The product of these features is a very robust measure to classify the FIPs. The features are sorted and divided into the number of found QR codes. If there are for instance three QR codes, there will be nine FIPs and the feature vector has nine elements. In the sorted vector the first three elements correspond to the first QR code, the 4th, 5th and 6th element to the 2nd QR code and so on.

The ROI creation process is a geometric calculation based on all of the points obtained in the FIPs detection step. Figure 4 shows the involved points and how the ROI is created based on the geometry. The goal is to obtain the four points P1, P2, P3 and Q. The points P1, P2, P3 are obtained by the FIPs and the point Q is the intersection of two lines defined by the two outer finder patterns.

Figure 4 ROI creation based on the detected finder patterns. The points P1, P2, P3 are computed from the detected finder patterns, the point Q is defined by the intersection of two lines defined by the two outer finder patterns.
Figure 4

ROI creation based on the detected finder patterns. The points P1, P2, P3 are computed from the detected finder patterns, the point Q is defined by the intersection of two lines defined by the two outer finder patterns.

2.1.2 Marker detection and pose estimation

For pose estimation the size of the markers that is encoded in the QR codes, is required. Since each QR code holds its size information, various QR code sizes can be used for different applications without giving the system this preknowledge in advance. The pose describes the transformation between the marker plane coordinate system and the camera coordinate system as a rotation matrix R and translation vector t. Two methods are proposed for pose estimation:

  • A method based on points correspondence, it is a conventional method but refined in accuracy

  • A method based on the optimization of the homography through template matching

2.1.3 Pose estimation based on point correspondences

The used QR code consists of 21 × 21 elements. Each single element is detected as a pattern with four corners by line fitting of all edges that lie in a row (see Figure 5). By computing the intersection of all lines, 484 points are detected and can be used for pose estimation. The input image is filtered with a Sobel edge detector in x and y direction. For line fitting, all the edges that lie in a row are used to fit one line. For this purpose a mask is created for each of this edge points. The mask is iteratively applied and the lines are fitted to all of the edge contours. Finally the intersections of all detected lines in both directions are computed.

Figure 5 Detection of point in the QR code by line fitting and computing the line intersections.
Figure 5

Detection of point in the QR code by line fitting and computing the line intersections.

Pose estimation based on optimization of the homography

This is a novel method aimed to obtain the pose of a QR code marker in space with high accuracy. The method performs an iterative template matching between a created QR code template and the QR code in the acquired image until an error function is minimized [5]. This error function depends on the position and orientation of the marker, and its numerical optimization is based on a non-linear unconstrained problem [6]. The algorithm is described in Figure 6.

Figure 6 Algorithm to compute the homography by performing an iterative template matching between a created QR code template and the QR code in the acquired image.
Figure 6

Algorithm to compute the homography by performing an iterative template matching between a created QR code template and the QR code in the acquired image.

The method starts by creating a template of the QR code. With the already computed ROI of the marker in the acquired image, a subimage f of the marker is created. Then an initial homography matrix H0 can be computed with four edge points (xi, yi) of the template and the corresponding edge points (ui, vi) of the ROI by setting up a system of equations based on the following formula, with a scalar si [7]:

(3)si(uivi1)=H(xiyi1)

After that, a new image g is created by transforming the QR code template to the ROI with the current homography Hi. The image f and g are compared to compute their difference and consequently evaluating it as an error value. Finally, depending on the error, the homography is iteratively optimized until the images are as similar as possible, i.e. the error function is optimized. With this optimized homography matrix, it is possible to calculate the pose according to [8].

3 Results and conclusion

To evaluate the precision of the pose estimation procedure reference measurements are done with the Optotrak Certus tracking system [9]. With a fixed pose of the Optotrack camera and the video camera, respectively, the transformations “Optotrack to video camera” and “video camera to QR code” are recorded and evaluated. Measurements are done at five different distance ranges (200–300 mm, 300–500 mm, 500–650 mm, 650–750 mm, 750–900 mm) for angles from 0° to 40° in all directions. The RMS error is computed from 20 measurements in each distance range. The camera for image acquisition is a HD camera from the XIMEA xiQ series with a resolution of 1280 × 1024 px and 60 frames per second.

The accuracy for the iterative homography method varies between 6 mm and 14 mm for the translation at a distance from 200 mm to 900 mm and QR code sizes of 50 × 50 mm and 100 × 100 mm. The rotation accuracy lies between 2° and 5°. The pose estimation based on point correspondences, by fitting lines to the QR code structure, shows better accuracy results compared to the iterative homography method. In this case, the translation accuracy ranges from 6 mm to 8 mm also at a distance from 200 mm to 900 mm and QR code sizes of 50 × 50 mm and 100 × 100 mm. The rotation accuracy ranges from 1° to 4°.

The current setup allows a detection and pose estimation process with one to five frames per second, but can be accelerated by optimizations on the CPU and GPU. QR codes can be detected for a tilting angle up to 60° dependent on the marker size and distance to the camera. The number of QR codes to track in one image is not limited.

The presented measurement system for surgical navigation purposes is a novel approach with several advantages compared to conventional optical or electromagnetic tracking systems. Especially the high level of integration by using one video camera integrated in a surgical light benefits a potential navigation application. The line of sight between the video camera and QR codes is less likely to blocked because the camera is positioned above the patient and the lighting of the surgical lamp is intuitively not interrupted for optimal illumination of the surgery area. The marker size and marker to tip transformation can be encoded in the QR code. Therefore, QR markers of any size can be attached to various instruments without system adaption. The presented algorithm detects several QR codes in an image by detecting the QR code finder patterns and classifying them according to certain features. A region of interest is created for each QR code in the image, which is the basis for exact QR code detection and pose estimation. The algorithms have been tested under several conditions and show robust results even for distorted, rotated, and tilted QR code markers.

It could be visually identified that the detection process for computing the point correspondences is still error-prone resulting in mismatches. Therefore the system accuracy is not sufficient to fulfil the requirements for a surgical navigation application yet, but can be improved in future work.

Author’s Statement

Research funding: The author state no funding involved. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, and has been approved by the authors’ institutional review board or equivalent committee.

References

[1] Balachandran R, Fitzpatrick M, Labadie R. Accuracy of image-guided surgical systems at the lateral skull base as clinically assessed using bone-anchored hearing aid posts as surgical targets. Otol Neurotol. 2008;29:1050–5.10.1097/MAO.0b013e3181859a08Search in Google Scholar PubMed PubMed Central

[2] Baszynski M, Moron Z, Tewel N. Electromagnetic navigation in medicine – basic issues, advantages and shortcomings, prospects of improvement. J Phys Conf Ser. 2010;238:012056.10.1088/1742-6596/238/1/012056Search in Google Scholar

[3] Engel S, Katanacho M, Keeve E, Uhlmann E. Verfahren, Anordnung und Computerprogrammprodukt zur Lageerfassung eines zu untersuchenden Objekts 2015, DE Patent DE 10 2015 212 352.9.Search in Google Scholar

[4] Kong S. QR code image correction based on corner detection and convex hull algorithm. J Multimed. 2013;8:662–8.10.4304/jmm.8.6.662-668Search in Google Scholar

[5] Briechle K, Hanebeck U. Template matching using fast normalized cross correlation. Institute of automatic Control Engineering, Technische Universitaet Muenchen, 2001.10.1117/12.421129Search in Google Scholar

[6] King D. Dlib-ml: a machine learning toolkit. J Mach Learn Res. 2009;10:1755–8.Search in Google Scholar

[7] Abawi D, Bienwald J, Dörner R. Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit: 2004: Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), 2004.Search in Google Scholar

[8] Zhang Z. A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell. 2000;22:1330–4.10.1109/34.888718Search in Google Scholar

[9] Mazumder MMG, Kim S, Park SJ. Precision and repeatability analysis of optotrak Certus as a tool for gait analysis utilizing a 3D robot. J Eng Technol Res. 2011;3:37–43.Search in Google Scholar

Published Online: 2016-9-30
Published in Print: 2016-9-1

©2016 Manuel Katanacho et al., licensee De Gruyter.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Articles in the same Issue

  1. Synthesis and characterization of PIL/pNIPAAm hybrid hydrogels
  2. Novel blood protein based scaffolds for cardiovascular tissue engineering
  3. Cell adhesion and viability of human endothelial cells on electrospun polymer scaffolds
  4. Effects of heat treatment and welding process on superelastic behaviour and microstructure of micro electron beam welded NiTi
  5. Long-term stable modifications of silicone elastomer for improved hemocompatibility
  6. The effect of thermal treatment on the mechanical properties of PLLA tubular specimens
  7. Biocompatible wear-resistant thick ceramic coating
  8. Protection of active implant electronics with organosilicon open air plasma coating for plastic overmolding
  9. Examination of dielectric strength of thin Parylene C films under various conditions
  10. Open air plasma deposited antimicrobial SiOx/TiOx composite films for biomedical applications
  11. Systemic analysis about residual chloroform in PLLA films
  12. A macrophage model of osseointegration
  13. Towards in silico prognosis using big data
  14. Technical concept and evaluation of a novel shoulder simulator with adaptive muscle force generation and free motion
  15. Usability evaluation of a locomotor therapy device considering different strategies
  16. Hypoxia-on-a-chip
  17. Integration of a semi-automatic in-vitro RFA procedure into an experimental setup
  18. Fabrication of MEMS-based 3D-μECoG-MEAs
  19. High speed digital interfacing for a neural data acquisition system
  20. Bionic forceps for the handling of sensitive tissue
  21. Experimental studies on 3D printing of barium titanate ceramics for medical applications
  22. Patient specific root-analogue dental implants – additive manufacturing and finite element analysis
  23. 3D printing – a key technology for tailored biomedical cell culture lab ware
  24. 3D printing of hydrogels in a temperature controlled environment with high spatial resolution
  25. Biocompatibility of photopolymers for additive manufacturing
  26. Biochemical piezoresistive sensors based on pH- and glucose-sensitive hydrogels for medical applications
  27. Novel wireless measurement system of pressure dedicated to in vivo studies
  28. Portable auricular device for real-time swallow and chew detection
  29. Detection of miRNA using a surface plasmon resonance biosensor and antibody amplification
  30. Simulation and evaluation of stimulation scenarios for targeted vestibular nerve excitation
  31. Deep brain stimulation: increasing efficiency by alternative waveforms
  32. Prediction of immediately occurring microsleep events from brain electric signals
  33. Determining cardiac vagal threshold from short term heart rate complexity
  34. Classification of cardiac excitation patterns during atrial fibrillation
  35. An algorithm to automatically determine the cycle length coverage to identify rotational activity during atrial fibrillation – a simulation study
  36. Deriving respiration from high resolution 12-channel-ECG during cycling exercise
  37. Reducing of gradient induced artifacts on the ECG signal during MRI examinations using Wilcoxon filter
  38. Automatic detection and mapping of double potentials in intracardiac electrograms
  39. Modeling the pelvic region for non-invasive pelvic intraoperative neuromonitoring
  40. Postprocessing algorithm for automated analysis of pelvic intraoperative neuromonitoring signals
  41. Best practice: surgeon driven application in pelvic operations
  42. Vasomotor assessment by camera-based photoplethysmography
  43. Classification of morphologic changes in photoplethysmographic waveforms
  44. Novel computation of pulse transit time from multi-channel PPG signals by wavelet transform
  45. Efficient design of FIR filter based low-pass differentiators for biomedical signal processing
  46. Nonlinear causal influences assessed by mutual compression entropy
  47. Comparative study of methods for solving the correspondence problem in EMD applications
  48. fNIRS for future use in auditory diagnostics
  49. Semi-automated detection of fractional shortening in zebrafish embryo heart videos
  50. Blood pressure measurement on the cheek
  51. Derivation of the respiratory rate from directly and indirectly measured respiratory signals using autocorrelation
  52. Left cardiac atrioventricular delay and inter-ventricular delay in cardiac resynchronization therapy responder and non-responder
  53. An automatic systolic peak detector of blood pressure waveforms using 4th order cumulants
  54. Real-time QRS detection using integrated variance for ECG gated cardiac MRI
  55. Preprocessing of unipolar signals acquired by a novel intracardiac mapping system
  56. In-vitro experiments to characterize ventricular electromechanics
  57. Continuous non-invasive monitoring of blood pressure in the operating room: a cuffless optical technology at the fingertip
  58. Application of microwave sensor technology in cardiovascular disease for plaque detection
  59. Artificial blood circulatory and special Ultrasound Doppler probes for detecting and sizing gaseous embolism
  60. Detection of microsleep events in a car driving simulation study using electrocardiographic features
  61. A method to determine the kink resistance of stents and stent delivery systems according to international standards
  62. Comparison of stented bifurcation and straight vessel 3D-simulation with a prior simulated velocity profile inlet
  63. Transient Euler-Lagrange/DEM simulation of stent thrombosis
  64. Automated control of the laser welding process of heart valve scaffolds
  65. Automation of a test bench for accessing the bendability of electrospun vascular grafts
  66. Influence of storage conditions on the release of growth factors in platelet-rich blood derivatives
  67. Cryopreservation of cells using defined serum-free cryoprotective agents
  68. New bioreactor vessel for tissue engineering of human nasal septal chondrocytes
  69. Determination of the membrane hydraulic permeability of MSCs
  70. Climate retainment in carbon dioxide incubators
  71. Multiple factors influencing OR ventilation system effectiveness
  72. Evaluation of an app-based stress protocol
  73. Medication process in Styrian hospitals
  74. Control tower to surgical theater
  75. Development of a skull phantom for the assessment of implant X-ray visibility
  76. Surgical navigation with QR codes
  77. Investigation of the pressure gradient of embolic protection devices
  78. Computer assistance in femoral derotation osteotomy: a bottom-up approach
  79. Automatic depth scanning system for 3D infrared thermography
  80. A service for monitoring the quality of intraoperative cone beam CT images
  81. Resectoscope with an easy to use twist mechanism for improved handling
  82. In vitro simulation of distribution processes following intramuscular injection
  83. Adjusting inkjet printhead parameters to deposit drugs into micro-sized reservoirs
  84. A flexible standalone system with integrated sensor feedback for multi-pad electrode FES of the hand
  85. Smart control for functional electrical stimulation with optimal pulse intensity
  86. Tactile display on the remaining hand for unilateral hand amputees
  87. Effects of sustained electrical stimulation on spasticity assessed by the pendulum test
  88. An improved tracking framework for ultrasound probe localization in image-guided radiosurgery
  89. Improvement of a subviral particle tracker by the use of a LAP-Kalman-algorithm
  90. Learning discriminative classification models for grading anal intraepithelial neoplasia
  91. Regularization of EIT reconstruction based on multi-scales wavelet transforms
  92. Assessing MRI susceptibility artefact through an indicator of image distortion
  93. EyeGuidance – a computer controlled system to guide eye movements
  94. A framework for feedback-based segmentation of 3D image stacks
  95. Doppler optical coherence tomography as a promising tool for detecting fluid in the human middle ear
  96. 3D Local in vivo Environment (LivE) imaging for single cell protein analysis of bone tissue
  97. Inside-Out access strategy using new trans-vascular catheter approach
  98. US/MRI fusion with new optical tracking and marker approach for interventional procedures inside the MRI suite
  99. Impact of different registration methods in MEG source analysis
  100. 3D segmentation of thyroid ultrasound images using active contours
  101. Designing a compact MRI motion phantom
  102. Cerebral cortex classification by conditional random fields applied to intraoperative thermal imaging
  103. Classification of indirect immunofluorescence images using thresholded local binary count features
  104. Analysis of muscle fatigue conditions using time-frequency images and GLCM features
  105. Numerical evaluation of image parameters of ETR-1
  106. Fabrication of a compliant phantom of the human aortic arch for use in Particle Image Velocimetry (PIV) experimentation
  107. Effect of the number of electrodes on the reconstructed lung shape in electrical impedance tomography
  108. Hardware dependencies of GPU-accelerated beamformer performances for microwave breast cancer detection
  109. Computer assisted assessment of progressing osteoradionecrosis of the jaw for clinical diagnosis and treatment
  110. Evaluation of reconstruction parameters of electrical impedance tomography on aorta detection during saline bolus injection
  111. Evaluation of open-source software for the lung segmentation
  112. Automatic determination of lung features of CF patients in CT scans
  113. Image analysis of self-organized multicellular patterns
  114. Effect of key parameters on synthesis of superparamagnetic nanoparticles (SPIONs)
  115. Radiopacity assessment of neurovascular implants
  116. Development of a desiccant based dielectric for monitoring humidity conditions in miniaturized hermetic implantable packages
  117. Development of an artifact-free aneurysm clip
  118. Enhancing the regeneration of bone defects by alkalizing the peri-implant zone – an in vitro approach
  119. Rapid prototyping of replica knee implants for in vitro testing
  120. Protecting ultra- and hyperhydrophilic implant surfaces in dry state from loss of wettability
  121. Advanced wettability analysis of implant surfaces
  122. Patient-specific hip prostheses designed by surgeons
  123. Plasma treatment on novel carbon fiber reinforced PEEK cages to enhance bioactivity
  124. Wear of a total intervertebral disc prosthesis
  125. Digital health and digital biomarkers – enabling value chains on health data
  126. Usability in the lifecycle of medical software development
  127. Influence of different test gases in a non-destructive 100% quality control system for medical devices
  128. Device development guided by user satisfaction survey on auricular vagus nerve stimulation
  129. Empirical assessment of the time course of innovation in biomedical engineering: first results of a comparative approach
  130. Effect of left atrial hypertrophy on P-wave morphology in a computational model
  131. Simulation of intracardiac electrograms around acute ablation lesions
  132. Parametrization of activation based cardiac electrophysiology models using bidomain model simulations
  133. Assessment of nasal resistance using computational fluid dynamics
  134. Resistance in a non-linear autoregressive model of pulmonary mechanics
  135. Inspiratory and expiratory elastance in a non-linear autoregressive model of pulmonary mechanics
  136. Determination of regional lung function in cystic fibrosis using electrical impedance tomography
  137. Development of parietal bone surrogates for parietal graft lift training
  138. Numerical simulation of mechanically stimulated bone remodelling
  139. Conversion of engineering stresses to Cauchy stresses in tensile and compression tests of thermoplastic polymers
  140. Numerical examinations of simplified spondylodesis models concerning energy absorption in magnetic resonance imaging
  141. Principle study on the signal connection at transabdominal fetal pulse oximetry
  142. Influence of Siluron® insertion on model drug distribution in the simulated vitreous body
  143. Evaluating different approaches to identify a three parameter gas exchange model
  144. Effects of fibrosis on the extracellular potential based on 3D reconstructions from histological sections of heart tissue
  145. From imaging to hemodynamics – how reconstruction kernels influence the blood flow predictions in intracranial aneurysms
  146. Flow optimised design of a novel point-of-care diagnostic device for the detection of disease specific biomarkers
  147. Improved FPGA controlled artificial vascular system for plethysmographic measurements
  148. Minimally spaced electrode positions for multi-functional chest sensors: ECG and respiratory signal estimation
  149. Automated detection of alveolar arches for nasoalveolar molding in cleft lip and palate treatment
  150. Control scheme selection in human-machine- interfaces by analysis of activity signals
  151. Event-based sampling for reducing communication load in realtime human motion analysis by wireless inertial sensor networks
  152. Automatic pairing of inertial sensors to lower limb segments – a plug-and-play approach
  153. Contactless respiratory monitoring system for magnetic resonance imaging applications using a laser range sensor
  154. Interactive monitoring system for visual respiratory biofeedback
  155. Development of a low-cost senor based aid for visually impaired people
  156. Patient assistive system for the shoulder joint
  157. A passive beating heart setup for interventional cardiology training
Downloaded on 21.4.2026 from https://www.degruyterbrill.com/document/doi/10.1515/cdbme-2016-0079/html
Scroll to top button