Framework for 2D-3D image fusion of infrared thermography with preoperative MRI

Introduction

Infrared thermography measures the emitted infrared radiation of the exposed cerebral cortex during neurosurgical interventions. According to the Stefan-Boltzmann law, this radiation correlates with the temperature of the object. In neurosurgery, the temperature distribution of the brain’s surface may help to differentiate between healthy and tumor tissue [6]. One such application was demonstrated by Steiner et al. [22] who unveiled the cortical perfusion by intraoperative infrared thermography. However, thermographic images are difficult to analyze as the infrared camera operates on a different spectrum than human visual perception. This introduces additional challenges when transferring information from infrared thermography to surgical staff. As intraoperative infrared thermography is used as a medical decision support system, it is important for non-technical personnel to interpret thermographic images as well as results originating from data analysis workflows. One approach to this challenge is to fusion the novel infrared thermography with prevailing imaging modalities.

Image fusion denotes the process of joining multiple image sources to one image. The images show the same scene, yet they may have been recorded by different modalities or contain overlapping parts or both. Image fusion requires the images to be aligned such that structures being visible in each appear at the same spatial position. This transformation is called image registration and requires establishing a coordinate transform [16] by feature matching [2, 3], optimizing a similarity measure [12, 15] or by calibration-based approaches [13]. The latter relies on external tracking systems, pre-calibrated instruments or manual measurements. An exhaustive taxonomy of image registration methods can be found in [11].

Feature-based image registration is done by extracting features in all images and solving the correspondence problem. Intensity-based features [7] rely on the detected intensities of pixels or groups of them. In contrast, shape-based features often rely on spatial relationships. Extracting features and solving the correspondence problem is a process that might introduce considerable computational complexity. If the application requires real-time or near-real-time performance, it is necessary to use features that can be calculated and correlated very fast. Furthermore, feature extraction will only work if all images represent similar structures or processes. As thermographic images are mainly influenced by cerebral perfusion while MRI images represent anatomical structures, the identification of corresponding features is cumbersome [4, 20].

Calibration-based image registration approaches are based on offline estimation of camera parameters. These parameters can be enriched by camera tracking information to establish an efficient coordinate transformation [13]. Contrary to feature-based approaches, there is no need for solving potentially expensive correspondence problems or minimization of similarity measures. Sergeeva et al. [21] demonstrate guided intraoperative resection during neurosurgery by calibration-based image registration. The probe of an ultrasound device is tracked by a neuronavigation system. Given the calibration parameters and the tracking data, the tip of the probe and the acquired ultrasound data can then be correlated with the 3D imaging data of the neuronavigation system. The absence of feature extraction and matching makes calibration-based image registration more robust when registering multimodal images. For this reason, we employ calibration-based image registration to join intraoperative 2D thermographic images with preoperative volumetric MRI data.

After successful image registration, the images are aligned and overlayed. In the affine transformation case covered in this study, the required parameters for projecting the 2D thermographic image onto the MRI dataset’s surface require a translation, rotation and a scaling operation. The position of the virtual 2D thermographic image relative to the 3D model resembles the position of the infrared camera’s sensor relative to the subject. In order to fusion both datasets, a projection step is necessary. Projection finds, for each pixel of the 2D thermographic image, the corresponding voxel of the 3D MRI dataset, and fuses the thermographic information with the anatomical information seen in the 3D MRI. By assuming a pin-hole camera model, a projective mapping of every pixel to its corresponding voxel depends on depth information of the imaged 2D scene [23]. If the scene is sufficiently planar, depth information can be recovered from the camera’s focal distance. Other approaches require use of stereo camera systems [20] or light field cameras [24]. Another efficient [17] approach is called texture mapping [4], which was used in [1] for the image fusion of thermographic images with 3D models. Yet, contrary to ray casting [9] or projective mappings, occluded pixels are assigned a color. Therefore, the 3D model is enriched by interpolated information which have not been measured by the infrared camera. In this work, we use texture mapping as it is a fast approach and promises good results.

Kaczmarek et al. [10] perform image fusion in order to analyze burnt skin during cardio surgery and for thermal mammography. For this purpose, they combine information of a 3D object model acquired via 3D scanning and triangulation with 2D thermographic images from an infrared camera. Contrary to the proposed calibration-based method, their approach relies on manual selection of corresponding points in the 3D mesh and the thermographic image. Additionally, their approach relies on a fixed camera position and requires recalibration once the camera’s orientation is changed. In Sanches et al. [19] also propose to improve the diagnostic usefulness of thermal imaging data by CTI and MRI.

The contribution of this study is an efficient framework for 2D-3D image registration and image fusion framework to enhance 2D thermographic images by anatomical structures of preoperative MRI recordings and vice versa. The presented method relies on performing calibration-based 2D-3D image registration of infrared thermography and MRI. By this, the calibration parameters have to be estimated only once and image registration and image fusion is primarily reduced to robust and computationally efficient coordinate transformations.

Materials and methods

Image-guided neurosurgery requires referencing the patient undergoing brain surgery with respect to his or her preoperative MRI dataset. This procedure is specific to the employed neuronavigation system. It is achieved by placing fiducial markers on the patient’s head prior to acquiring the MRI dataset. Afterwards, the surgeon has to manually define or segment the position of these fiducial markers in the recorded MRI dataset. During surgery, the surgeon touches each fiducial marker by the neuronavigation system’s pointing device. Hereby, a coordinate transform between the intraoperative scene and the preoperative MRI dataset is established. By attaching an instrument adapter (IA) to the infrared camera, the neuronavigation system now continuously provides us the position and orientation of the IA relative to the referenced patient.

In the discussed application domain, image registration denotes the process of estimating and using a transformation function to map 2D points from the infrared camera’s coordinate system to the MRI’s 3D coordinate system. This means that the spatial position and orientation of the virtual 2D thermographic image plane with respect to the surface of the MRI dataset resembles the relative position and orientation of the infrared camera to the exposed brain during neurosurgery. Subsequently, the image is projected onto the surface by texture mapping. The whole workflow is sketched in Figure 1. As affine functions are linear in homogeneous coordinates, we represent all 3D Euclidean coordinates p=(x, y, z) by 4D homogeneous coordinates p′=(x′=λx, y′=λy, z′=λz, λ) (see [8] for details) to simplify notation and computations. Note that p can be recovered from p′ by p=(x′λ, y′λ, z′λ).

Figure 1:

The parameters for image registration have to be estimated once in a calibration step.

Intraoperatively, each 2D image is then annotated by the 3D position of the 2D imaging device in the referenced 3D MRI coordinate system. This allows us to fusion the results of intraoperative data analysis workflows (purple) with preoperative MRI data. Lastly, the results are visualized using Amira.

Image registration

Tracking the IA allows us to estimate the spatial position and orientation of the infrared camera’s sensor array. In order to project the acquired 2D thermographic image onto the position and orientation of the camera’s sensor array in the MRI coordinate system v′→∈ℝ4 we have to estimate the parameters of the compound affine transformation T: ℝ⁴→ℝ⁴:

(1)v′→=T(v→)=Mscale·Mc·Ma·v→

M_c∈ℝ^4×4 describes the calibration matrix while M_a∈ℝ^4×4 handles additional transformations. Vector v→∈ℝ4 resembles the position and direction of the IA. By ignoring M_c and M_â, the thermographic image would be centered at the position of the IA. M_scale corrects the virtual pixel size to the real pixel size depending on the object distance. The whole image registration process consists of three elements:

1. Tracking position adjustment

2. Thermographic image plane adjustment

3. Pixel size correction

Tracking position adjustment:

The calibration matrix M_c consists of translational and rotational components M_TCalib∈ℝ^4×4 and M_RCalib∈ℝ^4×4, respectively:

(2)Mc=MRCalib·MTCalib

and describes the orientation adjustment of the image plane with respect to the orientation of the infrared camera (see Figure 2A). The rotational parameters compensate the directional difference between the axis of the IA and the surface normal of the infrared camera’s sensor:

Figure 2:

The tracked position of the attached instrument adapter is projected onto the position of the camera’s sensor.

The required operations (translation M_TCalib, rotation M_RCalib) to project the instrument adapter’s position v→ onto the infrared camera’s sensor position v′→ is shown (A). Image (B) visualizes the required point pairs P₁, P₂ and P₃, and for estimating rotational parameters for tracking position adjustment (refer to “Tracking position adjustment”).

(3)MRCalib = Rx(α) · Ry(β) · Rz(γ)

Rotation matrices R_x∈ℝ^4×4, R_y∈ℝ^4×4 and R_z∈ℝ^4×4 realize rotations in the X, Y and Z planes given 4D homogenous coordinates. Parameter estimation is achieved by the following two steps: first, the infrared camera is oriented, such that it points vertically downwards (as sketched in Figure 2B). The orientation is validated by comparing the respective dimension of all three non-coplanar point pairs P₁, P₂ and P₃ (see Figure 2B). Physically, these points are represented by fiducial markers attached to the infrared camera. Once the camera is oriented correctly, the two points of pairs P₁, P₂ and P₃ yield the same y, z or x coordinate, respectively. Second, a configuration of rotational parameters (α∈ℝ, β∈ℝ and γ∈ℝ) is estimated manually, such that the normal vector of the infrared camera’s image plane points in the same direction as the physical device (downwards). We chose to realize this process by a graphical user interface that provides feedback about the actual orientation of P₁, P₂ and P₃ as well as the orientation of the normal vector and allows altering rotational parameters.

The required translation parameters are estimated using spatially referenced MRI data of an imaging phantom which is imaged by the infrared camera (see Figures 3 and 4). The parameters (t₁, t₂ and t₃)(are determined manually such that the distance between the spatial positions of the imaging phantom in the MRI coordinate system with its imaged surface in the 2D thermographic image is minimized.

Figure 3:

Setup for 3D-2D image fusion.

(A) Neuronavigation system. (B) Imaging phantom with fiducial markers. (C) Instrument adapter. (D) Infrared camera. (E) Laptop running Amira to process tracking and image data.

Figure 4:

Thermographic images are joined with volumetric MRI data by parallel projection.

Texture mapping allows us to directly fusion the thermographic image onto the spatial referenced imaging phantom (A). Gray voxels of (B) originate from an MRI dataset of the imaging phantom, while yellow-to-red colored pixels encode its surface temperature distribution that had been detected by the infrared camera.

(4)MTCalib = (100t1010t2001t30001)

Thermographic image plane adjustment:

The framework uses two additional transformations to ease handling of the 2D thermographic image, which are collapsed into M_a∈ℝ^4×4. M_{TranslateCenter}∈ℝ^4×4 translates the virtual origin of the image from the corner to its center. After this translation, the center of scaling and rotation is at the center of the image. Second, the image is rotated by 180° around the global x- and y-axes by rotation matrix R_XY180∈ℝ^4×4 as the infrared camera provides us a mirrored image.

(5)Ma =RXY180 · MTranslateCenter

Pixel size correction:

Image fusion using orthogonal projection requires us to rescale the virtual 2D image such that the pixel resolution of the MRI dataset and thermographic images match. This process depends on estimating the object distance between the infrared camera and the imaged cortical surface. The correlation between the focal point p_F∈ℝ and object distance f(p_F):ℝ→ℝ is modeled by

(6)f(pF)=HH+(FAC/pf–INF)

INF∈ℝ represents the maximum observable distance, HH∈ℝ approximates the lens’ front principal point and FAC∈ℝ denotes a scaling factor. By creating a training set of focus-point-to-object distance pairs, the parameters {HH, FAC, INF} can be estimated by least-squares.

As we know the horizontal and vertical fields of view of the infrared camera’s lens FOV_h∈ℝ, FOV_v∈ℝ, we are able to calculate the actual vertical d_v∈ℝ and horizontal image size d_h∈ℝ at an arbitrary focus value F∈ℝ

(7)dh=2∗sin(FOVh)∗f(F)

(8)dv=2∗sin(FOVv)∗f(F)

Both estimates contribute to the image scaling matrix M_scale∈ℝ^4×4:

(9)MScale=(dh6400000dv4800000100001)

This matrix allows us to scale the image’s width and height to appropriate size. The constants 640 and 480 originate from the infrared camera’s focal plane array detector size of 640×480 elements (pixels). This transformation, therefore, restores true image dimensions and enables subsequent image fusion.

Image fusion

At this point, the image is transformed such that it is located on the infrared camera’s sensor array, with its size corresponding to the extent of the infrared camera frustum at the focal plane. The final task depicts the projection of the 2D thermographic image onto the surface of the 3D MRI dataset. Hereby, spatial information of the 3D MRI dataset is merged with temperature information extracted from the thermographic image. In the discussed neurosurgical application, an isosurface of the cerebral cortex has to be computed from preoperative MRI data. Brain segmentation algorithms (see [3]) fulfill this task by removing voxels not representing brain tissue. After successful brain segmentation, an isosurface of the cerebral cortex can be computed. As the registered 2D thermographic image is at the correct spatial position, orientation and scale, we are now able to map it onto this isosurface. As discussed, we realize this projection by texture mapping. Hereby, a 2D image patch is projected onto the 3D surface by mapping the image to surface coordinates. In our case, an orthogonal projection can be used to project the registered 2D thermographic image coordinates along the image normal vector onto the isosurface [1] (see Figure 5).

Figure 5:

The proposed framework allows the combination of intraoperative thermographic images with the extracted isosurface of the human brain.

The thermographic image was acquired during the resection of a hypotherm renal cell carcinoma metastasis (blue color).

Results

The validation MRI dataset was acquired by a Siemens Magnetom Verio MRI scanner imaging a novel phantom [26] (see Figure 4). The phantom consists of three parts of plastics, an inner balloon, a tube set-up and two syringes as well as necessary connectors. The casing of the phantom simulates the head of a patient. The lid with a circular hole simulates a trepanation. Fiducial markers are attached to three sides of the imaging phantom. Thermographic images were recorded with an InfraTec VarioCAM HD head 680 S infrared camera. We further employed the neuronavigation system BrainLab VectorVision 2.1.1 cranial, BrainLab iPlanNet 3.0 and respective IAs. Image registration and image fusion was done in Amira 5.5 [5].

IA

To track the spatial position and orientation of the infrared camera, it is necessary to attach an IA to the camera. The employed IA has to be calibrated in every image-guided session in order to prevent the usage of incorrectly calibrated devices. Brainlab VectorVision Cranial 2.1.1 provides information about the angular and axial error resulting from calibrating the IA. This procedure was performed 10 times and the respective results are shown in Table 2. An average angular error of 0.2° influences the image fusion process. The orientation of the IA affects the determination of the calibration parameters and, therefore, decreases the accuracy of the image registration process.

Table 2:

Calibrating the instrument adapter and mounting it to the infrared camera introduces inaccuracies to the image registration process due to manufacturing inaccuracies.

	Angle (°)	Tip (mm)
Calibration	0.2±0.1	0.1±0.1
IA mounting	96±0.5	479.4±0.4

Following this calibration step, the IA was attached to a pre-existing mounting of the infrared camera (see Figure 6). The camera was kept at a fixed position while the orientation of the IA was evaluated for 10 mountings. While repeatedly mounting the IA, we found a standard deviation of 0.5° of the IA’s orientation. This translates to a variable axial offset depending on the distance between the IA and the subject.

Figure 6:

Manufacturing tolerances affect the overall accuracy.

Image (A) shows the mounted instrument adapter, while the actual mounting can be seen in (B). Due to manufacturing tolerances, the instrument adapter can be tilted slightly to the left and the right, decreasing the overall accuracy. This issue can be circumvented by using specialized instrument adapters and mountings that weren’t available for this study.

Discussion

Calibration-based image registration requires a fixed alignment of our 2D camera and the IA. Furthermore, tracking information have to be provided by a neuronavigation system. Both requirements are commonly fulfilled in image-guided surgery. The whole framework was implemented as a C++ plugin for the visualization software Amira 5.5 [5]. Limitations of Amira forced us to approximate the physically accurate pinhole camera model by an orthogonal projection for image fusion. The missing perspective distortion is compensated by the previously discussed scaling operation M_scale. The proposed procedure works well under the assumption that the mapped surface is close to the focal plane. The actual projection is realized by texture mapping, meaning that the 2D thermographic image could be efficiently projected on any surface. The main drawback of texture mapping is that covert objects also get texturized by interpolated information. Yet, as we are imaging the approximately convex surface of the human brain, this drawback can be neglected.

We found that the calibration and the attachment of the IA to the imaging device is the most critical factor for the overall accuracy. To quantify this observation, the IA was placed at two extreme positions, denoted by –φ and +φ. +φ refers to the extreme position, where the adapter is tilted to the right, whilst –φ represents the opposite direction. This kind of attachment is possible as the mechanism to attach the IA to the mounting has a minimal manufacturing tolerance. In the worst case, this effect degrades the overall accuracy to 10.06 mm (see Table 3).

Further, the calibration strongly depends on the quality of the tracking data. Inaccurate spatial referencing of the patient or target as well as inaccurately defined fiducial marker positions degrade the overall accuracy. For best accuracy, it is necessary to refine calibration parameters of the image registration transform each time the IA is recalibrated to the neuronavigation system and re-mounted to the camera. Otherwise, the average error increases to 9 mm. By using pre-calibrated IAs and special mountings, an accuracy of 2.46 mm is achievable without refinement of calibration parameters.

Finally, we determined the overall error at varying orientations of the IR camera with optimal mounting of the IA. The camera was rotated such that it points orthogonal, 30° and 60° to the imaged object. The distance of the fiducial markers visible in the fusioned thermographic image plane to the respective position in the 3D MRI dataset was computed. The results at various orientations of the IR camera indicate a cumulative error of 2.46 mm (approximately 2.5–5 voxels) including all preceding influential factors (see Tables 4 and 5 ), while the accuracy is maximized if the camera points orthogonally to the imaged scene. An orthogonal orientation of the infrared camera also depicts a common intraoperative configuration as the signal-to-noise ratio of infrared cameras strongly depends on the angle of incidence of the measured infrared radiation. In this case, no perspective distortions are prevalent as well.

The whole framework is optimized for computational efficiency in order to enable the application in time-critical settings. It further allows subsequent data analysis methods to incorporate anatomical and functional information from preoperative MRI measurements (for example, tumor position or localization of eloquent areas of the cerebral cortex). Furthermore, medical personnel get a tool to validate intraoperative thermographic imaging data by their expectations from prevailing volumetric imaging.

Clinically, most of the required steps for registration and fusion can be done preoperatively. The infrared camera with attached IA has to be calibrated once in order to establish the mapping between the IA’s position and orientation and the camera’s sensor position and orientation. Prior to the actual surgery, the MRI dataset has to be imported into Amira followed by brain segmentation. Intraoperatively, the patient has to be referenced to the neuronavigation system. Attaching a sterile IA to the covered 2D infrared camera now enables the actual image registration and image fusion. All the required software is executed on a laptop. The communication between Amira and the neuronavigation system is handled by the developed Amira plugin via the standardized OpenIGTLink interface. This means that the software is not limited to neuronavigation systems of BrainLab and can be used with any neuronavigation system that implements OpenIGTLink. It might be favorable to either display the fusioned data on a laptop screen or to output the Amira window to the neuronavigation system by its video connector.

Summary

The proposed calibration-based image registration and image fusion framework allows the combination of intraoperative 2D imaging and preoperative 3D MRI data. In neurosurgery, joining intraoperative perfusion information with neuroanatomy provides valuable information to the surgeon. For this purpose, we propose a framework for image registration and image fusion of 3D MRI with 2D thermographic imaging data. The employed calibration-based image registration algorithm transforms the intraoperatively tracked position and orientation information of our 2D infrared camera into the 3D coordinate system originating from preoperative volumetric imaging. By application of an orthogonal projection, the 2D image is projected onto the respective surface of the 3D dataset. In order to quantify the projection accuracy and unveil further potential improvements of the framework, we applied an extensive evaluation scheme. The results indicate a mean accuracy of 2.46 mm given an appropriate setup. We further estimated an upper bound of the accuracy at 10.06 mm. Further work will focus on minimizing this upper bound in order to achieve reasonable accuracy especially when using surgical microscopes with sub-millimeter resolutions.