Cerebral cortex classification by conditional random fields applied to intraoperative thermal imaging

Nico Hoffmann; Edmund Koch; Uwe Petersohn; Matthias Kirsch; Gerald Steiner

doi:10.1515/cdbme-2016-0105

Artikel Open Access

Cerebral cortex classification by conditional random fields applied to intraoperative thermal imaging

Nico Hoffmann , Edmund Koch , Uwe Petersohn , Matthias Kirsch und Gerald Steiner

Veröffentlicht/Copyright: 30. September 2016

Veröffentlicht von

Veröffentlichen auch Sie bei De Gruyter Brill

Informationen für Autor*innen Erkunden Sie dieses Fachgebiet

Aus der Zeitschrift Current Directions in Biomedical Engineering Band 2 Heft 1

Abstract

Intraoperative thermal neuroimaging is a novel intraoperative imaging technique for the characterization of perfusion disorders, neural activity and other pathological changes of the brain. It bases on the correlation of (sub-)cortical metabolism and perfusion with the emitted heat of the cortical surface. In order to minimize required computational resources and prevent unwanted artefacts in subsequent data analysis workflows foreground detection is a important preprocessing technique to differentiate pixels representing the cerebral cortex from background objects. We propose an efficient classification framework that integrates characteristic dynamic thermal behaviour into this classification task to include additional discriminative features. The first stage of our framework consists of learning this representation of characteristic thermal time-frequency behaviour. This representation models latent interconnections in the time-frequency domain that cover specific, yet a priori unknown, thermal properties of the cortex. In a second stage these features are then used to classify each pixel’s state with conditional random fields. We quantitatively evaluate several approaches to learning high-level features and their impact to the overall prediction accuracy. The introduction of high-level features leads to a significant accuracy improvement compared to a baseline classifier.

Keywords: conditional random fields; 2D B-Spline; discrete wavelet transform; intraoperative thermal imaging; machine learning; representation learning

1 Introduction

Intraoperative thermal neuroimaging denotes an non-invasive and contactless imaging technique that measures the emitted temperature radiation of tissue during neurosurgical interventions. In neurosurgery, temperature gradients derive from heat transfers being caused by (sub-)cortical perfusion and neuronal activity. Gorbach et al. [1] and Shevelev et al. [2] showed the application of intraoperative thermography for tumour diagnostics. Steiner et. al. [3] demonstrated the detection of a ice-cold saline solution applied through a central line as a tool for perfusion diagnostics.

High-level feature representations are extracted from data in time-frequency domain so that they unveil time-dependent thermal behaviour of the human cortex. We further evaluate two common approaches to this feature representation learning task and finally analyse their influence to the overall performance. Latter is also compared to a simple baseline method.

2 Methods

Dynamic thermal signals can be described by a combination of several non-stationary characteristic signal components. These components originate from physiological sources as well as from noise. To approach this challenge subsequent signal analysis and classification is done in time-frequency domain to decompose the signal into its characteristic time-dependent components. As the physiological influences of cortical and subcortical perfusion as well as tissue composition are unknown and difficult to estimate there is no parametric model about cortical heat emission. For what reason we propose a machine learning framework that extracts empirical knowledge from learning data using linear feature representation schemes for pixelwise latent state prediction. The research related to human use has been complied with all the relevant national regulations, institutional policies and in accordance the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008). Informed consent has been obtained from all individuals included in this study.

2.1 Preliminaries

The global thermographic signal T∈ℛn×m respectively Ti∈ℛm of any recorded pixel 1 ≤ i ≤ n and m time points is superimposed by several components such as physiological signals and thermic background noise Tb⁢g∈ℛn×m. In order to prevent learning background signals the latter component has to be separated from T prior to further analysis.

The approximation of background noise is done by exploiting its rather smooth nature by a two dimensional B-Spline model^[1] to estimate and remove the background signal. Suppose we are given two one-dimensional B-Spline bases Bx⁢y∈ℛm1×n and Bt∈ℛm2×m consisting of local B-Spline functions given m₁ and m₂ knots to interpolate the data in spatial B_xy and temporal B_t domain. Adjacent local B-spline functions are joined at m₁ resp. m₁knots with m₁ ≪ n and m₂ ≪ m. Let further v⁢e⁢c⁢(T)∈ℛn⁢m be the column stacked version of our measured temperature data T and B_2d = B_xy ⊗ B_t the tensor product ⊗ of both 1D B-Spline bases. The smooth background signal T^bg = B_2dα^ can now be estimated by regressing vec(T) = B_2dα using least squares given the normal equations:

α^=(B2⁢dT⁢B2⁢d)−1⁢B2⁢dT⁢v⁢e⁢c⁢(T)

In order to catch dynamic thermographic effects that correlate with the actual imaged object subsequent analysis is done in time-frequency domain by applying the Discrete Wavelet transform to the data. Given a frequency band (scale) j, index k and wavelet ψ, the background corrected wavelet coefficients c_i( j, k) of pixel i read

ci⁢(j,k)=∑j∑k(Ti⁢(k)−Tib⁢g⁢(k))⁢2−j/2⁢ψ⁢(2−j⁢n−k)

The transformation is done in linear time using the fast wavelet transform based on pyramid algorithm. In the following we will drop ( j, k) to simplify the notation if both are clear from context.

2.2 Representation learning

Now we can proceed to finding a high-level representation of the thermal characteristics of our physiological signals. The representation should be robust to noise while preserving as much information as possible. In the following we will discuss two common approaches to learn a representation f(c_i) of the wavelet transformed thermal data c_i.

Figure 1

This validation dataset was subject to the PCA-CRF classifier and the results are shown in (B). Yellow colour indicate pixels with label cerebral cortex. Expert classification of the cerebral cortex is indicated by the blue solid line in both images.

2.2.1 Bag of frequency words

A common approach to unveil representative features is the bag of words model. In terms of wavelet coefficients ci∈ℛm for n pixels, we solve

μ=a⁢r⁢g⁢m⁢i⁢n{μ1,…,μp}⁢∑i=1n∑μj∈Sj||ci−μj||22

and get representative frequency wordsμj∈ℛm that describe our training dataset. The k-Means algorithm provides one solution to this task, so that our dictionary μ is defined by μ=[μ1..μp]T. This allows us to represent the L₂ distance of any new vector c_i to each of the p words μj∈ℛm (1 ≤ j ≤ p) in terms of our dictionary μ=[μ1,…,μp]T by

f⁢(ci)=||ci⊗1→p−μ||22

with length p column vector of ones 1→p and ⊗ denoting the Kronecker product.

2.2.2 PCA

In the past, PCA was applied to learn a representation of neuroimaging data [6]. Suppose we measure n correlated m-dimensional wavelet transformed time-series stored as n rows in matrix C∈ℛn×m. PCA now gives a set of m orthogonal vectors vj∈ℛm and weights wj∈ℛn with 1 ≤ j ≤ m so that ci=∑jwj⁢vj. It can be shown that these orthogonal vectors v_j correspond to the eigenvectors Φ=[v1,v2,…,vm] of the empirical covariance matrix ΣC=CT⁢C∈ℛm×m so that ΣC⁢Φ=Φ⁢Λ. Each eigenvector v_j now describes a characteristic high-level thermal behaviour of the data. The feature vector c_i of pixel i is represented by

(1)f⁢(ci)=Φp⁢ci

with Φp=[v1⁢…⁢vp]T given dictionary size p.

2.3 Learning unary potentials

Suppose each pixel i has latent state yi∈𝒴 given f(c_i). In case of BoW, f(c_i) denotes the distance of c_i to the learnt p representative frequency words μ_k (or analogously for the PCA approach v_k) with 1 ≤ k ≤ p. The set of states 𝒴={f⁢g,b⁢g} now denotes the two classes of the pixels. Those pixels belonging to the cerebral cortex are classified as foreground (fg) the others as background (bg). The probability p_i(y_i—f(c_i)) of each pixel belonging to state y_i given the feature representation can now be learnt by a Random Forests (RF) [7] consisting of t decision trees. The trained RF yields a measure of certainty (probability) that an observed encoded representation f(c_i) belongs to state y_i as of p(y_i = fg—f(c_i)) or p(y_i = bg—f(c_i)). An ensemble decision regarding the actual state y_i of the RF is obtained by averaging the output p_i^k(y_i—f(c_i)) of each single decision tree k:

p(yi|f(ci))=1/t∑k=1tpik(yi|f(ci))

The probability distribution is estimated by training each of the t decision trees on a bootstrap sample of our training data. This strategy effectively prevents overfitting as discussed by [7].

2.4 Structured classification of thermal neuroimaging data

In order to incorporate structural information into the classification process we propose a conditional random field (CRF) model. In this framework, the posterior distribution p(Y|ℱ𝒞) of our latent variables yi∈𝒴 and observations f⁢(ci)∈ℱ𝒞 can be formulated in terms of unary factors Ψ and pairwise factors ϕ on a undirected graph G = (V, E) as of

(2)logp(Y|ℱ𝒞)=∑i∈VΨi(yi,f(ci))+∑(i,i′)∈Eϕ(yi,yi′)

with V being the set of pixels and E the set of edges between adjacent pixels i and i′. By fulfilling the local Markov property, this function can be factored so that p(Y|ℱ𝒞) only depends on its direct neighbours. The unary potential Ψ_i(y_i, f(c_i)) encodes the prior probability learnt by the RF classifier whilst the pairwise potential ϕ(y_i, y_i′ enforces spatial homogeneity of the inferred labelling. Computations were further simplified by integrating the regular structure of the imaged data through a Potts model function

ϕ⁢(yi,yi′)=C⋅1yt=yt′

with indicator function 1yt=yt′ yielding one if yt=yt′ and zero otherwise. C is a smoothness penalty so that low values of C lead to rough solutions whilst the estimated labelling is getting smoother by increasing C. Since we are dealing with binary labels and Potts pairwise terms equation 2 is submodular what allows the application of very efficient inference method based on graph cuts. Minimizing equation 2 now corresponds to finding a maximum a-posteriori estimate of the labelling y.

3 Results and discussion

The results were achieved by evaluating five intraoperative thermal measurements of five different cases of length 1024 frames (20 s). For this purpose thermal neuroimaging data was acquired just after exposure of the cerebral cortex during neurosurgical tumour resections. Three out of these datasets were used for training plus testing and two for validation. The training sample consists of 30% equally sampled data points of all three test datasets equally representing cortical and background pixels. All computations were done on a workstation with dual Intel Xeon E5-2630, 128 GB Ram and Nvidia Geforce GTX Titan Black graphics card.

We evaluated both PCA and the Bag of Words model for feature representation learning. Baseline performance was quantified by training a RF on the average temperature distribution μT. The accuracy was computed by

a⁢c⁢c=T⁢C+T⁢BC+B

given multiple labeled datasets. TC denotes the number of true cortical-, TB the number of true background- and C as well as B represent the number of cortex and background pixels. These labellings were acquired by a medical expert. The results of both approaches are shown in Tables 1 and 2.

Table 1

The Random Forest classifier with PCA and BoW feature learning significantly outperform the baseline classifier on three training/testing as well as two validation datasets.

RF accuracy (%)	μT	BoW	PCA
Test	86.4	93.1	96.5
Validation	81.3	86.6	88.1

Table 2

Extending the classifier by structural constraints further improves the overall accuracy of the baseline- and both feature learning approaches. For reasons of comparison the approaches were applied to the same data as described in Table 1.

CRF accuracy (%)	μT	BoW	PCA
Test	88.7	96	98.5
Validation	87.7	89.1	89.2

By introducing high-level features into the classification process a significant improvement in accuracy between the baseline classifier and the extended version can be observed. The introduction of learnt high-level thermal features provides additional discriminative information resulting from characteristic thermal signatures. Structural information shrink the difference in accuracy between our baseline classifier and the extended ones. This is caused by temperature inhomogeneities correlating with tissue composition and perfusion which are compensated by the Potts model. In the present cortex classification task this behaviour seems favourable. Yet, in case of smaller objects like tumours or vessels further attention has to be paid to this behaviour in order to achieve reasonable true-positive rates.

4 Conclusion

Intraoperative thermal neuroimaging is a novel technique to image time-dependent cortical temperature variations during neurosurgical interventions. The main cause of temperature changes is cortical perfusion which is influenced by cell metabolism and tissue composition. The thermal processes of the exposed brain are not well understood, yet they provide valuable information to characterize tissue. In this work thermal process signatures are employed to improve the differentiation of pixels of the cerebral cortex to background pixels. For this purpose we propose a novel machine learning framework for analysis of intraoperative thermal neuroimaging data. The learning goal is to recognize dynamic temperature behaviour of the imaged human cortex. These high-level features are then incorporated into a subsequent tissue classification stage based on conditional random fields improving overall classification accuracy. In the future, this framework might enable a more fine-grain characterization of tissue composition based on its dynamic thermal behaviour.

Acknowledgement

The authors would like to thank all organizations and individuals that supported this research project.

Author’s Statement

Research funding: This work was supported by the European Social Fund (grant no. 100087783) and the Saxonian Ministry of Science and Art. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, all intraoperative procedures in this work were approved by the Human Ethics Committee of the Technische Universität Dresden (no. EK 323122008).

References

[1] Gorbach AM, Heiss JD, Kopylev L, Oldfield EH. Intraoperative infrared imaging of brain tumors. J Neurosurg. 2004;101;960–9.10.3171/jns.2004.101.6.0960Suche in Google Scholar

[2] Shevelev IA, Tsicalov EN, Gorbach AM, Budko KP, Sharaev GA. Thermoimaging of the brain. J Neurosci Methods. 1993;46:49–57.10.1016/0165-0270(93)90140-MSuche in Google Scholar

[3] Steiner G, Sobottka SB, Koch E, Schackert G, Kirsch M. Intraoperative imaging of cortical cerebral perfusion by time-resolved thermography and multivariate data analysis. J Biomed Opt. 2011;16:016001–016001–6.10.1117/1.3528011Suche in Google Scholar PubMed

[4] Eilers PH, Currie ID, Durban M. Fast and compact smoothing on large multidimensional grids. Comput Stat Data An. 2006;50:61–76.10.1016/j.csda.2004.07.008Suche in Google Scholar

[5] Durban M, Currie I, Eilers P. Multidimensional P-spline mixed models : a unified approach to smoothing on large grids. 2006;1:2–5.Suche in Google Scholar

[6] Lu H, Plataniotis KN, Venetsanopoulos AN. MPCA: multilinear principal component analysis of tensor objects. IEEE Trans Neural Netw. 2008;19:18–39.10.1109/TNN.2007.901277Suche in Google Scholar PubMed

[7] Breiman L. Random forests. Machine Learning. 2001;45:5–32.10.1023/A:1010933404324Suche in Google Scholar

Published Online: 2016-9-30

Published in Print: 2016-9-1

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Artikel in diesem Heft

https://doi.org/10.1515/cdbme-2016-0105

Schlagwörter für diesen Artikel

conditional random fields; 2D B-Spline; discrete wavelet transform; intraoperative thermal imaging; machine learning; representation learning

Creative Commons

BY-NC-ND 4.0