Home Evaluation of open-source software for the lung segmentation
Article Open Access

Evaluation of open-source software for the lung segmentation

  • Aseel Alnaser EMAIL logo , Bo Gong and Knut Moeller
Published/Copyright: September 30, 2016

Abstract

In this study we evaluated open-source software for lung segmentation. Several parameters which emphasize on functionality, usability, image quality and 3D export are considered for this evaluation. Based on these parameters, a scoring system is generated. Our preliminary evaluation results indicated that the Pulmonary toolkit obtains the best overall performance according to the scoring system. However, the ranking of software shows a certain variation among different criteria. The selection of software should regard the focus and the specific interest of the user.

1 Introduction

Segmentation of the lungs from Computer Tomography (CT) images is considered to be a fundamental vital step in the field of medical imaging and pulmonary image analysis [1]. The need for separating lungs from surrounding tissue, as well as generating 3D models makes it necessary to develop software and toolkits that can perform the segmentation robustly and accurately with minimal user intervention.

In this paper, open-source software for the semi-automatic segmentation of the lungs are compared in terms of their functionality, usability, image quality of segmentation and file or data types of 3D volumes that can be exported. The results obtained are used to evaluate the software, and to deduce which software has better performance for the segmentation and 3D model generation of the lungs.

2 Methods

2.1 Selection of software and toolkits

Seven software and toolkits for the segmentation of the lungs and the airways have been found through Internet:

  1. The Pulmonary toolkit/Github (Ptk for short)*

  2. YaDiv

  3. NIH-CIDI Lung Segmentation tool

  4. TurtleSeg

  5. MITK*

  6. ITK-SNAP*

  7. 3D Slicer*

*software was evaluated in detail

YaDiv requires Java and Java3D in order to work, and although these software were installed, the software did not work. The Lung segmentation tool by NIH-CIDI is only suitable for 2D segmentation. Therefore, only five software were evaluated for the segmentation of the lungs.

The Pulmonary toolkit/Github (Ptk): Ptk is a software suite for the analysis of 3D medical lung images, and is intended for academic and research use only. It requires MATLAB (version R2010 or later) as well as the MATLAB Image Processing toolbox. Some features of this software may additionally require a C++ compiler [2].

TurtleSeg: TurtleSeg provides accurate 3D image segmentation based on the 2D Turtlemap algorithm. The software is a non-open source; however, it was provided as a 32-bit trial version for 1 month after requesting a trial version [3].

Medical Imaging Interaction Toolkit (MITK): MITK is a free open-source software system which combines both the Insight Toolkit (ITK) and the Visualization Toolkit (VTK) with an application framework enabling several toolkits [4].

ITK-SNAP: ITK-SNAP is a software application that provides semi-automatic segmentation in 3D medical images using active contour methods, as well as manual delineation and image navigation. It is a free, open-source, and multi-platform software [5].

3D Slicer: 3D Slicer is a software platform for the analysis and visualization of medical images, and for research in image guided therapy. It enables registration and interactive segmentation, and is a free open source software [6].

2.2 Criteria used for evaluation

In evaluating the software, different parameters were analysed and compared for each software in order to carry out the evaluation in an objective manner. Functionality, usability, image quality and 3D export are regarded to be the most important parameters to survey a software for segmentation. This is because the software are expected to be able to segment the lungs exactly with minimal user-intervention, and has to enable the user to export segmented images of high quality in the form of images as well as 3D models for further processing and measurements.

A scoring system is introduced to make the evaluation process more quantitative and more accurate. For each of the above parameters, an evaluation standard is described which is further presented by points. In the end, the total score was given to each software.

2.2.1 Functionality

Functionality is defined in our study as the ability of the software to segment the lungs with minimal errors. The scale used for evaluating this parameter is:

4: Fully functional, the software can segment each lung on its own, the airways, and the lobes.

3: Functional, the software segments the lungs and the airways together, with the possibility of further segmenting/labelling by the user.

2: Functional, the software fully segments the airways and the lungs without labelling.

1: Functional, but the region segmented needs a lot of editing and smoothing by the user.

2.2.2 Usability

Usability indicates the level of easiness of using the software, and to what extent the software is able to run without intensive user intervention. The following scale has been used for the evaluation of the software:

4: Very easy to use, without user intervention

3: Easy to use, with very minimal user intervention

2: Easy to use, minimal user intervention required with some steps that need to be followed by the user

1: High user intervention is required

2.2.3 Quality of segmentation and 3D export

The quality of segmentation is regarded to be of high quality if the segmentation covers the needed lung region without leakage, and requires no further smoothing or editing. Heuristically, the following scale has been set:

4: high quality, 3: good quality, 2: satisfactory quality,

1: lower quality, requires a lot of editing and smoothing

The software which has more exporting file types of the generated 3D volume provides more options for further processing. One important requirement was the ability of the software to export Stereolithography (STL) files. The following scale was used for evaluation:

3: Can export STL and at least two other data types

2: Can export STL and one of remained types

1: Can export STL or one of remained types

2.3 Dataset

The lung segmentation and the evaluation of the software was performed by using a CT dataset of a male patient. The original dataset was 616 MB with 1219 slices, but it was reduced to 57.8 MB with only 59 slices in order to prevent the slowdown or crashing of software evaluated. The CT images were saved in DICOM form with 512 × 512 pixels.

3 Results

3.1 Functionality

Although all five evaluated software demonstrated ability to segment the lungs, there were variations in terms of to what extent is the software able to get a precise segmentation. Table 1 shows the results for the evaluation of this parameter.

Table 1

Functionality of the evaluated software.

Lungs seg.Airway seg.Lobes seg.labellingScore
Ptk++++4
TurtleSeg+1
MITK++2
ITK-SNAP+++3
3D Slicer++2

3.2 Usability

Table 2 shows the results of evaluating usability of the different software, and reflects the facility to use each software for lung segmentation.

Table 2

Usability of the evaluated software.

DescriptionScore
PtkSegmentation and other features are carried out automatically4
TurtlSegUser has to draw contours for each slice suggested by the software until the segmentation is satisfactory1
MITKThe user has to only specify the threshold3
ITK-SNAPThe user has to only add bubbles to the region that needs to be segmented, specify threshold and run the software2
3D SlicerVolume has to be specified, and threshold has to be adjusted2

3.3 Segmentation quality and 3D export

3.3.1 Quality of segmented images

The quality of the segmentation was based on the quality of the generated 3D models. The segmentation results using the specified dataset are shown in Figure 1. Heuristically, the segmentation quality is stated in Table 3.

Figure 1 Segmented 3D lung volume are shown in anterior view: Ptk (top left), MITK (top right), 3D Slicer (middle left), ITK-SNAP (middle right), and TurtleSeg (bottom left).
Figure 1

Segmented 3D lung volume are shown in anterior view: Ptk (top left), MITK (top right), 3D Slicer (middle left), ITK-SNAP (middle right), and TurtleSeg (bottom left).

Table 3

Evaluating the quality of segmented images in each software.

PtkTurtleSegMITKITK-SNAP3D slicer
DescriptionHighSatisfactoryGoodGoodLowera
Score42331

aNeeds a lot of editing and smoothing.

3.3.2 3D export

Due to the need of exporting 3D models of the lungs for further measurements and applications, it is of importance to evaluate this parameter quantitatively, and for that Table 4 states the evaluation for this parameter.

Table 4

Possible types of 3D export in each software.

STLVTKMesh fileXMLScore
Ptk+1
TurtleSeg++2
MITK++2
ITK-SNAP+++3
3D Slicer+++3

3.4 Final results

For the final evaluation of the software, the individual evaluation of the important parameters shown in the previous sections were added all together for the total evaluation, as shown in Figure 2.

Figure 2 Total evaluation results for the software.
Figure 2

Total evaluation results for the software.

4 Discussion and conclusion

In this study, different parameters are defined to be subcategories of the comparison between five software. It enables the overall evaluation of the software as well as the individual evaluation of specific parameters like functionality or 3D export. As shown in Figure 2, Ptk obtains the highest evaluation score (13 points) in overall. Especially, Ptk performs much better in functionality and usability, given that the user only needs to import the images into the GUI and to press on the segmentation tool. It produces lung segmentation with the best quality and provides further utilities such as airway segmentation. However, it can also be observed that the possible export type of the generated 3D volume is limited to mesh data only.

MITK is also relatively easy to use, as the user needs to only perform the segmentation, and to follow it by adjusting the threshold. As for ITK-SNAP, it has similar evaluation score of MITK for functionality and usability, however, it performed better in functionality given that the user is able to further label and edit the generated 3D volume after segmentation is finished. ITK-SNAP involves more user interaction than MITK, and for that MITK is regarded to be more user friendly. The segmented lung regions in 3D can be exported in several types with MITK as well as ITK-SNAP.

As for the main limitation in our study, it was the use of only one CT dataset for the evaluation of the five software. The segmentation quality could be further evaluated with more CT data of patients with variant lung diseases. Moreover, the evaluation points ensure that the evaluation process is done in a quantitative and an objective manner.

Author’s Statement

Research funding: This work was supported by the BMBF grant no. 03FH038I3 (MOSES) and by the European Commission under grant no. FP7-PIRSES 318943 (eTime). Conflict of interest: Authors state no conflict of interest. Material and methods: Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: The research related to human use complies with all the relevant national regulations, institutional policies and was performed in accordance with the tenets of the Helsinki Declaration, and has been approved by the authors’ institutional review board or equivalent committee.

References

[1] Mansoor A, Bagci U, Foster B, Xu Z, Papadakis GZ, Folio LR, et al. Segmentation and image analysis of abnormal lungs at CT: current approaches, challenges, and future trends. INFORMATICS, July–August 2015.10.1148/rg.2015140232Search in Google Scholar PubMed PubMed Central

[2] Lung medical image analysis and visualisation software for Matlab. 2016. [Online]. Available from: https://github.com/tomdoel/pulmonarytoolkit. [Zugriff am 04 04 2016].Search in Google Scholar

[3] TurtleSeg. Oxipita Inc, 2010–2016. [Online]. Available from: http://www.turtleseg.org/index.php?page=home. [Zugriff am 04 04 2016].Search in Google Scholar

[4] German Cancer Research Center, 2016. [Online]. Available from: http://mitk.org/wiki/The_Medical_Imaging_Interaction_Toolkit_(MITK). [Zugriff am 04 04 2016].Search in Google Scholar

[5] Itk-SNAP, 2014. [Online]. Available from: http://www.itksnap.org/pmwiki/pmwiki.php. [Zugriff am 04 04 2016].Search in Google Scholar

[6] 3D Slicer Wiki pages, 2015. [Online]. Available from: http://wiki.slicer.org/slicerWiki/index.php/Main_Page. [Zugriff am 04 04 2016].Search in Google Scholar

Published Online: 2016-9-30
Published in Print: 2016-9-1

©2016 Aseel Alnaser et al., licensee De Gruyter.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Articles in the same Issue

  1. Synthesis and characterization of PIL/pNIPAAm hybrid hydrogels
  2. Novel blood protein based scaffolds for cardiovascular tissue engineering
  3. Cell adhesion and viability of human endothelial cells on electrospun polymer scaffolds
  4. Effects of heat treatment and welding process on superelastic behaviour and microstructure of micro electron beam welded NiTi
  5. Long-term stable modifications of silicone elastomer for improved hemocompatibility
  6. The effect of thermal treatment on the mechanical properties of PLLA tubular specimens
  7. Biocompatible wear-resistant thick ceramic coating
  8. Protection of active implant electronics with organosilicon open air plasma coating for plastic overmolding
  9. Examination of dielectric strength of thin Parylene C films under various conditions
  10. Open air plasma deposited antimicrobial SiOx/TiOx composite films for biomedical applications
  11. Systemic analysis about residual chloroform in PLLA films
  12. A macrophage model of osseointegration
  13. Towards in silico prognosis using big data
  14. Technical concept and evaluation of a novel shoulder simulator with adaptive muscle force generation and free motion
  15. Usability evaluation of a locomotor therapy device considering different strategies
  16. Hypoxia-on-a-chip
  17. Integration of a semi-automatic in-vitro RFA procedure into an experimental setup
  18. Fabrication of MEMS-based 3D-μECoG-MEAs
  19. High speed digital interfacing for a neural data acquisition system
  20. Bionic forceps for the handling of sensitive tissue
  21. Experimental studies on 3D printing of barium titanate ceramics for medical applications
  22. Patient specific root-analogue dental implants – additive manufacturing and finite element analysis
  23. 3D printing – a key technology for tailored biomedical cell culture lab ware
  24. 3D printing of hydrogels in a temperature controlled environment with high spatial resolution
  25. Biocompatibility of photopolymers for additive manufacturing
  26. Biochemical piezoresistive sensors based on pH- and glucose-sensitive hydrogels for medical applications
  27. Novel wireless measurement system of pressure dedicated to in vivo studies
  28. Portable auricular device for real-time swallow and chew detection
  29. Detection of miRNA using a surface plasmon resonance biosensor and antibody amplification
  30. Simulation and evaluation of stimulation scenarios for targeted vestibular nerve excitation
  31. Deep brain stimulation: increasing efficiency by alternative waveforms
  32. Prediction of immediately occurring microsleep events from brain electric signals
  33. Determining cardiac vagal threshold from short term heart rate complexity
  34. Classification of cardiac excitation patterns during atrial fibrillation
  35. An algorithm to automatically determine the cycle length coverage to identify rotational activity during atrial fibrillation – a simulation study
  36. Deriving respiration from high resolution 12-channel-ECG during cycling exercise
  37. Reducing of gradient induced artifacts on the ECG signal during MRI examinations using Wilcoxon filter
  38. Automatic detection and mapping of double potentials in intracardiac electrograms
  39. Modeling the pelvic region for non-invasive pelvic intraoperative neuromonitoring
  40. Postprocessing algorithm for automated analysis of pelvic intraoperative neuromonitoring signals
  41. Best practice: surgeon driven application in pelvic operations
  42. Vasomotor assessment by camera-based photoplethysmography
  43. Classification of morphologic changes in photoplethysmographic waveforms
  44. Novel computation of pulse transit time from multi-channel PPG signals by wavelet transform
  45. Efficient design of FIR filter based low-pass differentiators for biomedical signal processing
  46. Nonlinear causal influences assessed by mutual compression entropy
  47. Comparative study of methods for solving the correspondence problem in EMD applications
  48. fNIRS for future use in auditory diagnostics
  49. Semi-automated detection of fractional shortening in zebrafish embryo heart videos
  50. Blood pressure measurement on the cheek
  51. Derivation of the respiratory rate from directly and indirectly measured respiratory signals using autocorrelation
  52. Left cardiac atrioventricular delay and inter-ventricular delay in cardiac resynchronization therapy responder and non-responder
  53. An automatic systolic peak detector of blood pressure waveforms using 4th order cumulants
  54. Real-time QRS detection using integrated variance for ECG gated cardiac MRI
  55. Preprocessing of unipolar signals acquired by a novel intracardiac mapping system
  56. In-vitro experiments to characterize ventricular electromechanics
  57. Continuous non-invasive monitoring of blood pressure in the operating room: a cuffless optical technology at the fingertip
  58. Application of microwave sensor technology in cardiovascular disease for plaque detection
  59. Artificial blood circulatory and special Ultrasound Doppler probes for detecting and sizing gaseous embolism
  60. Detection of microsleep events in a car driving simulation study using electrocardiographic features
  61. A method to determine the kink resistance of stents and stent delivery systems according to international standards
  62. Comparison of stented bifurcation and straight vessel 3D-simulation with a prior simulated velocity profile inlet
  63. Transient Euler-Lagrange/DEM simulation of stent thrombosis
  64. Automated control of the laser welding process of heart valve scaffolds
  65. Automation of a test bench for accessing the bendability of electrospun vascular grafts
  66. Influence of storage conditions on the release of growth factors in platelet-rich blood derivatives
  67. Cryopreservation of cells using defined serum-free cryoprotective agents
  68. New bioreactor vessel for tissue engineering of human nasal septal chondrocytes
  69. Determination of the membrane hydraulic permeability of MSCs
  70. Climate retainment in carbon dioxide incubators
  71. Multiple factors influencing OR ventilation system effectiveness
  72. Evaluation of an app-based stress protocol
  73. Medication process in Styrian hospitals
  74. Control tower to surgical theater
  75. Development of a skull phantom for the assessment of implant X-ray visibility
  76. Surgical navigation with QR codes
  77. Investigation of the pressure gradient of embolic protection devices
  78. Computer assistance in femoral derotation osteotomy: a bottom-up approach
  79. Automatic depth scanning system for 3D infrared thermography
  80. A service for monitoring the quality of intraoperative cone beam CT images
  81. Resectoscope with an easy to use twist mechanism for improved handling
  82. In vitro simulation of distribution processes following intramuscular injection
  83. Adjusting inkjet printhead parameters to deposit drugs into micro-sized reservoirs
  84. A flexible standalone system with integrated sensor feedback for multi-pad electrode FES of the hand
  85. Smart control for functional electrical stimulation with optimal pulse intensity
  86. Tactile display on the remaining hand for unilateral hand amputees
  87. Effects of sustained electrical stimulation on spasticity assessed by the pendulum test
  88. An improved tracking framework for ultrasound probe localization in image-guided radiosurgery
  89. Improvement of a subviral particle tracker by the use of a LAP-Kalman-algorithm
  90. Learning discriminative classification models for grading anal intraepithelial neoplasia
  91. Regularization of EIT reconstruction based on multi-scales wavelet transforms
  92. Assessing MRI susceptibility artefact through an indicator of image distortion
  93. EyeGuidance – a computer controlled system to guide eye movements
  94. A framework for feedback-based segmentation of 3D image stacks
  95. Doppler optical coherence tomography as a promising tool for detecting fluid in the human middle ear
  96. 3D Local in vivo Environment (LivE) imaging for single cell protein analysis of bone tissue
  97. Inside-Out access strategy using new trans-vascular catheter approach
  98. US/MRI fusion with new optical tracking and marker approach for interventional procedures inside the MRI suite
  99. Impact of different registration methods in MEG source analysis
  100. 3D segmentation of thyroid ultrasound images using active contours
  101. Designing a compact MRI motion phantom
  102. Cerebral cortex classification by conditional random fields applied to intraoperative thermal imaging
  103. Classification of indirect immunofluorescence images using thresholded local binary count features
  104. Analysis of muscle fatigue conditions using time-frequency images and GLCM features
  105. Numerical evaluation of image parameters of ETR-1
  106. Fabrication of a compliant phantom of the human aortic arch for use in Particle Image Velocimetry (PIV) experimentation
  107. Effect of the number of electrodes on the reconstructed lung shape in electrical impedance tomography
  108. Hardware dependencies of GPU-accelerated beamformer performances for microwave breast cancer detection
  109. Computer assisted assessment of progressing osteoradionecrosis of the jaw for clinical diagnosis and treatment
  110. Evaluation of reconstruction parameters of electrical impedance tomography on aorta detection during saline bolus injection
  111. Evaluation of open-source software for the lung segmentation
  112. Automatic determination of lung features of CF patients in CT scans
  113. Image analysis of self-organized multicellular patterns
  114. Effect of key parameters on synthesis of superparamagnetic nanoparticles (SPIONs)
  115. Radiopacity assessment of neurovascular implants
  116. Development of a desiccant based dielectric for monitoring humidity conditions in miniaturized hermetic implantable packages
  117. Development of an artifact-free aneurysm clip
  118. Enhancing the regeneration of bone defects by alkalizing the peri-implant zone – an in vitro approach
  119. Rapid prototyping of replica knee implants for in vitro testing
  120. Protecting ultra- and hyperhydrophilic implant surfaces in dry state from loss of wettability
  121. Advanced wettability analysis of implant surfaces
  122. Patient-specific hip prostheses designed by surgeons
  123. Plasma treatment on novel carbon fiber reinforced PEEK cages to enhance bioactivity
  124. Wear of a total intervertebral disc prosthesis
  125. Digital health and digital biomarkers – enabling value chains on health data
  126. Usability in the lifecycle of medical software development
  127. Influence of different test gases in a non-destructive 100% quality control system for medical devices
  128. Device development guided by user satisfaction survey on auricular vagus nerve stimulation
  129. Empirical assessment of the time course of innovation in biomedical engineering: first results of a comparative approach
  130. Effect of left atrial hypertrophy on P-wave morphology in a computational model
  131. Simulation of intracardiac electrograms around acute ablation lesions
  132. Parametrization of activation based cardiac electrophysiology models using bidomain model simulations
  133. Assessment of nasal resistance using computational fluid dynamics
  134. Resistance in a non-linear autoregressive model of pulmonary mechanics
  135. Inspiratory and expiratory elastance in a non-linear autoregressive model of pulmonary mechanics
  136. Determination of regional lung function in cystic fibrosis using electrical impedance tomography
  137. Development of parietal bone surrogates for parietal graft lift training
  138. Numerical simulation of mechanically stimulated bone remodelling
  139. Conversion of engineering stresses to Cauchy stresses in tensile and compression tests of thermoplastic polymers
  140. Numerical examinations of simplified spondylodesis models concerning energy absorption in magnetic resonance imaging
  141. Principle study on the signal connection at transabdominal fetal pulse oximetry
  142. Influence of Siluron® insertion on model drug distribution in the simulated vitreous body
  143. Evaluating different approaches to identify a three parameter gas exchange model
  144. Effects of fibrosis on the extracellular potential based on 3D reconstructions from histological sections of heart tissue
  145. From imaging to hemodynamics – how reconstruction kernels influence the blood flow predictions in intracranial aneurysms
  146. Flow optimised design of a novel point-of-care diagnostic device for the detection of disease specific biomarkers
  147. Improved FPGA controlled artificial vascular system for plethysmographic measurements
  148. Minimally spaced electrode positions for multi-functional chest sensors: ECG and respiratory signal estimation
  149. Automated detection of alveolar arches for nasoalveolar molding in cleft lip and palate treatment
  150. Control scheme selection in human-machine- interfaces by analysis of activity signals
  151. Event-based sampling for reducing communication load in realtime human motion analysis by wireless inertial sensor networks
  152. Automatic pairing of inertial sensors to lower limb segments – a plug-and-play approach
  153. Contactless respiratory monitoring system for magnetic resonance imaging applications using a laser range sensor
  154. Interactive monitoring system for visual respiratory biofeedback
  155. Development of a low-cost senor based aid for visually impaired people
  156. Patient assistive system for the shoulder joint
  157. A passive beating heart setup for interventional cardiology training
Downloaded on 12.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cdbme-2016-0114/html
Scroll to top button