Assessment of deep-learning-based resolution recovery algorithm relative to imaging system resolution and feature size

2.1.1 DeepNet

DeepScout leverages high-resolution 3D microscopy subvolume datasets to train a neural network model powered by the DeepNet architecture (part of the ZEISS Advanced Reconstruction Toolbox [ART]) [31], [32]. DeepNet utilizes a U-Net-like architecture for super-resolution image volume reconstruction taking 512 × 512 × 5 image tiles as an input, hence we sometimes refer to it as 2.5D network. The architecture comprises a contracting path and an expansive path with skip connections. The contracting path, consisting of convolutional and pooling layers, reduces spatial dimensions while increasing channels to capture features at different scales. With upsampling and convolutional layers, the expansive path increases spatial dimensions while reducing channels and recovering spatial information via skip connections. After the U-net part the architecture is supplemented by additional convolutional layers which further improves feature recovery. Final loss function L is defined as a weighted sum of squared errors calculated between the target and intermediate image output (after the U-net) ℓ ₁ and between the target and final output (after additional layers) ℓ ₂:

Loss function is also constrained to the reconstructable circular field of view defined by the cone-beam geometry. Since the training input and target images are originating from independent noise realizations, the network can also reduce noise in the output image. Note that the input low resolution image is reconstructed using FDK algorithm with the matching voxel size and field of view to that of the target high resolution volume (pre-upsampling).

Model training employs the Adam optimizer for 10,000 iterations with a learning rate of 2 × 10⁻⁴. Batch size was equal to 4. All available data was used for training, the typical split of the training data into training and validation sets was not performed, since from the prior experience we know that 10,000 iterations was sufficient for convergence.

During inference of the whole image volume typically at least 1,000³ voxels or more, the trained model processes the overlapping patches of the input volume to ensure smooth transitions, which are then reconstructed into a full 3D output using a weighted averaging scheme. 576 × 576 image tiles were merged with a 32-pixel overlap on each side, using a linear blending method with a triangular weight profile to ensure smooth transitions between the tiles. In the slice direction, the tiles were processed as stacks of 5 slices each and combined using an averaging method with a 1-slice stride, resulting in a 4-slice overlap.

All image reconstructions, model training and inferences in this paper were using dual RTX A6000 GPU workstation.

2.1.2 SRCNN

Super-resolution convolutional neural network (SRCNN), first introduced for 2D imaging, was a pioneering method that significantly advanced super-resolution methods for image restoration [11]. SRCNN enhances image resolution by learning to map from low-resolution to high-resolution images using convolutional neural networks. This approach addresses the limitations of traditional interpolation techniques by employing a data-driven approach, learning from a large dataset of low- and high-resolution image pairs.

The modified SRCNN employed in this study utilizes a fully 3D convolutional neural network with batch normalization layers for improved training stability and employs the Adam optimizer for more efficient optimization [10]. Its architecture mainly consists of three convolutional layers: Patch Extraction and Representation, Non-Linear Mapping, and Reconstruction. The training process involves preparing datasets and generating high- and low-resolution data polluted by noise. Mean Squared Error (MSE) serves as the loss function, and the network is trained using the Adam optimizer to minimize the loss. However, the network’s relatively shallow architecture limits its ability to learn complex mappings, potentially leading to suboptimal performance on intricate textures and fine details [33]. It can be quite sensitive to weight initialization, leading to training instability and potentially suboptimal results [34]. Moreover, it uses a fixed and relatively small receptive field size (e.g., 13 × 13 pixels in the original paper), limiting its ability to capture a larger image context, which is particularly problematic for larger upscaling factors [35].

For SRCNN training, the 3D image datasets were preprocessed into patches using the ‘create_patches’ function, with specified patch size and stride. The images were then split into training and validation sets. The training was implemented over multiple epochs by processing batches of input patches through the network, computing the loss, and updating weights via backpropagation. To monitor generalization, Tensorboard was utilized for training logs and losses. The model with the lowest validation loss was saved as the best model.

2.1.3 Richardson-Lucy

Richardson-Lucy deconvolution algorithm, a classical image restoration technique, was adopted for various imaging methods, including XRM [3], [5]. Based on Bayesian statistics, this iterative algorithm estimates the true object distribution, which, when convolved with an imaging system’s PSF, would produce the input blurred image [6]. For 3D imaging, the RL algorithm is applied to the volume data, with iterative refinements of the initial input [36]. This process continues until a convergence criterion or the defined number of iterations is met. However, this technique requires a careful selection of the parameters, knowledge of the system PSF, regularization, and stopping criteria to achieve optimal results. Additional details regarding the technique and the mathematical framework can be found elsewhere [37], [38]. RL deconvolution was applied to blurred (LRES) images with noise added at 3, 5, 7, and 9 µm resolutions. In each case the 3D resolution kernel inside RL deconvolution was equal to the blurring kernel used for data degradation, for 30 iterations.

2.5.1 Gaussian fit analysis

A Gaussian fit method was employed to provide a quantitative measure of spatial resolution loss when using different imaging setups. The higher-resolution image was repeatedly convolved with a 3D Gaussian kernel until the best least squares fit was identified between the convolved image and the lower-resolution volume. The FWHM of the Gaussian kernel that provided the best fit would then serve as the measure of spatial resolution difference. This method is insensitive to hallucinations and evaluates the spatial resolution of the images.

2.5.2 Multiscale-SSIM

Multiscale-SSIM (3D) index was used to generate quality maps for the processed images with respect to the ground truth phantoms as a reference, with an in-built MATLAB (Ver. R2024a) function (multissim3) [43]. The multiscale-SSIM index for the quality maps combines the SSIM index of several image versions at different scales (i.e., downscaled resolutions) to output a local value for each voxel within the 3D volume. Similar to SSIM, a value closer to 1 indicates a good match with the ground truth, while a value closer to 0 indicates poor quality. The clustered voxels with lower values, therefore, correspond to regions with a higher probability of hallucinatory features to be present.

3.1.1 Image quality

The feature recovery performance of Richardson-Lucy, SRCNN, and DeepNet algorithms was evaluated quantitatively by comparing them with the generated ground-truth foam phantom image. The Foam datasets shown in Figure 1 were used to train both DeepNet and the SRCNN models, which were then applied to the alternate realization of the Foam. For consistency, the quantitative metrics were compared between the results from the alternate realization of the Foam for all three algorithms.

Figure 3 presents a quantitative assessment of the image quality using three widely adopted metrics: mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). Along with unique insights into image quality, these metrics also provide an objective comparison of the images with respect to the ground truth, avoiding human subjectivity. However, it should be noted that they have their limitations. MSE and PSNR are insensitive to structural changes and, therefore, lack correlation with perceived image quality [44]. Lower MSE values and higher PSNR signify a closer representation of the ground truth in terms of pixel intensity distribution and signal strength to the noise, which might not be a complete representation of the actual image quality [45]. The SSIM instead aids in assessing the preservation of structural details, with higher image quality corresponding to values closer to 1.

Figure 3:

Image quality metrics (a) mean square error (MSE, lower is better), (b) peak signal-to-noise ratio (PSNR, higher is better), and (c) structural similarity index measure (SSIM, closer to 1 is better), of all the images analyzed in this study.

With the deterioration in spatial resolution, the image quality metrics worsened for all methods. This is expected because of the increasing resolution ratio between the ground truth and the input images. DeepNet consistently outperformed the other two methods across all metrics and resolutions. The lower MSE, higher SSIM, and high PSNR values suggest its effectiveness at pixel-level accuracy and overall structural preservation. The SRCNN followed slightly behind but with a steeper drop in image quality at lower resolutions, especially lacking in feature preservation. While SRCNN and DeepNet inference on the alternate foam realization for 3 µm, show comparable MSE and PSNR performance, the following results diverge as the shallow network of SRCNN cannot compensate for larger resolution losses. Furthermore, the deeper network architecture of DeepNet ensured a higher SSIM and, therefore, a lower probability of hallucinatory artifacts. Richardson-Lucy demonstrates reasonably good performance as a classical deconvolution method, at least according to these quantified metrics. The low SSIM (∼0.94), even for the 3 µm image, suggests that it fails to recover certain structural features, which is only exacerbated with increasing resolution ratios. In the end, both CNN algorithms show better recovery than RL, from the learned correlations between the LRES and HRES pairs during the training step.

A qualitative comparison of the central XY slice of the results for the alternate Foam realization with each algorithm, along with the noisy input images with varying degrees of image degradation, is shown in Figure 4. Visual assessment of the images provided additional information for evaluating these algorithms. The classical RL deconvolution method severely underperformed, with blurry images, lowered brightness, and reduced edge contrast. The smaller features become unrecoverable at lower resolutions (i.e., 7 and 9 µm), while the larger features were reduced in size due to edge blurring. Clusters of small pores were either lost or incorrectly recovered as fused irregular, larger-sized pores. Additionally, speckling artifacts were apparent in the bright phase despite good noise reduction in the darker regions.

Figure 4:

A qualitative comparison (slice # 256/512) of the resolution recovery performance with DeepNet, SRCNN and Richarson-Lucy deconvolution models with the degrading image quality of the alternate foam realization. The input images are shown in the top row. Note: Refer to Figure 8 for the ground truth of alternate dataset.

Although the image quality metrics were suboptimal for SRCNN (Figure 3), visual inspection reveals a good improvement over RL deconvolution, especially for 3 and 5 µm images. On closer examination, these images had certain textural artifacts in both dark and bright phases of Foam, confirming these small differences at a pixel level largely impacted the image quality metrics. Furthermore, SRCNN was ineffective in preserving smaller pores at lower resolutions (i.e., 7 and 9 µm), while the larger pores were mostly preserved. Similar to RL, clusters of small pores were recovered as merged irregular pores. The 9 µm image exhibited a severe recovery loss, with distinct textural artifacts near the edges of larger pores and the bulk phases. These artifacts are most likely due to the image hallucinations during the model inference step, exacerbated by larger resolution ratios.

DeepNet showed excellent recovery across all four resolution scales along with edge preservation. Similar to SRCNN, it inherently denoises the images during training since the training pair contains two independent noise realizations but results in very few textural artifacts, even for the recovered 9 µm image. However, some visual discrepancies were observed for the 9–1 µm images, particularly for the smaller pore sizes. These are most likely due to the large resolution gap between LRES-HRES training pairs, potentially resulting in image hallucinations during inference.

3.1.2 Quantitative feature evaluation

To quantitatively assess the performance of the algorithms in recovering the structural details, the foam images were analyzed as porous materials, enabling the comparison of porosity and pore size distributions with the ground truth. The same automated image processing workflow, initially optimized for the ground truth images, was applied to all the results. It allowed a comparison of the results from each recovered image to the ground truth, which serves as baseline. The quality of the results could have a minor effect on the segmentation and the separation of pores that are closer to each other, due to noise, lack of contrast or blurred edges.

The pore recovery performance, presented as the percentage of pores identified relative to ground truth (1,380 pores for both realizations) with automated segmentation, is shown in Figure 5. DeepNet recovered over 95 % of the pores detected for low resolutions (3, 5, and 7 µm) when applied to both the training and the alternate datasets (Figure 5). Even with a high-resolution ratio at 9 µm, it recovered over 85 % of the pores, demonstrating strong recoverability performance. In contrast, the SRCNN model performed well for the 3 and 5 µm images but showed a linear decline in recovery for reducing resolutions, to only 60 % at 9 µm. Richardson-Lucy method exhibited the poorest performance with suboptimal recovery even for the highest resolution (3 µm) image, followed by a sharp decrease at lower resolutions.

Figure 5:

The percent pore recovery performance of the algorithms evaluated in this study, with respect to the ground truth. The baseline count for the ground truth was calculated to be 1,380 pores, which was used as a reference.

While pore recovery performance shows the effect of the recovered image quality on quantification, further analysis was carried out to study recovery of pores based on their total volume, sizes and shape. Figure 6 shows a quantitative comparison of porosity and pore size distributions between the processed images and the ground truth. The porosities, calculated as the ratio of pore volume to the foam’s bulk volume mask, are indicated within the label boxes. The pore size distributions are plotted as histograms with a 5 µm bin width, where the spherical equivalent diameter of the volume of each segmented pore represents the sizes.

Figure 6:

Quantitative comparison of the porosity and the pore size distributions of the foam datasets with the ground truth as a reference. The porosities (ϕ) are highlighted in the label boxes. Note: Panel (a) shows results of DeepNet model on the training data, while (b), (c) and (d) show the results of DeepNet, SRCNN, RL for the alternate foam realization.

Although the total pore recovery with the RL algorithm was below 90 % for the 3 µm image, the porosity estimate was higher than that of the ground truth (Figure 6d). This was primarily due to the insufficient edge contrast affecting the segmentation of large pores (diameters greater than 25 µm), resulting in larger diameters than those from the ground truth. In contrast, the lower pore count was mainly due to the loss of small pores with diameters below 20 µm. The porosities and the recovered pores declined sharply, with a decrease in the spatial resolution of the input images.

The SRCNN results for the 3 µm image exhibited an excellent match with the ground truth both for pore recovery and size distributions. The porosity estimates for all resolutions were within 10 % of the actual value, mainly due to the recovery of large pores (Figure 6c). However, lower resolutions negatively affected the smaller pores (diameters below 25 µm).

DeepNet shows superior pore recovery across multiple resolutions (Figure 6a and b), with recovered porosities for both Foam realizations within 6 % of the ground truth baseline. The pore size distributions indicate that the trained models accurately recover the medium to large-sized pores (i.e., between 25 and 70 µm), while the losses were primarily from the smaller pores. Additionally, the results from the inference images indicate that pore recovery of smaller pores was better than that on the training images. This is most likely the reason for the slightly higher percent pore recovery with the application of trained models on the alternate Foam realization shown in Figure 5. Notably, the remarkable recovery of pores larger than 60 µm for all images highlights the ability of DeepNet to recover large features accurately, provided the loss in spatial resolution does not overshadow the feature sizes.

While the pore size distributions show how the models recover the segmented pores, they fail to identify how accurately the features are recovered. This is especially important for lower spatial resolution images, where some of the smaller pores have sizes comparable to the standard deviation of the applied Gaussian blurring function. For low-resolution data, clusters of tiny pores could be inferred as voids with irregular shapes, which cannot be identified with equivalent diameters. To address this, the sphericity parameter was analyzed, with values ranging between 0 and 1, which defines how closely the shape of a feature resembles a perfect sphere (with 1 representing a perfect sphere):

where V and A are the volume and boundary surface area of the segmented pore.

The average sphericity of the pores for the ground truth Foam images was found to be 0.990. While smaller pores have a pixelated irregular surface area, resulting in values slightly lower than 1, the majority of the segmented pores were within an acceptable range of 0.975 and 1 (refer to Figure 7). Due to stochasticity in the pore shapes and sizes, the surface area estimates for the segmented pores, although accurate, might not be precise. In our study, this resulted in sphericity values greater than 1 (in the third decimal), which were treated as outliers and rounded to 1.

Figure 7:

The box plots of the sphericity parameter as a metric to evaluate the accuracy of the recovered features compared to the ground truth. The horizontal line within the box represents the mean (also shown with a numeric label), and the whiskers represent the standard deviation. The data points are shown as an overlay for visual reference. Note: Panel (a) shows results of DeepNet model on the training data, while (b), (c) and (d) show the results of DeepNet, SRCNN, RL for the alternate foam realization.

The sphericities of the recovered pores with different models are presented as box plots in Figure 7. The average sphericities with RL deviated considerably from the ground truth, even for the 3 µm image (Figure 7d). For lower resolutions, the averages decreased further before increasing again. This is primarily due to the inability of the RL method to recover most pores (as can be observed visually in Figure 4), while the increased sphericity for the 7 and 9 µm images was due to the large, rounded pores being recovered, skewing the averages.

The SRCNN models fared better for the 3 and 5 µm images, with the averages closer to the ground truth. However, lowered average sphericities and larger standard deviations were observed for the 7 and 9 µm images. Although SRCNN recovered up to 60 percent of the total pores, the lowered sphericity indicates that most were poorly defined. This is most likely due to the hallucinatory effects that are exacerbated by noise and low spatial resolution.

The DeepNet models exhibit a remarkable recovery in pore shapes for resolution ratios up to 7:1. Notably, the average sphericities for all images were within 5 % of the ground truth. Although most data points were clustered closer to 1, deviations as low as 0.75 were found for the 9 µm images. Further investigation revealed this was primarily due to hallucinatory artifacts propagating into the image during model inference.

3.1.3 Image artifacts and spatial resolution

The comparative study utilizing the simulated foam phantoms demonstrates that DeepNet excels in recovering both feature size and shape while inherently denoising the images. The selection of the HRES-LRES pairs and the magnitude of the resolution gap impacted the outcome. However, results support that excellent and reliable results are achievable, even with spatial resolution ratios of up to 7:1 between the input training images.

In real-world scenarios, image quality is influenced by a complex interplay of acquisition parameters such as source voltage (impacting contrast and beam hardening), number of projections and their exposure time (affecting the signal-to-noise), and voxel size (affecting the minimum resolvable feature size). Additional losses may arise from mechanical limitations of the instrument itself. Therefore, the loss in spatial resolution in the acquired image is due to a cumulative point spread function that is a combination of various effects that is challenging to address with conventional image restoration methods such as the Richardson-Lucy iterative deconvolution.

The complexity of this problem naturally pivots towards the reliance on advanced deep learning-based convolutional neural network approaches. While SRCNN shows excellent feature recovery with small resolution differences between the HRES and LRES images, DeepNet leverages its deeper network architecture and a much larger number of trained parameters. It recovers image sharpness over wider resolution differences and retains the features within the phantom. This result aligns well with the findings of the previous work [46], [47].

Multiscale-SSIM (MS-SSIM) quality maps were generated for the 3 and 9 µm images of the alternate foam realization (Figure 8) to provide a spatially resolved assessment of localized image fidelity relative to ground truth. In these maps, darker regions correspond to higher MS-SSIM values (close to 1), indicating strong structural similarity, while the brighter areas signify lower values, highlighting regions of potential structural dissimilarity and possible hallucination artifacts.

Figure 8:

A comparison of the multiscale-SSIM quality maps of the 3 and 9 µm results for the alternate foam realization. The brighter hotspots represent the pixels with low SSIM values, highlighting the uncertainty of the recovered structural features. Note: The SSIM scale for the colormap varies for each quality map.

The DeepNet reconstructions demonstrated high fidelity, as evidenced by the predominantly dark MS-SSIM maps for both 3 µm and 9 µm resolutions. In the 3 µm image, subtle brighter regions are primarily observed at pore boundaries, likely attributed to minor discrepancies in edge sharpness. However, in the 9 µm image, more pronounced bright hotspots emerge, particularly in areas with smaller pores. These localized reductions in MS-SSIM suggest either the presence of spurious pores (hallucinations) or the absence of actual pores in the reconstruction, leading to structural mismatches with the ground truth.

In contrast, the RL and SRCNN reconstructions exhibit more extensive regions of brightness in their MS-SSIM maps, especially at 9 µm resolution. This indicates widespread structural dissimilarity and a higher prevalence of artifacts or distortions. Among the CNN algorithms, the SRCNN images show regions of extreme brightness, suggesting a greater probability of generating hallucinatory features. Textural inconsistencies (shown in Figure 4) present for both SRCNN and RL further support this observation.

Figure 9 shows the results for the Gaussian fit analysis for the alternate Foam images, highlighting the mean square error (MSE) when compared to the ground truth image convolved with increasing Gaussian blur kernel size from 0 to 10. The calculated MSE quantifies the discrepancy between the blurred ground truth image and the images processed using different algorithms. A higher error at lower kernel sizes arises from the high-frequency features or hallucinations present in the image compared to the blurred ground truth. At higher kernel sizes, the image becomes dominated by over-blurred features, resulting in a significantly larger MSE. Generally, the Gaussian kernel size corresponding to the lowest error for each curve can be considered the effective spatial resolution of the recovered images.

Figure 9:

A comparison of the effective spatial resolution of results from each algorithm on the alternate foam realization, with the Gaussian fit analysis. The lowest point of error for each curve (highlighted with droplines) corresponds to the Gaussian kernel size of the convolved ground truth image, with the best match.

These findings suggest that CNN based algorithms show large improvements in the effective spatial resolution compared to classical methods. While RL shows a reasonably good outcome for the 3 µm image, it fails for images with more severe image quality degradation. Among the CNN models, DeepNet outperforms significantly, with the effective spatial resolution of its images closely matching the ground truth. The SRCNN models perform well at lower degradation levels but exhibit a faster decline with increasing image quality loss. With deeper neural network architectures like DeepNet, it is possible to recover image resolution that closely matches the ground truth, even with severe degradation.

The multi-faceted image analysis conducted on the simulated foam phantoms comprehensively evaluated the performance of each algorithm from different perspectives. While DeepNet demonstrates remarkable feature recovery capabilities, as evidenced by the quantitative analysis, the MS-SSIM quality maps reveal the potential for hallucinations, particularly at higher resolution ratios, which can compromise data reliability. However, within the recommended resolution limits (up to 7:1, depending on the sample’s fine features), DeepNet exhibits exceptional performance. It is crucial to acknowledge that imaging artifacts, such as photon starvation or phase artifacts, can negatively impact the training process. These artifacts may be misinterpreted as genuine features during training, leading to erroneous propagation into processed images during inference.