Learning flat optics for extended depth of field microscopy imaging

1 Introduction

A conventional imaging system can produce sharp images for objects within the depth of field (DOF), which is the range around the focused depth of the scene. The DOF coverage is inversely proportional to the numerical aperture (NA), i.e., a smaller NA leads to a larger DOF. Although specific applications may require smaller DOFs, a larger DOF is often preferable to obtain sharp images of objects at varying depths. In particular, microscopy requires high NA objectives to capture precise details surrounding the focused depth of an object. However, due to the shallow DOF of high NA imaging systems, microscopic imaging systems often use depth scanning techniques to cover the entire depth range of interest, typically much larger than the imaging system DOF [1]. Depth scanning techniques are often insufficient due to the light scattering from objects outside the intended image plane, resulting in artifacts or severe noise. Acquiring cross-sectional data using this approach requires sweeping a focused point across the entire sample, which inherently imposes temporal limitations on the frame rate and prevents snapshot acquisition. To address these challenges, various techniques have been developed, such as decoupled illumination and detection in light-sheet microscopy [2], dynamic remote focusing [3, 4], and spatial and spectral multiplexing [5, 6]. Despite their potential advantages, these methods often necessitate a specialized and intricate optical configuration, which may render them both costly and difficult to implement in microscopy applications. In addition, computational approaches such as Fourier ptychographic microscopy demonstrated extended depth of field (EDOF) imaging capabilities [7]. However, this method relies on image reconstruction that assumes a thin sample illuminated by oblique plane waves, rendering it unsuitable for clinical fluorescence imaging applications.

The integration of wavefront encoding with computational reconstruction methods offers a cost-effective and efficient approach to improve EDOF imaging performance. Specifically, computational reconstruction methods, such as deep learning-based approaches and iterative optimization algorithms, have demonstrated substantial advancements in enhancing the EDOF imaging performance by effectively mitigating defocus blur, noise, and artifacts [8–12]. These methods leverage the optimization of both optical components and the reconstruction algorithm, enabling the system to enhance the EDOF performance effectively. End-to-end optimization frameworks, which jointly optimize the optical design and the associated reconstruction algorithms, have emerged as a powerful tool for addressing the EDOF challenge. This approach allows the system to learn and adapt to specific imaging requirements, thereby improving overall performance and enabling better control over trade-offs in resolution, noise, and depth range [8]. Wavefront coding involves the use of a phase element, such as a diffractive optical element (DOE) or a free-form refractive lens, placed at the aperture plane [13–19]. The main objective of EDOF wavefront coding is to achieve a depth-invariant point spread function (PSF) while preserving information at all spatial frequencies. The research presented in [10] employs DOEs to realize EDOF microscopy across 200 μm DOF range, a methodology closely aligned with our study. However, DOE wavefront coding displays restricted EDOF capabilities, a limitation that may be attributed to constraints within the space-bandwidth product (SBP). This factor determines the information content captured by the imaging system. A detailed comparison with such method, called DeepDOF, is elaborated in Section 3.

In comparison, metasurfaces, which are ultra-thin meta-optics composed of subwavelength nano-antennas, offer increased design flexibility and a superior SBP compared to DOEs [20, 21]. These structures facilitate precise control over phase, amplitude, and polarization of light at the nanoscale, allowing almost arbitrary modification of the complex optical functions on a thin, planar device. The advantages of metasurfaces can be attributed to the rich modal characteristics of meta-optical scatterers, enabling multifunctional capabilities beyond traditional DOEs, encompassing polarization, frequency, and angle multiplexing [22–25]. Consequently, metasurfaces exhibit a greater potential for addressing the EDOF microscopy challenge more effectively than conventional DOEs, with researchers having already leveraged their benefits in various applications such as flat optics for imaging [26, 27], polarization control [28], and holography [29]. Despite the promising potential of meta-optics, current metasurface imaging methodologies demonstrate limited EDOF imaging capabilities. The EDOF range for any given imaging system can be quantified using the defocus coefficient, wherein a larger EDOF corresponds to a higher defocus coefficient. Details regarding these defocus coefficients and their relationship with the depth of field range and optical parameters are comprehensively covered in Section 2.1. Currently, most methods are designed to accommodate systems with a maximum defocus coefficient limited to around 75, with some using mechanical displacement as a strategy to extend imaging capability [30–32]. Although the narrow defocus ranges may be adequate for certain applications, it is comparatively limited in broader scientific and industrial contexts. As such, there remains a need for more flexible and versatile imaging solutions that can accommodate a broader depth range without sacrificing image quality. In contrast, the study in [33], addresses a problem characterized by a maximum defocus coefficient comparable to ours, valued at around 245. However, the demonstrated image quality is considerably lower compared to the results achieved in our research. It is worth noting that other metasurface-based imaging implementations exist, which are capable of encoding spatial, spectral, and polarization information while maintaining satisfactory imaging performance [34, 35]. Moreover, computational metasurface designs that yield a large field-of-view for full-color metasurface operation, without significant degradation of imaging performance, have been reported [21].

In this study, we propose an end-to-end optimization framework designed for acquiring high-resolution images across an extensive DOF range within a microscopy system. The optics and post-processing algorithm are modeled as parts of the end-to-end differentiable computational image acquisition system, allowing for simultaneously optimizing both components. Our computational EDOF microscope employs a hybrid approach that combines a 4f microscopy optical setup with a learned wavefront modulating optical element at the Fourier plane. We explore metasurfaces and (conventional) DOEs for implementing such novel design modulation. The encoded image acquired at the sensor is post-processed by a convolutional neural network (CNN) that implements deblurring to achieve an EDOF image of the specimen. The optimization procedure involves tuning of critical parameters within two primary components of the system: the optical component, characterized by the phase modulation function, and the image reconstruction component, realized through the deblurring convolutional neural network (D-CNN). This process is facilitated by an end-to-end learning methodology that utilizes a dataset of sharp images to steer the optimization. This methodology emphasizes the importance of carefully outlining sampling requirements for various depth-of-field targets in order to achieve optimal imaging results. As a result, our systematic design methodology significantly outperforms existing state-of-the-art EDOF imaging techniques in terms of image quality and depth range.

2 Methods

The EDOF microscopy problem addresses the difficulty of capturing high-resolution images across an extensive DOF range within a microscopy system. The objective of resolving this problem is to improve the imaging performance of the system by expanding its DOF, allowing the acquisition of sharp images over a larger depth range without necessitating mechanical focus adjustments. Figure 1 presents the proposed 4f system designed to address the EDOF microscopy problem, which consists of an optical module, a sensor, and a subsequent deblurring module for post-processing. The optical module employs a 4f imaging configuration that incorporates a phase coding mask at the aperture position. This mask, an optical element crafted to modulate incident light, introduces phase shifts across various regions through a transparent material with spatially varying geometry. The purpose of using a phase coding mask is to manipulate the optical wavefront, enabling a specific PSF or other desired properties within the optical system. In addressing this problem, the 4f system components remain fixed while optimizing the EDOF microscopy imaging system by learning the spatial distribution of the phase coding mask and the D-CNN weights for the targeted DOF ranges. Consequently, the objective is to achieve a defocus-invariant PSF within the optical system, resulting in improved EDOF microscopy imaging performance.

Figure 1:

A 4f system computational EDOF microscope model that combines the optical module, sensor, and D-CNN.

Figure 2 depicts the proposed end-to-end learning procedure for the EDOF microscopy problem. This process accepts two inputs as high-resolution image patches and the predetermined depth range. The architecture of the framework consists of two main components: an optical layer and a D-CNN layer. During the training phase, the sensor image is generated through the optical layer, which involves simulating the image acquisition process based on depth value and the input image. The resulting sensor image is then fed into the D-CNN, which estimates the sharp, deblurred image as its output. The backpropagation algorithm is used to update the spatial phase distribution of the phase coding mask and the weights of the D-CNN at each iteration of the learning procedure, thus optimizing the parameters of the EDOF microscope in an end-to-end manner. The optimized phase coding mask is realized as a meta-optic element (metalens) or DOE, depending on the spatial phase distribution and sampling. After training is completed for the targeted DOF range, the defocus-invariant optical element and D-CNN model are obtained in the 4f system computational EDOF microscope model (Figure 2). Further details regarding the optical and D-CNN layers are provided below.

Figure 2:

Overall representation of the end-to-end learning framework for joint optimization of the phase coding mask and the D-CNN. The end-to-end framework consists of an optical layer that simulates image formation for the learned optics, and the deblurring layer employs U-Net architecture to reconstruct in-focus images within the desired DOF range. The optical element can be realized via a metalens, parametrized by the radius map, or a DOE, parametrized by the height map. The choice of representation relies upon the maximum Δs calculated for the intended depth range.

2.1 Optical layer

The optical layer is a computational module that operates based on the principles of wave optics and performs sensor image formation. In the following, we discuss sensor image formation by employing wave optics in particular and characterize the parameters of the EDOF microscope for the targeted DOF range for a 4f shift-invariant imaging system as illustrated in Figure 1. Here, the specimen is illuminated by a monochromatic, spatially incoherent light source. The spatial coordinates in the Fourier and sensor planes are denoted as (x ₁, y ₁) and (x ₂, y ₂), respectively. A transparent biological specimen in the system can be represented as a stack of 2D images corresponding to a fixed scene depth where a slice of such a stack corresponds to I _z (x ₂, y ₂) at the scene depth z. The contribution of such a slice to the sensor image, I z s ( x 2 , y 2 ) , is determined by convolution with the depth-dependent PSF, h _z (x ₂, y ₂):

(1) I z s ( x 2 , y 2 ) = I z ( x 2 , y 2 )   *   h z ( x 2 , y 2 ) .

The final image captured at the sensor, I ^s (x ₂, y ₂), is then determined by integrating I z s ( x 2 , y 2 ) over all depth values possible in the scene, considering

(2) I s ( x 2 , y 2 ) = ∫ I z s ( x 2 , y 2 ) d z + η s ,

where η _s is a sensor noise. In our simulations, we consider the noise as a zero-mean Gaussian model with η s ≈ N 0 , σ s 2 , where σ _s represents the standard deviation of the Gaussian noise. It is important to note that while we have assumed Gaussian noise for our experiments, the proposed method can be easily adapted to handle other noise models, such as Poisson distributed noise, by adjusting the noise assumption within the optimization process. Considering Eqs. 1 and 2, the recovery of a sharp image directly depends on h _z (x ₂, y ₂). The PSF on the sensor plane can be modeled using Fourier optics as the square of the Fourier transform of the generalized pupil function:

(3) h z ( x 2 , y 2 ) = | F { P ( x 1 , y 1 ) } | 2 ,

where F{.} denotes the Fourier transform operator, and P(x ₁, y ₁) is the pupil function, which describes the relative amplitude and phase changes of the wavefront at the Fourier plane:

(4) P ( x 1 , y 1 ) = A ( x 1 , y 1 ) e i ϕ ( x 1 , y 1 ) .

The pupil can be modulated using a phase coding mask introducing a phase term ϕ ^M (x ₁, y ₁) to enhance the defocus invariance of the PSF in the targeted DOF range. The resultant phase term becomes:

(5) ϕ z ( x 1 , y 1 ) = ϕ z D F ( x 1 , y 1 ) + ϕ M ( x 1 , y 1 ) .

The defocus aberration due to the mismatch between in-focus depth z ₀ and the actual depth z of a scene point is

(6) ϕ z D F ( x 1 , y 1 ) = ψ z x 1 2 + y 1 2 r 2 ,

where ψ z = π λ 1 z − 1 z 0 r 2 is the defocus coefficient and r is the radius of the pupil. To represent all spatial frequencies k _x , k _y supported by the objective lens on the pupil plane, the pupil size must be chosen using

(7) r ≥ f 1 k k x 2 + k y 2 ,

where f ₁ is the focal length of the tube lens, and k is the wave number. The frequency support of the PSF, in particular, determines the reconstruction quality. To avoid aliasing due to undersampling of the defocused pupil function, which would otherwise result in a miscalculated PSF, the minimum sampling rate of the simulated phase coding mask must be determined. The sampling rate Δs for a given depth range should satisfy [11]

(8) Δ s ≤ r π 8 ψ max ,

where ψ max = max | ψ z − | , | ψ z + | , the maximum defocus value within the scene. At the same time, we use such a relation as a guide in choosing the search space for the phase modulation function of the optical element, matching its degree of freedom with the complexity of the EDOF imaging problem at hand. This is observed to have a significant help in the end-to-end learning process, i.e., its convergence to the optimal solution. The optimized optical element can be realized as a metalens or DOE according to the calculated maximum Δs for the targeted depth range. The metalens and DOE are parameterized by a radius map and a height map, respectively, details of which are provided in the following sections.

During training, the optical element is optimized to minimize the impact of the defocus phase and ensure depth-invariant PSFs through the utilization of learned phase coding mask elements. For each iteration within the forward pass, this optimization is achieved by calculating Eq. (3) and optimizing the ϕ ^M (x ₁, y ₁) phase modulation term. For both the DOE and the metalens, the amplitude A(x ₁, y ₁) of the pupil function is kept constant within the aperture diameter and is modeled as a circular function. The camera parameters f ₁, r, and Δs are predefined as optical parameters. r and Δs is calculated for the selected objective lens and targeted depth range using Eqs. (7) and (8), respectively.

For the DOE design, the phase modulation can be controlled through unit cell height. Specifically, the phase shift is given by the equation:

(9) ϕ M ( x 1 , y 1 ) = k ( n − 1 ) h ( x 1 , y 1 ) ,

where n denotes the refractive index of the DOE material which is 1.5, specifically at the design wavelength of 550 nm within the green spectrum. The DOE is assumed to be lossless; therefore the amplitude is kept fixed as the circular function within the aperture diameter.

For the metalens design, phase accumulation is achieved through the waveguiding effect [36], whereby the height of the nanopillars is selected to provide 2π phase coverage across a range of radii. While the smallest possible diameter is primarily limited by fabrication constraints, the largest diameter is 50 nm smaller than Δs, as set by Eq. (8). To ensure high efficiency, the nanopillar height Δh is optimized at the 550 nm design wavelength. The phase and transmission responses are simulated via a finite-difference time-domain (FDTD) analysis, which involves varying the radius of the gallium nitride (GaN) nanopillar on a sapphire glass substrate. As depicted in Figure 3(a), the analysis results in full phase coverage (0–2π) with high transmission (overall greater than 83 %) for 550 nm.

Figure 3:

Design and simulation of EDOF imaging metalenses. The metalenses are made up of GaN nanopillars with a refractive index of 2.42, where the thickness (Δh), sampling (Δs), and radius (Δr) are the design parameters. (a) Simulation of the nanopillars’ transmission amplitude and phase response via FDTD. (b) Illustration of the full metalens design, transforming the spatial phase profile into a spatial radius (of nanopillar) distribution with respect to the target phase.

Irrespective of the desired target phase, designing a metalens involves converting a spatial phase profile into a corresponding spatial radius distribution. A resulting ϕ ^M (x ₁, y ₁) phase profile, generated from the end-to-end framework, is illustrated in Figure 3(b). This profile is then translated into the full metalens, taking into account the simulated phase and radius response. This approach enables the design of highly precise and effective metalens that meets the desired phase requirements.

3 Results

Our design is aimed to mitigate the trade-off between DOF and spatial resolution for varying target DOFs by experimenting with different objective lenses within the 4f imaging setup, as illustrated in Figure 1. This trade-off is mathematically expressed as follows:

This demonstrates that the spatial resolution is higher for a shallower DOF range; therefore, a shallower DOF range optimization problem is comparatively less challenging to solve. The complexity of the problem, however, is affiliated with the selected lenses and the desired DOF range. It is quantified and expressed using the defocus coefficient (ψ _z ) and presented alongside the system parameters for different simulations in Table 1.

Table 1 demonstrates that solving the EDOF problem for a higher NA lens system is considerably more challenging task, where the sampling requirements, determined by Eq. (8), become more demanding due to the broader bandwidth of the defocused pupil function, necessitating finer sampling for accurate results. Such a case, thus, advocates the metalens to address the underlying stricter sampling requirement. Conversely, for the same EDOF range, using a low NA lens-based system with a learned DOE can yield a satisfactory solution. However, it should be noted that the final magnification would be significantly lower in such a system. Therefore, the lens selection should be based on the requirements of the sample to be visualized. To the best of our knowledge, certain biological specimens, such as the developing embryo, exhibit a diameter that increases from approximately 200 μm to nearly 3 mm [41]. As such, we have selected to target DOF ranges of 200 μm, 2 mm, and 3 mm for our study.

To further support the results, we compare the proposed algorithm with two existing methods. Initially, a cubic mask is adopted as the conventional method for DOF extension [13], which modulates the phase as

where α determines the number of 2π transitions. In this case, a fixed cubic phase mask is combined with a U-Net and subsequently trained to perform deblurring for all EDOF targets. To select an appropriate mask for each DOF range, the modulation transfer function (MTF) is evaluated across five different depths by tuning α, to be insensitive to defocus and represent all spatial frequencies. α values of 6π, 200π, and 360π are selected for the DOF ranges of 200 μm, 2 mm, and 3 mm, respectively.

Additionally, the DeepDOF algorithm proposed by Jin et al. [10] is adopted as a more recent advanced EDOF method based on the end-to-end learning framework and Zernike basis (Z _n ) representation. Notably, the paper does not provide a relationship between the choice of the number of Zernike polynomials and the EDOF range. To align the optical setup parameters with our methodology and ascertain the appropriate number of parameters (n) for each depth range, we calculated the Fourier transform of the Z _n and found the bandwidth of the resultant signal. Considering the Nyquist theorem, we calculated the sampling rate for the Z _n . Based on these calculations, we selected the first 11, 1100, and 2000 Z _n for DOF targets of 200 μm, 2 mm, and 3 mm, respectively.

We present a quantitative analysis of the proposed method in comparison with existing approaches by analyzing the performance metrics, specifically peak signal-to-noise ratios (PSNRs) and structural similarities (SSIMs). For each test setup, the PSNR and SSIM values are derived and depicted in Figure 4. Additionally, the mean PSNR and SSIM values are calculated for all images within the 150 test images of the dataset and presented in Table 2. To calculate the average values, we assign each test image to predetermined depths, uniformly distributed throughout the scene depth range. Subsequently, the PSNR and SSIM values are averaged.

Figure 4:

Simulated performance of a cubic-mask-enabled computational 4f system, DeepDOF model, computational EDOF microscope, optimized for various target depths. For all DOFs, our method shows superior performance.

As demonstrated in Table 2, the proposed method outperforms existing approaches in terms of PSNR and SSIM values, both for the sample image shown in Figure 4 and the mean values across the entire test dataset. In particular, the D-CNN alone, as inferred from the fixed Cubic mask and U-Net simulations, is inadequate. Upon increasing the target DOF, the image becomes excessively blurry at the sensor level, and the U-Net cannot recover the images effectively. Comparable results are visible in the DeepDOF approach, which is linked to the sub-sampling of the frequency domain that arises from using the Zernike basis representation. It is important to acknowledge that [10] is optimized using 55 Zernike basis, for a target DOF of 200 μm. The starred result in Table 2 displays the outcomes obtained when employing 55 Zernike basis. In this case, we retain the same setup parameters as in our other simulations, only modifying the number of Zernike basis utilized to represent the height map in the method, resulting in a higher sampling than necessary. The results demonstrate that increasing the sampling does not improve the performance, thus confirming that our calculated setup parameters sufficiently address the system complexity. Additionally, in our U-Net implementation, different from [10], we adapted a modified U-Net architecture for the deblurring layer. The main difference resides in the final layer of our U-Net architecture, where we incorporated the residual image and applied the clamp function, enhancing both the accuracy and efficiency of the deblurring process.

An alternative method for evaluating EDOF performance involves examining the characteristics of MTFs across various depths, with MTFs representing the magnitudes of the Fourier transforms corresponding to their associated PSFs [12]. To facilitate efficient depth-agnostic deblurring throughout the whole depth range of interest, MTF pass-bands should be as wide as possible to recover features at various spatial frequencies while maintaining a high degree of similarity among themselves. Figure 5 presents the final metasurface designs for the 3 mm DOF scenario, as well as the MTFs at three distinct depths. The depth dependency of the MTFs in the proposed method is decreased compared to both the DeepDOF and Fixed cubic cases. The MTFs of the proposed method display greater similarity among themselves and avoid crossing zero, ensuring the preservation of information during recovery. This similarity is attributed to depth-agnostic deblurring assisting MTFs to remain consistent across all targeted depths. To compare the frequency support of each method, we defined a threshold of 0.1 and examined the MTF magnitude across the frequency spectrum and depths, which formally determines the maximum spatial resolution transmitted through the optics. The MTFs of the proposed method resulted in broadband coverage across the entire spectrum, illustrating that even at higher spatial frequencies, MTFs do not approach zero. This finding corresponds to the clear visibility of small structures. Conversely, at lower spatial frequencies, the MTF is closer to 1, representing the ability to clearly visualize large structures. Indeed, as demonstrated in the other methods, all MTFs can clearly visualize large structures, but for smaller structures, quality is inadequate.

Figure 5:

The optimized metalens radius maps, as well as the MTFs at three distinct depths, are presented for the existing and proposed methods, encompassing the maximum defocus values within the 3 mm DOF scene and the in-focus depth.

3.1 Fabrication error analysis

The robustness of the proposed EDOF microscope model is tested considering various levels of inaccuracies for the possible fabrication errors of metasurface and DOE. In particular, the effects of such fabrication inaccuracies are modeled by introducing random Gaussian-distributed noise with a standard deviation σ _r , to the optimized optical elements, as the radius and DOE height error during the test stage. The tested metasurface radius deviation levels are σ _r = 5 nm, 12 nm, while the DOE height deviation is assumed as σ _h = 30 nm, 50 nm. Figure 6 presents the outcomes for the various EDOF configurations. As it can be inferred from the figure, the post-processing method performs adequately for lower fabrication error levels but exhibits limited robustness in the presence of higher error margins, leading to a noticeable reduction in image quality, particularly at the boundaries of the targeted EDOF. The robustness of the algorithm can be increased by incorporating fabrication error boundaries during training or retraining the D-CNN following the fabrication of optical components. Additionally, the implementation of advanced denoising algorithms can contribute to the enhancement of the results.

Figure 6:

Reconstruction results for increasing radius and height-map fabrication error levels.

4 Future work and conclusion

In conclusion, we implemented an end-to-end optimization framework that effectively addresses the physical limitations inherent in the 4f system EDOF microscope. Our approach offers two primary advantages over existing methods. Firstly, the incorporation of metasurface optics within our system has facilitated the achievement of the most extensive EDOF range reported in the literature thus far. In extreme EDOF scenarios, specifically when employing a higher NA lens, meta-optics offer a distinct advantage due to their increased design flexibility and superior SBP. Secondly, our method effectively minimizes defocus by systematically incorporating relevant optical system parameters throughout the optimization process. The results obtained from our simulations demonstrate that our proposed technique outperforms the current state-of-the-art methods in EDOF microscopy imaging, delivering consistently high performance across a broad range of EDOF values.

In future work, we aim to investigate the co-design of optics and post-processing for broadband EDOF imaging. Moreover, we aim to explore the application of our approach in addressing the challenges associated with light field microscopy. Our next objective is fabricating a selected metasurface, followed by a comprehensive evaluation of its performance within the context of the 4f system setup. We believe that these research directions will contribute significantly to the ongoing advancement of microscopy imaging, ultimately leading to more sophisticated and versatile optical systems.