Neural network enabled wide field-of-view imaging with hyperbolic metalenses

2.1 Design, fabrication and optical characterization of the metalens

The hyperbolic metalens used in this work has a phase profile given by the expression

where r is the radial distance from the center of the lens, and λ and f are the design wavelength and focal length, respectively. The fabricated metalens has a diameter of D = 5 mm and f = 1.813 mm, designed at a working wavelength of λ = 850 nm with a numerical aperture of NA = 0.81. The (wrapped) hyperbolic phase profile was mapped using amorphous silicon (a-Si) nanopillars with a circular cross-section (to maintain polarization-insensitive response) on a fused silica substrate. These pillars are arranged in a hexagonal lattice (lattice constant of 350 nm) and have a fixed height of 500 nm and diameters in the range of 140–264 nm (Figure 1a–d). The simulated transmittance and phase as a function of the pillar diameter are also plotted in Figure 1e.

Figure 1:

Fabrication of the hyperbolic metalens and the imaging setup. (a) Schematic of the hexagonal unit cell of the metalens with a lattice constant of a = 350 nm, consisting of cylindrical nanopillars with a height of H = 500 nm and diameters, D, ranging from 140 nm to 264 nm. (b) Optical image of a wafer with an array of metalenses patterned using deep UV immersion photolithography. (c, d) Scanning electron micrographs of the metalens depicting the patterned a-Si nanopillars on a glass substrate. (e) The simulated phase and transmittance of uniform a-Si nanopillar arrays with height of 500 nm and diameters ranging from 140 to 264 nm with a step size of 2 nm. (f) Optical image of the fabricated 5 mm diameter metalens. (g) Optical image of the hyperbolic metalens camera used in imaging, where the (h) metalens (white circle) is mounted directly in front of the CMOS detector (red square).

The samples are fabricated using a 12-inch, deep ultraviolet (UV) immersion photolithography scanner (see details in Methods) and optically characterized using a goniometric optical setup (Figure 2a). This characterization setup, which has a calculated magnification of 83.3 (resulting in an effective detector pixel size of 41.4 nm) allows imaging the PSFs of the fabricated hyperbolic metalens at various angles of incidence (AOI). Figure 2b compares these measurements against theoretical PSFs calculated with Fourier optics simulations (details in Supplementary information). As can be seen, there is a close match between the measured and simulated PSFs, with minor discrepancies likely attributed to fabrication errors.

Figure 2:

Characterization of the hyperbolic metalens. (a) Optical setup for angles of incidence dependent optical characterization of the hyperbolic metalens PSFs. The laser (850 nm wavelength) output isexpanded using two lenses and an aperture to distribute the intensity uniformly across the metalens. These are mounted on a rotating arm of a goniometer that enables different AOI illumination on the metalens. The metalens focuses the collimated laser at the focal plane, which is then imaged onto the CMOS detector using an objective lens (Olympus, MPLAPON100X) and a tube lens (150 mm focal distance). (b) (top) Experimentally measured PSFs at different angles of incidence compared to (middle) simulated PSFs for the hyperbolic metalens. We use a log-normalized colormap for better visualization of these PSFs. The scalebars (white) have sizes of 10 µm and 2 µm for the main image and its inset, respectively. (bottom) The horizontal line profiles of the measured (red) and simulated (blue) PSFs.

2.2 Hyperbolic metalens camera

The hyperbolic metalens camera comprises only two components (Figure 1g and h): the metalens and a complementary metal oxide semiconductor (CMOS) detector (Thorlabs, Zelux-CS165MU), resulting in an ultra-compact design. The scene is illuminated with a light-emitting diode (LED) with a dominant wavelength of 850 nm (Thorlabs M850L3) and bandwidth of 30 nm (setup figure in Supplementary information). The hyperbolic metalens, mounted at a distance f = 1.813 mm from the detector, focuses the illuminated scene onto the sensor to form the image. In this work, we used only the detector’s central 512 × 512 pixels, corresponding to an angular field-of-view (FOV) of 54°, due to memory constraints during network training. Beyond this computational limitation, the practical FOV of our hyperbolic metalens camera is also restricted by physical factors: at larger angles, the PSFs eventually spread beyond the detector size, and exhibit weaker signal, making them difficult to capture and accurately model the physics of image formation for our eigenPSF method.

2.3 Restormer deblurring

Figure 3 shows the schematic of our computational deblurring approach for hyperbolic metalens imaging. Using our imaging setup, we first measured the PSFs at different AOIs ranging from 0° to 40° (Figure 3a), with denser sampling at smaller angles to capture rapidly varying PSF shapes and sparser sampling at larger angles where the PSFs primarily grow in size (see Supplementary information). Note that these imaging PSFs are different from the measured PSFs depicted in Figure 2b as there is no external magnification in the imaging setup. These imaging PSFs are computationally rotated to populate a PSF map that covers the full extent of an image corresponding to an angular FOV of 54° as shown in Figure 3a. These spatially-varying PSFs are then eigendecomposed into spatially-invariant eigenPSF bases weighted by the corresponding eigencoefficients [26] (see Supplementary information) to simulate spatially-varying blur applied to ground truth images from Google’s Open Images dataset [37], [38]. This enables efficient and accurate generation of large training datasets with spatially varying PSFs, a task that would otherwise be prohibitively slow or experimentally impractical. Each simulated blurred image is corrupted with noise by augmenting a measured flatfield through random rotations and flips (Figure 3b). The total time taken to simulate 3,500 noisy and blurred images on a single NVIDIA L40 GPU was approximately 10 min.

Figure 3:

Schematic for computational deblurring using the Restormer architecture trained on eigenPSF-simulated images. (a) The hyperbolic metalens is characterized using the measured PSFs, and the flatfield records the noise profile of the imaging system. (b) Simulated images are obtained using the eigenPSF method to apply spatially varying blur on the ground truth dataset, and further corrupted by noise created by augmenting the flatfield measurement. (c) The Restormer architecture is used to deblur and denoise the images. Illustration created using PlotNeuralNet [39].

A Restormer network [40] is trained using these 3,500 simulated images as input, and their corresponding ground truth images as the desired output (Figure 3c). We use the default parameters and loss functions described in the original paper [40] for the Restormer network, except reducing the number of channels to [36, 72, 144, 288] for layers L1 to L4, respectively due to GPU memory constraints. This constitutes a total of 14.8 million trainable parameters in our Restormer network. Using 4 NVIDIA L40 GPUs, training for 200 epochs with a batch size of 1 took approximately 48 h.

Figure 4 shows the raw measurements from our hyperbolic metalens camera and the corresponding results of Restormer deblurring on the images (see Supplementary information for more results). The characteristic aberrations due to the hyperbolic lens phase profile are evident in the sharp features at the center of the image and the increasing coma at larger incidence angles. The images here are not diffraction-limited due to the broadband LED used to illuminate the scenes and photos. The physical size of the detector pixels also limits the resolution of the measurements.

Figure 4:

Deblurring images from the hyperbolic metalens camera using the trained, reference-free Restormer network. The (a) measured and (b) deblurred images of scenes taken around a lab. The (c) measured and (d) deblurred images of printed photos placed before the camera. All images have the same angular FOV of 54°.

Despite the spatially-varying aberrations in the measurements, the trained Restormer network is able to deblur the full FOV of the images in real-time (∼50 ms per image), recovering features even toward the edges of the images. By using a reference-free dataset, we avoid overfitting to specific imaging conditions during the training of the Restormer network. This is further demonstrated in Figure 5 where under varying illumination directions and obstructions, our trained Restormer is still able to recover features even with low lighting at various regions of both the scene and printed USAF card.

Figure 5:

Deblurring images from the hyperbolic metalens camera with varying illumination direction and obstructions. The (a) measured and (b) deblurred images of the same lab scene under different lighting. The (c) measured and (d) deblurred images of a printed USAF card, where the card was tilted in the last column. All images have the same angular FOV of 54°.

Figure 6 further demonstrates the improved quality of deblurring from our trained Restormer network over other existing state-of-the-art approaches. The first of which is an Autograd implementation of the eigenCWD algorithm [26] which utilizes PyTorch’s [41] inbuilt automatic differentiation engine to perform optimization instead of using analytical gradients (details in Supplementary information). The reconstruction from this iterative approach in Figure 6 is contaminated with noisy artifacts as it only accounts for spatially-varying blur and not the noise characteristics of the sensor. In addition, as an iterative algorithm, the Autograd implementation of eigenCWD is incapable of real-time deblurring (∼1 min per image on a single NVIDIA L40 GPU). Using the same dataset, we also trained a Multiscale neural network architecture [42] which has recently been used in image reconstruction applications for metalenses [31], [32] (details in Supplementary information). However, we observe residual smeared artifacts in the output of the Multiscale network, likely attributed to the presence of noise in the training dataset which the network is unable to fully remove. This suggests that the Restormer network remains robust against noise and demonstrates improved performance in spatially-varying deconvolution over existing state-of-the-art methods.

Figure 6:

Comparing deblurring performance using various algorithms. The Restormer network surpasses both Autograd and Multiscale methods in both spatially-varying deblurring capabilities as well as suppressing noise.

4.1 Fabrication of metalens

A 193 nm argon fluoride (ArF) deep-ultraviolet (DUV) immersion photolithography process combined with a dry etching process is used to fabricate the hyperbolic metalenses. The metalens comprises millions of amorphous silicon (a-Si) nanopillars, which are patterned on a 350 nm-thick a-Si film deposited using plasma-enhanced chemical vapor deposition (PECVD) on a 12-inch fused silica wafer. The wafer was then diced into small coupons, and the individual a-Si metalenses were subsequently etched using a dry etching process. Hence, the metalens pattern was transferred from the photoresist to the a-Si film, forming a-Si nanopillars. The 480 nm a-Si pillars were etched in multiple smaller steps instead of a continuous step. This process is recommended, especially when performing deep etching with high aspect ratio pillars. After every etching step, the chamber undergoes a 5-min cooling period before the next step. This not only provides smoother sidewalls but also protects the pillars from undercutting. A residue layer of SiO₂ (30 nm) remains on top after a-Si etching as a part of the etching hard mask, but it possesses no hindrance to the optical performance, and therefore, we do not remove it.