Optical computing metasurfaces: applications and advances

3.1.1 Dielectric nano-antenna

Dielectric metasurfaces are typically constructed using materials with lower electrical conductivity and reduced losses, resulting in reduced energy losses in the optical and microwave frequency ranges. This renders dielectric metasurfaces particularly advantageous in optical computing applications demanding high efficiency and transmission. The dielectric metasurface is widely used in optical computing because of its high refractive index, low internal loss, and high transmission efficiency.

A. Chizari et al. propose a dielectric reflection meta-reflect-array (MRA) that overcomes the large volume problems associated with optical metamaterials and also demonstrates several structural advantages [105]. The meta-array supports Mie co-oscillation of the electric dipole and Mie resonance of the magnetic dipole, as well as multiple reflection patterns within the spacer layers, and thus enables full spatial control of the amplitude and phase distribution of the reflected cross-polarized light by changing the size of the resonator. By controlling the amplitude and phase distribution of the reflected cross-polarized light and simulating the reflection calculation system, the optical simulation computation operation is realized. Abdollahramezani et al. proposed an equation solver based on cascaded all-dielectric optical computing metasurface [106]. The structure diagram is shown in Figure 2(a). The phase and amplitude of the incident light wave can be controlled independently by changing the long and short axis of the nano-silicon ellipse structure. On this basis, the near-infrared (NIR) ordinary differential equation and the integral-differential equation solver are designed.

Figure 2:

Basic modulation structure units. (a) Schematic diagram of the all-dielectric metasurface equation solver. (b) Schematic of performing a mathematical operation for a reflective metasurface. (c) Operation procedure of using the OCNN to recognize handwriting numbers from the MNIST database. The OCNN consists of a convolution layer with two kernels, a pooling and a fully connected layer. The output gives the answer whether the input image is “1” or “2.” (d) Schematic view of switchable THz quarter-wave plate design. The top left inset is a zoom image of the unit cell in the fabricated sample (P = 150, L _x = 90, L _y = 124, l _x = 9, l _y = 5, and w = 9 μm, and the scale bar is 50 μm). The top right inset is the simulated ellipticities of the output THz waves. (a) Reproduced with permission [106]. Copyright 2017, Optica Publishing Group. (b) Reproduced with permission [64]. Copyright 2017, Optica Publishing Group. (c) Reproduced with permission [65]. Copyright 2021, Nature Publishing Group. (d) Reproduced with permission [50]. Copyright 2015, Nature Publishing Group.

3.1.2 Plasmon or metallic nano-antenna

There are a large number of freely moving electrons inside the metal, and these electrons dominate the electromagnetic properties of the metal under the action of the external electric field. Free electrons in metals can be coupled to incident light at the interface between metal and medium and produce surface plasmons [107]. In 2017, Zhu et al. realized spatial optical simulation differential operation in a small angle range [108]. A double-layer metal grating based on surface plasmon is proposed to achieve simultaneous differentiation in both temporal and spatial domains of optical signals. By using the unidirectional excitation of the surface plasmon polaritons (SPP) of the asymmetric metal grating, the vertical incident spatial and temporal differentiation are realized [109]. In addition, using the free control of the amplitude and phase of the light wave by metal–insulator–metal (MIM) metasurface, spatial simulation calculators are proposed and their operation capability is tested experimentally [99]. The MIM metasurface utilizes gap-surface plasmon mode to achieve independent amplitude and binary phase control of the reflection coefficient of 800 nm NIR light. The spatial simulation optical differentiator and integrator are designed and their computational functions are verified. The fused quartz spacer on top of the silver dendritic structure and the metasurface of the quasiperiodic silver dendritic substrate are proposed [64], the structure of which is shown in Figure 2(b). The phase of the common polarized light reflected by changing the shape of the silver dendritic structure is distributed spatially. The system consists of a Fourier transforming (FT) block and a reflective metasurface with a position-dependent reflection coefficient r(x, y). The metasurface green and red bands are differentially operated and the metasurface-based differential processor can be used for real-time edge detection and image contrast enhancement.

3.1.3 Reconfigurable nano-antenna

Phase change material (PCM) [110], [111] refers to a class of functional optical materials that undergo structural phase change under the action of external fields (such as force, heat, light, electricity, et al.). This unconventional physical property is caused by the bonding mechanism changes [112]. PCM is accompanied by significant optical constant changes (Δn > 1, Δk produces orders of magnitude changes). Phase change materials commonly used in optical frequency bands include VO₂, Ge₂Sb₂Te₅ (GST) [113], [114], and polyvinyl alcohol (PVA) [93], [115].

GST is composed of germanium (Ge), antimony (Sb), and tellurium (Te) alloys. The phase change materials can transition from a disordered amorphous state to an ordered crystalline state through a heat-quench cycle [116]. In the infrared region, GST has a high refractive index in both crystalline and amorphous phases, indicating that the GST nanostructure should be able to support strong resonances closely related to its material phase [117], [118], [119]. Chalcogenide phase change materials have also become an example of achieving a nonvolatile, all-optical dielectric metasurface, which exhibits a low-loss Mie resonance in contrast to precious metals [120], [121].

Wu et al. proposed a multimode photon computing core based on an array of programmable mode converters [65]. The structure of the multimode photon computing core, as shown in Figure 2(c), is constructed with waveguides on a metasurface of phase-change materials. The programmable converter uses the refractive index change of the phase change material GST during the phase change to control the waveguide spatial mode with very high accuracy in mode contrast up to 64 levels. A metasurface optical computing system is constructed to perform image processing and recognition tasks with high precision, as shown on the right side of Figure 2(c). The system has a wide operating bandwidth and a small volume and is expected to be applied to photonic neural networks with ultra-high computational throughput. Based on the ability of the GST metasurface to regulate the light field, this structure will also become one of the important solutions for future optical computing.

Vanadium dioxide (VO₂), as a reliable reconfigurable metasurface material, has also received extensive attention. VO₂ exhibits an insulator–metal transition (IMT) near room temperature (T _IMT ≈ 67 °C in bulk crystals) [122], [123], [124]. It exhibits refractive index changes over a wide spectral range. The variation in VO₂ properties stems from the change in its crystal structure [125], [126]. VO₂ has a monoclinic lattice with the electrons localized in the atomic bonds at low temperatures. Compared with chalcogenide materials, their phase transition temperature is lower and they are easily stimulated by heat, electricity, and light.

By loading VO₂ components on metasurface structures, an ultrathin switchable THz quarter wave plate (QWP) with a switching range of 34 GHz was experimentally demonstrated in Figure 2(d) [50]. It shows a schematic of the switchable QWP, which is composed of ultrathin asymmetric cross-shaped resonator arrays. VO₂ pads are inserted at the end of the cross-shaped resonators. A linear normal incident THz wave polarized at θ = 45° to the two slots is converted into a circularly polarized light. The inserted VO₂ is able to change the effective length of the metal resonators through the phase transition controlled.

3.1.4 Programmable electric-control nano-antenna (FPGA/MEMS)

In 2014, Cui’s team proposed a programmable metasurface, the unit is composed of a passive metasurface structure and a switching diode [127]. The switching diode has “on” and “off” states, and the reflected phase of the unit in the two states has a phase difference of 180°, thus forming a 1 bit phase code. Moreover, a Field Programmable Gate Array (FPGA) is used to provide the control voltage, such as high and low levels corresponding to the code “1” and “0,” respectively. As shown in Figure 3(a), the preset coding sequence is then input to different units on the metasurface via different PINs by the FPGA, thereby dynamically changing the digital coding pattern and realizing programmable control of different electromagnetic functions.

Figure 3:

Electronic control structure units. (a) A flow diagram for realizing a programmable metasurface controlled by the FPGA. (b) Conceptual illustration of a space-time-coding digital metasurface. The increasing need for dynamic manipulation of light in modern optical systems is driving the development of electrically and optically tuned reconfigurable metasurfaces. (c) Colored scanning electron microscope (SEM) image of the MEMS Fano metasurface. (d) Realization of switch between bright-field image and edge image by tunable optical differentiator composed of metasurface and liquid-crystal phase plate. (a) Reproduced with permission [127]. Copyright 2014, Nature Publishing Group. (b) Reproduced with permission [128]. Copyright 2018, Nature Publishing Group. (c) Reproduced with permission [63]. Copyright 2018, Nature Publishing Group. (d) Reproduced with permission [72]. Copyright 2022, Optica Publishing Group.

Additionally, the concept of spatiotemporal coded digital metasurface was proposed based on digital coding and programmable metasurface [128]. As shown in Figure 3(b), a two-dimensional array composed of M × N programmable basic units. Each unit is integrated with adjustable devices whose amplitude, phase, and polarization can be controlled in real time. The external control FPGA based on programmable devices can simultaneously output M × N control signals to control each unit of the metasurface, where each signal is a specific coding sequence that varies periodically in real time. As shown to the right of Figure 3(b), the green and red dots represent the coding states “0” and “1,” respectively.

MEMS can accurately generate a certain external force (Coulomb force, amperage force, etc.) on the target structure through an external electric field, magnetic field, thermal stimulation, and other ways on the micro/nano-scale and quantitatively deform its geometry by breaking the original mechanical balance of the metasurface, so as to accurately and dynamically regulate its optical response. The excitation and active tuning of sharp Fano resonances are proposed on MEMS reconfigurable metasurface that exhibits multiple-input–output (MIO) properties in both near and far-field optical properties [63]. The reconfigurable geometry of the MEMS Fano-metasurface provides various structural meta-stable states by using two independently controllable electrical inputs and an optical/near-field readout that enables the realization of digital logic gates. The Fano metasurface MEMS-based is shown in Figure 3(c). The MEMS-integrated metasurface has the advantages of low power consumption, large modulation range, and full band regulation from visible light to terahertz, which is one of the competitive schemes for dynamic light modulation calculation. The true (ON) and false (OFF) states of the far-field Fano resonance amplitude are represented by the binary digits “1” and “0,” respectively. The THz far-field transmission spectrum shows the XOR logic properties of the MEMS resonator in different metastable structural states (00), (10), (01), and (11). Voltage inputs (V1 and V2) applied to a single resonator can be programmed using sequential trigger bits {0,1} to control the drive heights (up/down) of SRR-1 and SRR-2, respectively. However, MEMS-based compact photonic devices are usually more complex in the process of structural design and characterization, and the component processing is also very challenging, and the cost is high.

3.1.5 Liquid crystals (LCs) nano-antenna

The combination of liquid crystals (LCs) and metasurfaces can also play its unique advantages in optical computing. LCs are composed of molecules with different orientations that can be controlled by external stimuli, such as electric fields or heating [129]. Therefore, LCs are often used as ambient materials in metasurfaces [130]. With the change of phase, the refractive index and dielectric constant of LCs also change. Compared with crystal materials, semiconductor materials, metal materials, and metamaterials, LCs have great flexibility in material design [131], [132]. The versatility of LCs have also led to its widespread application in combination with metasurface nanophotonic devices. A LCs-based tunable scheme is proposed to achieve tunable edge enhancement images based on computational metasurface as shown in Figure 3(d) [72].When light waves pass through the metasurface, the metasurface can perform spatial difference operations. This optical differential operation can be achieved by the interaction between two orthogonal polarization components. By adjusting the external voltage applied to the liquid crystal phase plate, the different phase delay between the two orthogonal polarization components is introduced. Although the edge-enhanced image of the scheme is only one-dimensional, it can quickly switch between the bright field image and the edge image. Therefore, it has a broad application prospect in high-contrast microscopy. Trevon Badloe et al. designed a dual-mode metalens system integrated with LCs that can electrically switch between bright-field imaging and edge-enhanced imaging modes at the millisecond level. The design is integrated into the same single flat optical device, significantly reducing the size, bulk, and weight of the system [133].

With the increasing demands for refined photoelectrical devices, the liquid crystal material gradually reveals a lot of deficiencies. The response time of the liquid crystal device is usually in the number of ten milliseconds, which is difficult to match with some fast response photoelectrical components. Secondly, because the liquid crystal itself is a large molecular material, its molecular weight can reach 200∼500 g/mol, the electrode spacing and pixel size of the reported liquid crystal metasurface are usually large, and it is difficult to reach the sub-wavelength scale. In addition, the liquid crystal device still has shortcomings in the aspects of self-generating thermal response, beam deflection velocity, and large angular deflection efficiency [134], [135]. Nevertheless, the regulatory capability of liquid crystal still occupies an important position in the application of dynamic active photonic devices, and it is a material that needs to be continuously explored in the future of metasurface.

3.2.1 Analytical method

The Sobel operator is a popular edge detection method with low algorithm complexity, which can be used to calculate the image gradient approximation [137], [138] and edge detection [139], [140]. In a digital image f (x, y), where (x, y) represents Cartesian spatial coordinates, where G _x and G _y are the gradient components, respectively, in x and y directions. G _x and G _y are obtained by convolution of the image using the Sobel operator, where G _x = f(x, y) ⊗ I _x , G _y = f(x, y) ⊗ I _y . The Sobel operator uses a 3 × 3 convolution kernel in both horizontal and vertical directions, and I _x and I _y are shown in Table 1.

The convolution operation results are obtained by using the horizontal operator and vertical operator template and the pixels on the image. An appropriate threshold (TH) is compared with G, where, G = G x 2 + G y 2 [141]. When TH ≥ G, the edge of the image is determined. Otherwise, this pixel is ignored as a nonedge pixel. However, the edges of most images are complicated, so the calculation of image edges using only horizontal and vertical Sobel operators will cause a large error.

Compared with the first-order Sobel operator, the Laplacian operation as an isotropic second-order differential is an essential mathematical calculation in various fields of physical science and engineering [142], [143]. The optical analog Laplacian is regarded as an effective tool for solving complex and large-scale mathematical problems [144].

The optical analog computing of Laplace operation for a 2D spatial function can be expressed in the Cartesian coordinate as:

where, ∇² is the Laplacian operator, and E(x, y) is the electric field distribution of the input signal. So, the Laplace operation on the optical field is:

where k 2 = k x 2 + k y 2 and is the in-plane wave vector. So, the optical transfer function (OTF) of the Laplacian should be T k = − k 2 , which is a quadratic function of the in-plane wave vector of the input light field. Therefore, for normal incidence, the transmittance of the Laplacian should be zero, and as the incidence angle increases, the transmittance should increase.

Many optical computing metasurfaces based on Laplacian operators have been reported. A. Saba et al. proposed a second-order differentiator based on an all-dielectric metasurface [145]. Using the grating guided wave mode, the device can realize the two-dimensional Laplacian operator operation for S-polarized incident waves. When the incident light passes through the Laplacian metasurface, the incident light field is automatically processed, as shown in Figure 4(a) [69]. By exciting the quasi-bound state in the continuum (BIC), an optical transfer function for nearly perfect isotropic second-order differentiation has been obtained with a higher spatial resolution. The Laplace calculation result can be obtained quickly by recording the output optical field. Specifically, a two-dimensional Laplace operator operating in visible and near-infrared bands was designed. Accordingly, an image is illuminated through a Laplacian metasurface, and image edges can be obtained at the output plane [146]. The experimental results show that the edge detection resolution of the device was as high as 4 μm. By integrating the photonic crystal-based differentiator with the metasurface lens, a compact differential computing system can be constructed.

Figure 4:

Optical computing methods. (a) Illustration of a dielectric metasurface transforming an input 2D spatial function to another function as a Laplace operator. (b) The schematic of optimize algorithm. (c) Optical system of D²NN. L ₁, L ₂, L ₃, and Ln: first, second, third, and nth diffractive layer. (d) A 3D-printed D²NN successfully classifies handwritten input digits (0–9) based on 10 different detector regions at the output plane of the network, each corresponding to one digit. (a) Reproduced with permission [69]. Copyright 2021, Optica Publishing Group. (b) Reproduced with permission [148]. Copyright 2018, Nature Publishing Group. (c)–(d) Reproduced with permission [155]. Copyright 2018, American Association for the Advancement of Science.

3.2.2 Optimization algorithm

Genetic algorithms (GAs) are stochastic search optimizers that are based on the concepts of evolution and natural selection [147], [148], shown in Figure 4(b). GAs are inherently parallel algorithms. Therefore, GAs are widely used in dealing with multidimensional function domain, discrete solution domain problem, and nondifferentiable objective function. Furthermore, a metasurface based on an adaptive genetic algorithm (AGA) is proposed, capable of solve the complex problems in the field of optics. It can diffract TE and TM excited waveguide modes in any chosen direction. These applications demonstrate the advantages of combining metasurface and AGA technology.

3.2.3 Neural network optimization algorithm

Deep learning is a crucial tool for performing advanced inference tasks at present [70], [149], [150], [151], [152], [153], [154]. An all-optical diffractive deep neural network (D²NN) architecture is introduced as a physical mechanism for performing machine learning [155]. The architecture can accomplish various functions based on the design of passive diffraction layers using deep learning. The D²NN is placed vertically in the direction of light propagation by the multilevel diffraction optical element (DOEs) with preset distribution. The incident light field propagates forward in free space through DOEs. All pixel values (phase-only, amplitude-only, or complex-amplitude) are optimized by an error backpropagation algorithm similar to their counterparts in deep learning. The spatial light intensity distribution in the input plane and output plane corresponds to the input vector and output vector, respectively, as shown in Figure 4(c). The information processing capability of the D²NN system depends on the number of diffraction layers [156]. The whole optimization system is linearly transformed from the input light field to the output light field. As an example, D²NN recognition of handwritten input “5” is demonstrated in Figure 4(d), where the blue dotted squares represent the trained detector regions for each digit. This architecture can be used as a linear classifier and has been proven to be able to optically classify handwrites digit images in the Modified National Institute of Standards and Technology (MNIST) dataset and fashion product images in the Fashion-MNIST dataset with moderately high accuracy [157].

3.3.1 Edge detection and enhancement

An image is the basis component of human vision. The image edge is defined as the boundary region where the brightness, phase, and polarization of the target image undergo significant changes. Generally, human eyes are more sensitive to the edge information than the flat area. Hence, the image edge information occupies a critical position in human and machine vision.

The technique of optical edge detection was developed by Zelnick in 1942 to improve the cell’s contrast [158]. Then, by introducing slow periodic spatial modulation on the metasurface, the nonlocal response can be significantly improved. A spatially modulated metasurface with a suitably designed nonlocal response is proposed for performing different types of mathematical operations on optical signals [159]. The metasurface consists of an array of periodically resonating particles in the X–Y plane, as shown in Figure 5(a), which consists of split-ring resonators (SRRs) parallel to the X–Z plane (magnetic dipole moment parallel to the y-axis) [160]. The metasurface is excited by TM polarized waves propagating in the x–z plane. The metasurface with broken x and z symmetry results in asymmetric responses with respect to ±k _x , achieving the required first-derivative operation. However, the disadvantage of this method is that for x-polarized illumination, the metasurface allows detection along the x-axis but not along the y; it performs edge detection only for TM waves. Nevertheless, many scenarios in practical applications use unpolarized light to illuminate the image, as shown in Figure 5(b). The nonlocal metasurfaces perform basic mathematical operations, paving the way toward fast and power-efficient ultrathin devices for edge detection and optical image processing.

Figure 5:

Edge enhancement metasurfaces. (a) Schematic of a metasurface consisting of a periodic array of resonant particles in the x–y plane, formed by split ring resonators (SRR) parallel to the x–z plane (magnetic dipole moment parallel to the y axis). (b) The output image used to illuminate a metasurface with unpolarized light. (c) Schematic diagram of metasurface edge detection. (d) Images of the undyed onion epidermal cells with a 20× objective lens. Traditional bright-field images captured with LCP incident light at the wavelength of 480 nm. Spiral phase contrast images captured with RCP incident light at the corresponding wavelengths. Scale bar: 100 μm. (e) When PB phase gradient metasurface is illuminated by collimated LP beam, two separated LCP and RCP beam components are observed. (f) Original image without edge detection. Different portion of the edge shows up for different metasurface orientations. Arrows indicate the edge-detection direction. (g) Schematic diagram of metasurface edge detection of a 2D optical differentiator. (h) Output edge image with metasurface period of 27.36 µm. (a)–(b) Reproduced with permission [159]. Copyright 2018, American Physical Society. (c) and (d) Reproduced with permission [161]. Copyright 2020, American Chemical Society. (e) and (f) Reproduced with permission [162]. Copyright 2019, National Academy of Sciences.(g)–(h) Reproduced with permission [71]. Copyright 2023, Optica Publishing Group.

By designing the different photon spin response characteristics, the dual-path spin vortex PB-metasurface is proposed in Figure 5(c) [161]. When the LCP incident light is incident on the metasurface, the light is generated with a constant phase profile and Gaussian intensity distribution, and the rotation of the incident polarization is flipped to RCP. For RCP incident light, the rotation of polarization is flipped to LCP. And the light produced has a spiral phase distribution and a donut-shaped intensity distribution. Moreover, the system can easily detect samples with a small refractive index difference from the background environment, such as biological cells, as shown in Figure 5(d). However, the metasurface system is hard to calculate such as first-order differentiation. In addition, edge enhancement methods based on the vortex phase cannot precisely control edge resolution, resulting in limited detection accuracy.

An optical spatial differentiator is proposed, which consists of a designed PB phase metasurface inserted between two orthogonal linear polarizers [162]. As shown in Figure 5(e), when a linear-polarized plane wave is incident on a PB phase gradient metasurface with phase φ x , y = π x Λ , the LCP and RCP components gain additional phase +2φ and −2φ. It can be understood that the metasurface divides the input horizontally polarized beam into LCP and RCP, and there is a certain lateral displacement of the two beams traveling in different directions. The edge portion contains only the LCP or RCP components, so after passing through the vertical analyzer, only the outer edge information is left. Furthermore, radial edge detection based on metasurface can be easily extended to 2D edge detection by changing the one-dimensional phase gradient φ x , y = π x Λ to a radial phase gradient φ x , y = π x 2 + y 2 Λ . As shown in Figure 5(f), an image of “1960 UCSD” without using the analyzer is represented. Therefore, the system needs to be used with an analyzer in most cases. Similarly, Figure 5(g) shows how the medium metasurface performs two-dimensional edge detection [71]. The highest resolution achieved is 1.7 μm at the period of 27.36 µm (Figure 5(h)), which is close to the diffraction limit of the optical system. Compared to 1000 µm size metasurfaces reported in previous work [163], a higher contrast ratio is obtained with a smaller device size, which is conducive to miniaturization. Meanwhile, elliptical cylinders with simpler structures are adopted, which make it easier to integrate with the CMOS process and facilitate fabrication.

3.3.2 Object recognition

Object recognition plays an important role in optical computing fields. A diffraction neural network is proposed to perform machine learning, and this structure can realize various functions of passive diffraction layers designed based on deep learning. The all-optical deep learning framework can analyze and detect features in all-optical images at light speed to realize image classification, such as handwritten digits and fashion products. In 2022, Cui’s group proposes an optical-electronic hybrid neural network (OENN), which consists of a single titanium dioxide (TiO₂) metasurface and a fully connected electron layer [164]. The schematic diagram of an OENN and the corresponding framework are shown in Figure 6(a). The ultra-compact metasurface extracts the features of the input digits. Then, a low-complexity electronic fully connected layer weights and sums all the features, leading to a final prediction. The combination of nonlocal neural layer and nonlinear transformation greatly expands the capacity of neural networks. Therefore, the classification accuracy of handwritten digit recognition can still be as high as 98.05 % using a single metasurface layer.

Figure 6:

Classification computing metasurfaces. (a) The schematics of the OENN and the corresponding framework. (b) Optical MDNN layout of polarization-dependent object classification. (c) The fabricated MDNN (1st and 4th columns), the output field intensity detected for these four-category classification MDNN in x-or y-polarization (2nd and 5th columns), and experimentally detected energy distribution (3rd and 6th columns). (a) Reproduced with permission [164]. Copyright 2022, Wiley-VCH. (b) and (c) Reproduced with permission [66]. Copyright 2022, Nature Publishing Group.

All optical diffraction neural networks (ODNN) can perform specific optical computation functions by training the diffraction phase plane to form a cascaded physical network. A polarization-multiplexed metasurface-based diffraction neural network is proposed and integrated with a complementary metal-oxide semiconductors (CMOS) image sensor to create an on-chip multitasking architecture in visible range [66]. As shown in Figure 6(b), the physical model is mainly composed of three parts: the linear polarized plane object wave to be recognized as the input layer; the polarization-multiplexed metasurfaces as the hidden layer; and the image sensor on the CMOS chip as the output layer. The phase distribution of the meta-neurons under each polarization channel is obtained by error backpropagation and stochastic gradient descent algorithm. Furthermore, this configuration can form a multi-channel sensing and computing all-in-one chip architecture. The results of MDNN recognition of different handwritten digits and fashion items in two polarization channels are shown in Figure 6(c). By testing 160 groups of correctly recognized images, it is found that the recognizing accuracy of handwritten digits and fashion products reaches 93.75 % and 95 % in the experiment and simulation, respectively. Although the method can identify numbers and fashion items at the same time, its disadvantage is that it can only identify four numbers rather than all numbers 0–9 at present, as well as the fashion products.

3.3.3 Gesture or movement recognition

Gesture is a basic human characteristic and an indispensable part of the process of interpersonal communication. The gesture recognition technology has made it possible for man–machine interaction. By mapping commands to outputs for device control, human postures can be explicitly identified as representations of input and processing [101], [165]. Particularly, an intelligent metasurface artificial neural networks system is proposed to convert measured microwave data into an image of the entire human body [166]. Figure 7(a) schematically illustrates three building blocks of the data flow pipeline. The microwave data collected by the intelligent metasurface are instantly processed with an imaging convolutional neural network (CNN) to reconstruct the image of the whole human body. The human breathing is identified by the time-frequency analysis of microwave data. It is worth mentioning that the proposed intelligent metasurface system also works well on stray Wi-Fi signals.

Figure 7:

Gesture or movement recognition. (a) Microwave data processing flow by using deep learning CNN cluster. (b) The working principle of the noncontact hand gesture recognition method based on the programmable metasurface sample with a machine learning algorithm. (a) Reproduced with permission [166]. Copyright 2019, Nature Publishing Group. (b) Reproduced with permission [68]. Copyright 2022, Wiley-VCH.

Meanwhile, a noncontact gesture recognition method based on transmission programmable metasurface is proposed [68]. As shown in Figure 7(b), the inner forearm is mapped to a 2D plane, where the x and y axes are parallel and perpendicular to the direction of the arm, respectively. Using dynamic control of electromagnetic focusing in wavefront engineering has more focused spots to obtain comprehensive echo data. Concretely, it is processed by machine learning to realize high-precision noncontact gesture recognition. In the frequency range of 5–6 GHz, echo coefficients under 5 different focal points controlled by a programmable metasurface were recorded for successfully identifying 18 gesture and wrist movements. Compared with the above system [166], the types of gestures recognized have been improved. However, there are some drawbacks to the system, such as the need for the arm to remain stable during testing. And the arm needs to be wrapped within 10 cm of the metasurface, which can cause discomfort in practical applications. At present, it is difficult to achieve accurate classification and identification between individuals, so in future studies, more data can be collected from different populations. And design a deep neural network model that may help solve cross-individual problems.

3.3.4 Logical operation

The logical operations demonstrate the key role of electronic computing, which can perform general calculations and possess fast processing speed, low crosstalk, and high throughput. However, the optical logic gates currently reported rely heavily on precise control of the input optical signals, including phase difference, polarization, intensity, and the size of the incident beam. Due to the complexity and difficulty of these precise controls, the output optical logic states may have instability and low-intensity contrast [167]. In addition, lighting devices are considered to achieve the miniaturization of optical logic gates. Therefore, getting rid of these complex controls remain challenging in a compact photonic system and achieving full logic functionality.

A special class of structurally reconfigurable metasurfaces based on micro/nano electromechanical systems (MEMS/NEMS) offers unique advantages in actively manipulating the near field in all three spatial directions by exploiting sensitive changes in micro/nanoscale motion. A reconfigurable MEMS Fano resonant metasurface is proposed with a multi-input–output (MIO) state [63]. It has two independently controlled electrical inputs and an optical read-out at a terahertz frequency to perform logical operations. Similar to the digital XOR logic operation shown in Figure 8(a), when the input voltage states are different, the Fano output is true (F = 1), otherwise the Fano output is false (F = 0) (when both inputs are true (11) or false (00), the output state F = 0). However, MEMS-based metasurfaces are usually more complex in the process of structural design and characterization, and the component processing is also very challenging, and the cost is high.

Figure 8:

Logical computational metasurfaces. (a) Pictorial representation of realizing the OTP secured wireless communication channel by performing the XOR logic operation to encode the private message (m) with the secret key (k) and is sent through the public channel as optical signals. The message is retrieved securely (decrypted) at the destination end by performing the inverse XOR operation on the measured optical states (F) with the secret key (k). (b) Layout of a diffractive neural network for photon-based logic operations. (c) The concept of the FCNN used for logical operation. (d) Schematic of OAM mode logical operation based on ODNN. (a) Reproduced with permission [63]. Copyright 2018, Nature Publishing Group. (b) Reproduced with permission [67]. Copyright 2020, Nature Publishing Group. (c) Reproduced with permission [101]. Copyright 2022, Optica Publishing Group. (d) Reproduced with permission [168]. Copyright 2021, Optica Publishing Group.

In addition, the optical logic calculation can also be carried out through diffraction neural networks using plane waves as incident signal [169]. The incident plane wave is spatially encoded by specific logical operations in the input layer. Then, the composite Huygens metasurface decodes by the hidden layer as shown in Figure 8(b) [67]. After well training, the coded light is scattered through a carefully designed metasurface to one small designated region of the output layer. The detected spot provides information on the output logic state, one representing the logic state “1” and the other representing the logic state “0.” Concretely, with a trained diffraction neural network, three kinds of logical operations (NOT, OR, AND) can be implemented at microwave frequencies. The disadvantage of this system is that only three kinds of logical calculation can be performed, and the types of logical calculation need to be improved.

Since the output result of the optical logic operator has only two states of “1” and “0,” this is very similar to the binary classification in deep learning architectures [170]. Therefore, optical logic operators can be constructed according to relevant machine learning theories. Similarly, a compact logical calculator based on a phase metasurface can also be realized by mapping nodes in a trained fully connected neural network (FCNN) [101]. When an incident wave illuminates a selected operating region of the metasurface, the corresponding cell is activated. Then, the wave scatters to two specified regions presenting a logical state in the output layer. After training the FCNN, the basic logical operation functions (AND, OR, NOT) can be implemented on the same metasurface, as shown in Figure 8(c). In addition, the artificial neural network with only one hidden layer is realized by using a single metasurface. That not only reduces the resources and time of training the network but also provides the route for the realization of compact photonic processors and integrated optical computing systems. The neural network utilizes only two optical layers, which provides remarkable improvements in training speed and computational resource usage.

Beyond that, the OAM mode as a kind of freedom is modulated by optical diffraction neural networks (ODNNs) [155], [171], [172] demonstrates good information processing light field modulation and capabilities [168]. The logical operation principal diagram of OAM mode based on ODNN is shown in Figure 8(d). The model solves the linear response of multiple optical fields through the connection of multi-layer phase and amplitude. It accurately realizes seven basic binary logic operations and half adder of AND, OR, NOT, NAND, NOR, XNOR, and XOR gates in simulation. This method not only save the physical properties of the input beam but also can effectively complete and extend to multiple logical operations, which provides a potential solution for the practical application of optical digital computing. The disadvantage is that the logic gate is realized through the cascade metasurface, which requires high requirements in experiment and manufacture.

3.3.5 Mathematic function computation

A mathematic function can be decomposed into linear combinations of basic functions. Shi et al. designed an all-silicon monolayer coded metasurface grating to realize addition and subtraction operations [173]. Similarly, simple mathematical operations through metasurfaces have been implemented in recent years [174], [175]. Furthermore, Zhao et al. propose a compact metasurface-based platform driven by deep learning that is capable of performing four basic trigonometric operations at the speed of light [176]. The advantage is that the four trigonometric operations of EM based can be realized under specific input values, and the optimized visual output makes the operation results clear and recognizable. Furthermore, only one hidden layer is used in the diffraction neural network, which greatly reduces the resources and time required to train the network and improves the integration with other photonic systems. The proposed concept work can provide breakthrough inspiration for other optical computing work such as ultra-fast photonic signal processors.

The geometric pattern is from a finite number of basic patterns, thus also enabling beam shaping. Figure 9 graphically shows the generation of diffractive-multiplexed generalized vortex beam (GVB) arrays based on dielectric geometry Dammann vortex metasurface (DVM) [177]. The spatial intensity characteristics of each diffraction stage are no longer the traditional toroidal profile vortex beam, but a free-form distribution carrying OAMs. Through the Taylor series expansion of the vortex grating function, the azimuth phase distribution of each diffraction order can be obtained. Specifically, different configurations of the GVB are generated in different diffraction orders. Thus, the field pattern representing a linear combination of preset basis functions is encoded on the DVM, which can generate GVB arrays with arbitrary angular phase differential design flexibility. Therefore, the generated GVB beams differ from each other in different spatial diffraction orders rather than the regular rings. It provides an optical representation of the features of a mathematical calculation. Such a DVM could be a gateway to explore various optical computing applications.

Figure 9:

Scheme of DVM based on dielectric geometric metasurface. Diffraction channels (5 × 5) with different intensity patterns of GVBs under orthogonal circular polarization input and output condition. The basis function curves of differential adder and subtractor. Reproduced with permission [177]. Copyright 2022, American Association for the Advancement of Science.

3.3.6 On-chip optical computing

Optical computing has been demonstrated to enable significant improvements in terms of processing speeds and energy efficiency. Traditional photonic integrated circuit (PIC) devices based on optical waveguides are usually bulky and lack the ability to fully manipulate the light wave at the sub-wavelength scale for optical computing [178]. In recent years, researchers have made a preliminary exploration into the realization of optical computation through on-chip metasurfaces. On-chip metasurfaces build bridge waveguide optics and free-space optics.

A one-dimensional high-contrast transmit array (HCTA) lens defined on a silicon substrate on a standard insulator is proposed, which is a high transmission (<1 dB loss) and above 200 nm bandwidth [179]. By designing 1D HCTA, ultra-short, low-loss, and wide-band mode size converters and meta-systems are obtained to perform Fourier transforms and spatial differentiation. In addition, the cascaded HCTA is used to demonstrate the metasystem functions of Fourier transform and differentiation. The metasurfaces optical computing system has the potential to realize on-chip optical transformation and mathematical operation. Moreover, an integrated end-to-end photonic deep neural network (PDNN) is proposed in Figure 10(a) [180]. The PDNN performs image classification by directly processing the optical waves impacting an array of on-chip pixels as neurons. The PDNN chip was experimentally verified for two-class and four-class handwritten letters classification, achieving accuracies higher than 93.8 % and 89.8 %. Directly, clock-free processing of optical data using PDNN eliminates analog-to-digital conversion and the need for large memory modules. So the research also provides faster and more energy-efficient neural networks for the next generations of deep learning systems. In addition, Fu et al. proposed an on-chip optical diffractive optical neural network (DONN) based on a silicon-on-insulator platform that can perform machine learning tasks with high integration and low power characteristics [181]. The on-chip DONN has the advantages of all-optical passive operation and massive-scale neuron integration. The accuracy of DONN on 1-hidden-layer and 3-hidden-layer on Iris plant dataset classification task is 86.7 % and 90 %, respectively. Phase errors are generated during the fabrication process, the testing accuracy of this method without compensation is 56.7 % and 60.0 %, respectively, and its accuracy needs to be further improved.

Figure 10:

On-chip optical computing metasurfaces. (a) Schematic of the on-chip PDNN structure. (b) NNM trained to recognize handwritten digits. (c) Structure diagram of OCPU. (d) The OCPU simultaneously performs three different groups of convolutional operations using incoherent light. (a) Reproduced with permission [180]. Copyright 2023, Nature Publishing Group. (b) Reproduced with permission [182]. Copyright 2019, Optica Publishing Group. (c)–(d) Reproduced with permission [183]. Copyright 2023, Nature Publishing Group.

On the other hand, a nanophotonic neural medium (NNM) has been proposed to enable artificial neural computation in a continuous and layer-free fashion [182]. Figure 10(b) shows the NNM in action where a two-dimensional (2D) medium is trained to recognize grayscale handwritten digits. On the right side of the NNM, the optical energy concentrates in different locations depending on the image’s classification labels. Inside the NNM, the nanostructure generates strong interference and the light is directed to one of 10 output locations, with the output with the highest share of energy intensity classified as inferred. The computational density of the system is much higher than that of free space and on-chip optical neural networks, and the recognizable numbers achieve 0–9 recognition. However, the average accuracy of the system is more than 79 %, and its recognition accuracy needs to be improved.

In addition, a compact on-chip optical convolutional processing unit (OCPU) was manufactured on a low-loss silicon nitride platform to demonstrate its capability for large-scale integration [183]. The structure diagram of the designed OCPU is shown in Figure 10(c), the input vector is simultaneously modulated on the amplitude of four incoherent light waves with the same initial amplitudes via electro-optical modulation. Figure 10(d) shows that the OCPU simultaneously performs three different groups of convolutional operations using incoherent light. The OCPU includes two 4 × 4 multimode interference cells and four phase shifters and simultaneously performs a convolutional operation with three user-defined 2 × 2 real-valued kernels. Dynamic reconfiguration to extract the desired feature images is realized by tuning the four phase shifters (PSs). That’s experimental accuracy of ten-class of MNIST handwritten digits reached 92.17 %. And the proposed OCPU has the potential of large-scale on-chip integration by simply increasing the number of ports and wavelength division multiplexing at each port. Therefore, it provides the development direction for the optical computing of OCPU with higher processing speed and lower power consumption.