Band 12, Heft 5 - Special Issue: Neural network learning with photonics and for photonic circuit design

The prospect of massive parallelism of optics enabling fast and low energy cost operations is attracting interest for novel photonic circuits where 3-dimensional (3D) implementations have a high potential for scalability. Since the technology for data input–output channels is 2-dimensional (2D), there is an unavoidable need to take 2D-nD transformations into account. Similarly, the 3D-2D and its reverse transformations are also tackled in a variety of fields such as optical tomography, additive manufacturing, and 3D optical memories. Here, we review how these 3D-2D transformations are tackled using iterative techniques and neural networks. This high-level comparison across different, yet related fields could yield a useful perspective for 3D optical design.

The simultaneous advances in artificial neural networks and photonic integration technologies have spurred extensive research in optical computing and optical neural networks (ONNs). The potential to simultaneously exploit multiple physical dimensions of time, wavelength and space give ONNs the ability to achieve computing operations with high parallelism and large-data throughput. Different photonic multiplexing techniques based on these multiple degrees of freedom have enabled ONNs with large-scale interconnectivity and linear computing functions. Here, we review the recent advances of ONNs based on different approaches to photonic multiplexing, and present our outlook on key technologies needed to further advance these photonic multiplexing/hybrid-multiplexing techniques of ONNs.

Integrated neuromorphic photonic circuits aim to power complex artificial neural networks (ANNs) in an energy and time efficient way by exploiting the large bandwidth and the low loss of photonic structures. However, scaling photonic circuits to match the requirements of modern ANNs still remains challenging. In this perspective, we give an overview over the usual sizes of matrices processed in ANNs and compare them with the capability of existing photonic matrix processors. To address shortcomings of existing architectures, we propose a time multiplexed matrix processing scheme which virtually increases the size of a physical photonic crossbar array without requiring any additional electrical post-processing. We investigate the underlying process of time multiplexed incoherent optical accumulation and achieve accumulation accuracy of 98.9% with 1 ns pulses. Assuming state of the art active components and a reasonable crossbar array size, this processor architecture would enable matrix vector multiplications with 16,000 × 64 matrices all optically on an estimated area of 51.2 mm 2 , while performing more than 110 trillion multiply and accumulate operations per second.

The rapid development of artificial intelligence has stimulated the interest in the novel designs of photonic neural networks. As three-dimensional (3D) neural networks, the diffractive neural networks (DNNs) relying on the diffractive phenomena of light, has demonstrated their superb performance in the direct parallel processing of two-dimensional (2D) optical data at the speed of light. Despite the outstanding achievements, DNNs utilize centimeter-scale devices to generate the input data passively, making the miniaturization and on-chip integration of DNNs a challenging task. Here, we provide our perspective on utilizing addressable vertical-cavity surface-emitting laser (VCSEL) arrays as a promising data input device and integrated platform to achieve compact, active DNNs for next-generation on-chip vertical-stacked photonic neural networks. Based on the VCSEL array, micron-scale 3D photonic chip with a modulation bandwidth at tens of GHz can be available. The possible future directions and challenges of the 3D photonic chip are analyzed.

Emerging neuromorphic hardware promises to solve certain problems faster and with higher energy efficiency than traditional computing by using physical processes that take place at the device level as the computational primitives in neural networks. While initial results in photonic neuromorphic hardware are very promising, such hardware requires programming or “training” that is often power-hungry and time-consuming. In this article, we examine the online learning paradigm, where the machinery for training is built deeply into the hardware itself. We argue that some form of online learning will be necessary if photonic neuromorphic hardware is to achieve its true potential.

In recent years, the computational demands of deep learning applications have necessitated the introduction of energy-efficient hardware accelerators. Optical neural networks are a promising option; however, thus far they have been largely limited by the lack of energy-efficient nonlinear optical functions. Here, we experimentally demonstrate an all-optical Rectified Linear Unit (ReLU), which is the most widely used nonlinear activation function for deep learning, using a periodically-poled thin-film lithium niobate nanophotonic waveguide and achieve ultra-low energies in the regime of femtojoules per activation with near-instantaneous operation. Our results provide a clear and practical path towards truly all-optical, energy-efficient nanophotonic deep learning.

Excitable optoelectronic devices represent one of the key building blocks for implementation of artificial spiking neurons in neuromorphic (brain-inspired) photonic systems. This work introduces and experimentally investigates an opto-electro-optical (O/E/O) artificial neuron built with a resonant tunnelling diode (RTD) coupled to a photodetector as a receiver and a vertical cavity surface emitting laser as a transmitter. We demonstrate a well-defined excitability threshold, above which the neuron produces optical spiking responses with characteristic neural-like refractory period. We utilise its fan-in capability to perform in-device coincidence detection (logical AND) and exclusive logical OR (XOR) tasks. These results provide first experimental validation of deterministic triggering and tasks in an RTD-based spiking optoelectronic neuron with both input and output optical (I/O) terminals. Furthermore, we also investigate in simulation the prospects of the proposed system for nanophotonic implementation in a monolithic design combining a nanoscale RTD element and a nanolaser; therefore demonstrating the potential of integrated RTD-based excitable nodes for low footprint, high-speed optoelectronic spiking neurons in future neuromorphic photonic hardware.

Photonic reservoir computing has been intensively investigated to solve machine learning tasks effectively. A simple learning procedure of output weights is used for reservoir computing. However, the lack of training of input-node and inter-node connection weights limits the performance of reservoir computing. The use of multiple reservoirs can be a solution to overcome this limitation of reservoir computing. In this study, we investigate parallel and deep configurations of delay-based all-optical reservoir computing using semiconductor lasers with optical feedback by combining multiple reservoirs to improve the performance of reservoir computing. Furthermore, we propose a hybrid configuration to maximize the benefits of parallel and deep reservoirs. We perform the chaotic time-series prediction task, nonlinear channel equalization task, and memory capacity measurement. Then, we compare the performance of single, parallel, deep, and hybrid reservoir configurations. We find that deep reservoirs are suitable for a chaotic time-series prediction task, whereas parallel reservoirs are suitable for a nonlinear channel equalization task. Hybrid reservoirs outperform other configurations for all three tasks. We further optimize the number of reservoirs for each reservoir configuration. Multiple reservoirs show great potential for the improvement of reservoir computing, which in turn can be applied for high-performance edge computing.

We show that coherent laser networks (CLNs) exhibit emergent neural computing capabilities. The proposed scheme is built on harnessing the collective behavior of laser networks for storing a number of phase patterns as stable fixed points of the governing dynamical equations and retrieving such patterns through proper excitation conditions, thus exhibiting an associative memory property. It is discussed that despite the large storage capacity of the network, the large overlap between fixed-point patterns effectively limits pattern retrieval to only two images. Next, we show that this restriction can be uplifted by using nonreciprocal coupling between lasers and this allows for utilizing a large storage capacity. This work opens new possibilities for neural computation with coherent laser networks as novel analog processors. In addition, the underlying dynamical model discussed here suggests a novel energy-based recurrent neural network that handles continuous data as opposed to Hopfield networks and Boltzmann machines that are intrinsically binary systems.

Photonic neural networks are brain-inspired information processing technology using photons instead of electrons to perform artificial intelligence (AI) tasks. However, existing architectures are designed for a single task but fail to multiplex different tasks in parallel within a single monolithic system due to the task competition that deteriorates the model performance. This paper proposes a novel optical multitask learning system by designing multiwavelength diffractive deep neural networks (D 2 NNs) with the joint optimization method. By encoding multitask inputs into multiwavelength channels, the system can increase the computing throughput and significantly alleviate the competition to perform multiple tasks in parallel with high accuracy. We design the two-task and four-task D 2 NNs with two and four spectral channels, respectively, for classifying different inputs from MNIST, FMNIST, KMNIST, and EMNIST databases. The numerical evaluations demonstrate that, under the same network size, multiwavelength D 2 NNs achieve significantly higher classification accuracies for multitask learning than single-wavelength D 2 NNs. Furthermore, by increasing the network size, the multiwavelength D 2 NNs for simultaneously performing multiple tasks achieve comparable classification accuracies with respect to the individual training of multiple single-wavelength D 2 NNs to perform tasks separately. Our work paves the way for developing the wavelength-division multiplexing technology to achieve high-throughput neuromorphic photonic computing and more general AI systems to perform multiple tasks in parallel.

Permutation matrices form an important computational building block frequently used in various fields including, e.g., communications, information security, and data processing. Optical implementation of permutation operators with relatively large number of input–output interconnections based on power-efficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to all-optically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output field-of-view using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can all-optically perform arbitrarily selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in, e.g., security, image encryption, and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THz-bands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks.

Photonic reservoirs are machine learning based systems that boast energy efficiency and speediness. Thus they can be deployed as optical processors in fiber communication systems to aid or replace digital signal equalization. In this paper, we simulate the use of a passive photonic reservoir to target nonlinearity-induced errors originating from self-phase modulation in the fiber and from the nonlinear response of the modulator. A 64-level quadrature-amplitude modulated signal is directly detected using the recently proposed Kramers–Kronig (KK) receiver. We train the readout weights by backpropagating through the receiver pipeline, thereby providing extra nonlinearity. Statistically computed bit error rates for fiber lengths of up to 100 km fall below 1 × 10 −3 bit error rate, outperforming an optical feed-forward equalizer as a linear benchmark. This can find applications in inter-datacenter communications that benefit from the hardware simplicity of a KK receiver and the low power and low latency processing of a photonic reservoir.

In the reservoir computing literature, the information processing capacity is frequently used to characterize the computing capabilities of a reservoir. However, it remains unclear how the information processing capacity connects to the performance on specific tasks. We demonstrate on a set of standard benchmark tasks that the total information processing capacity correlates poorly with task specific performance. Further, we derive an expression for the normalized mean square error of a task as a weighted function of the individual information processing capacities. Mathematically, the derivation requires the task to have the same input distribution as used to calculate the information processing capacities. We test our method on a range of tasks that violate this requirement and find good qualitative agreement between the predicted and the actual errors as long as the task input sequences do not have long autocorrelation times. Our method offers deeper insight into the principles governing reservoir computing performance. It also increases the utility of the evaluation of information processing capacities, which are typically defined on i.i.d. input, even if specific tasks deliver inputs stemming from different distributions. Moreover, it offers the possibility of reducing the experimental cost of optimizing physical reservoirs, such as those implemented in photonic systems.

Photonic reservoir computing has been demonstrated to be able to solve various complex problems. Although training a reservoir computing system is much simpler compared to other neural network approaches, it still requires considerable amounts of resources which becomes an issue when retraining is required. Transfer learning is a technique that allows us to re-use information between tasks, thereby reducing the cost of retraining. We propose transfer learning as a viable technique to compensate for the unavoidable parameter drift in experimental setups. Solving this parameter drift usually requires retraining the system, which is very time and energy consuming. Based on numerical studies on a delay-based reservoir computing system with semiconductor lasers, we investigate the use of transfer learning to mitigate these parameter fluctuations. Additionally, we demonstrate that transfer learning applied to two slightly different tasks allows us to reduce the amount of input samples required for training of the second task, thus reducing the amount of retraining.

Analog photonic computing comprises a promising candidate for accelerating the linear operations of deep neural networks (DNNs), since it provides ultrahigh bandwidth, low footprint and low power consumption computing capabilities. However, the confined photonic hardware size, along with the limited bit precision of high-speed electro-optical components, impose stringent requirements towards surpassing the performance levels of current digital processors. Herein, we propose and experimentally demonstrate a speed-optimized dynamic precision neural network (NN) inference via tiled matrix multiplication (TMM) on a low-radix silicon photonic processor. We introduce a theoretical model that relates the noise figure of a photonic neuron with the bit precision requirements per neural layer. The inference evaluation of an NN trained for the classification of the IRIS dataset is, then, experimentally performed over a silicon coherent photonic neuron that can support optical TMM up to 50 GHz, allowing, simultaneously, for dynamic-precision calculations. Targeting on a high-accuracy and speed-optimized classification performance, we experimentally applied the model-extracted mixed-precision NN inference scheme via the respective alteration of the operational compute rates per neural layer. This dynamic-precision NN inference revealed a 55% decrease in the execution time of the linear operations compared to a fixed-precision scheme, without degrading its accuracy.

Multiport interferometers based on integrated beamsplitter meshes have recently captured interest as a platform for many emerging technologies. In this paper, we present a novel architecture for multiport interferometers based on the sine–cosine fractal decomposition of a unitary matrix. Our architecture is unique in that it is self-similar, enabling the construction of modular multi-chiplet devices. Due to this modularity, our design enjoys improved resilience to hardware imperfections as compared to conventional multiport interferometers. Additionally, the structure of our circuit enables systematic truncation, which is key in reducing the hardware footprint of the chip as well as compute time in training optical neural networks, while maintaining full connectivity. Numerical simulations show that truncation of these meshes gives robust performance even under large fabrication errors. This design is a step forward in the construction of large-scale programmable photonics, removing a major hurdle in scaling up to practical machine learning and quantum computing applications.

Programmable feedforward photonic meshes of Mach–Zehnder interferometers are computational optical circuits that have many classical and quantum computing applications including machine learning, sensing, and telecommunications. Such devices can form the basis of energy-efficient photonic neural networks, which solve complex tasks using photonics-accelerated matrix multiplication on a chip, and which may require calibration and training mechanisms. Such training can benefit from internal optical power monitoring and physical gradient measurement for optimizing controllable phase shifts to maximize some task merit function. Here, we design and experimentally verify a new architecture capable of power monitoring any waveguide segment in a feedforward photonic circuit. Our scheme is experimentally realized by modulating phase shifters in a 6 × 6 triangular mesh silicon photonic chip, which can non-invasively (i.e., without any internal “power taps”) resolve optical powers in a 3 × 3 triangular mesh based on response measurements in only two output detectors. We measure roughly 3% average error over 1000 trials in the presence of systematic manufacturing and environmental drift errors and verify scalability of our procedure to more modes via simulation.

Multilayer films with continuously varying indices for each layer have attracted great deal of attention due to their superior optical, mechanical, and thermal properties. However, difficulties in fabrication have limited their application and study in scientific literature compared to multilayer films with fixed index layers. In this work we propose a neural network based inverse design technique enabled by a differentiable analytical solver for realistic design and fabrication of single material variable-index multilayer films. This approach generates multilayer films with excellent performance under ideal conditions. We furthermore address the issue of how to translate these ideal designs into practical useful devices which will naturally suffer from growth imperfections. By integrating simulated systematic and random errors just as a deposition tool would into the optimization process, we demonstrated that the same neural network that produced the ideal device can be retrained to produce designs compensating for systematic deposition errors. Furthermore, the proposed approach corrects for systematic errors even in the presence of random fabrication imperfections. The results outlined in this paper provide a practical and experimentally viable approach for the design of single material multilayer film stacks for an extremely wide variety of practical applications with high performance.

Silicon photonics enables compact integrated photonic devices with versatile functionalities and mass manufacturing capability. However, the optimization of high-performance free-form optical devices is still challenging due to the complex light-matter interaction involved that requires time-consuming electromagnetic simulations. This problem becomes even more prominent when multiple devices are required, typically requiring separate iterative optimizations. To facilitate multi-task inverse design, we propose a topology optimization method based on deep neural network (DNN) in low-dimensional Fourier domain. The DNN takes target optical responses as inputs and predicts low-frequency Fourier components, which are then utilized to reconstruct device geometries. Removing high-frequency components for reduced design degree-of-freedom (DOF) helps control minimal features and speed up training. For demonstration, the proposed method is utilized for wavelength filter design. The trained DNN can design multiple filters instantly and concurrently with high accuracy. Totally different targets can also be further optimized through transfer learning on existing network with greatly reduced optimization rounds. Our approach can be also adapted to other free-form photonic devices, including a waveguide-coupled single-photon source that we demonstrate to prove generalizability. Such DNN-assisted topology optimization significantly reduces the time and resources required for multi-task optimization, enabling large-scale photonic device design in various applications.