Scalable and compact photonic neural chip with low learning-capability-loss

1 Introduction

Neural-like computation is desired because the neural system possesses much higher performance and lower energy consumption than current computers based on Von Neumann architecture for a wide range of tasks like perception communication, learning and decision making [1], [2], [3]. Motivated by these superiorities of the neural-like computation, artificial neural network (ANN, Figure 1a) is becoming increasingly attractive as a powerful tool to solve a large class of problems from face recognition to natural language processing [1], [2], [3], [4], [5]. However, the implementation of ANN is an ultra-computationally expensive task. Especially, it requires dense matrix computation [1, 6, 7]. Conventional digital electrical instantiation of matrix unit typically suffer from high communication overheads, expensive digital multiply-accumulate operations (MAC), high latency [6, 7]. Moreover, due to the slow-down of the size shrinking of the transistor driven by Moore’s Law, the performance of the digital matrix unit is approaching its physical-limitation, and become increasingly difficult to fulfill the needs of the swift-developing ANN technology on computation speed and power efficiency brought by larger model size and bigger data volume [1]. Alternatively, photonic processing is suggested as a candidate beyond Moore’s Law for dense matrix computation with stark advantages in bandwidth density, latency [6], [7], [8], [9]. As such, it is becoming increasingly attractive that photonic neural network (PNN, Figure 1b) can obtain lower energy cost but at much higher clock rate to the digital electronic counterpart, and several implementations, including coherent nanophotonic circuits, diffraction optics, photonic comb, etc., are developed [4, 5, 8, 10], [11], [12], [13], [14], [15].

Figure 1:

The schematic illustrition of the SVD mesh and pseudo-real-value mesh for matrix-multiplier in PNN.

(a) General artificial neural network architecture composed of an input layer, a number of hidden layers and anoutput layer. (b) Decomposition of the general neural network into individual layers implemented by the optical matrix multiplier and nonlinearity units that compose each layer of the artificial neural network. Most operations are the performed on the optical matrix multiplier, which conventionally can be realized by (c) SVD-mesh containing N ² 2 × 2 MZIs. (d) Schematic illustration of the pseudo-real-value matrix unitary MZI mesh for matrix expression; a schematic of (e) universal GridUnitary mesh, (f) nonuniversal P-GridUnitary mesh, and (g) nonuniversal FFTUnitary mesh. (Meaning of the abbreviation in figure, PD: photo diode; MMI: multimode interference; DC: directional coupler.)

Practically, among a plenty of photonic processing architectures [5, 8, 12, 13, 15], there has been much progress towards “universal linear optics” architecture: photonic circuits that can be defined to execute all possible linear optical matrix transformations on a given set of input modes [8, 16]. Such systems are typically built using planar meshes of beam splitters, which are easy to fabricate and to individually control [8, 17]. Basically, the optical matrix multiplier based on such mesh can be deemed as the cascaded interconnections of several 2 × 2 MZIs (Figure 1c) with some types of geometrical topology to realize unitary linear transformation [16], [17], [18]. Furthermore, utilizing the well-known singular value decomposition (SVD) algorithm (Figure 1c) [5]:

i.e., arbitrary real-value matrix multiplier can be factorized by two unitary matrixes U,V expressed by MZIs using Reck’s (or Clement’s) scheme, and a diagonal matrix Σ that can also be represented by MZIs. Such that the computationally expensive MAC can be operated on optical domain. However, at least N ² 2 × 2 MZI cells are needed to realize arbitrary N × N matrix (Figure 1c). The square increase of the MZI number versus matrix dimension strongly limits the scalability of such architecture. It is possible to substitute the universal unitary mesh with nonuniversal one requiring less MZIs to improve the scalability [17, 19], but the cost is the impaired matrix expressivity, and accordingly degraded learning-capability in implemented PNN [17].

Moreover, due to the diffraction-limitation, the photonics device is almost impossible to shrink its size like the electrical device (e.g., the transistor) beneficial from upgrading the process node. The plasmonic devices might break the diffraction limitation and achieves subwavelength dimension [20], [21], [22], [23], but so far, they are too lossy to integrate on large scale. Hence, large scale photonic circuit integration would be restricted by the fast expansion of the chip size, e.g., Lightmatter’s Mars device integrated 64 × 64 MZI with nano-optical-electro-mechanical-system as the phase-shifting elements on a 150 mm² chip, and accordingly the predicted chip size for 1024 × 1024 would be prohibitively high as 384 cm² and difficult to realize [19]. Besides, the energy efficiency of current photonic matrix unit utilizing thermal or electrical phase shifter seems not as low as expected, e.g., the power required to maintain the state of the MZI by thermal-optical phase shifter is typically from few to tens mW per shifter [6, 7, 24], which, in some cases, may account for ∼87% of total energy consumption in an intact photonic computing system [24]. Hence it is rather challenging for presented photonic processing architecture to copy the success of Moore’s law on electronic computing to upgrade the performances by reducing the device size as well as consumed power. Whereas, it is more significant to exploring superior architecture to overcome the scale-limitation of current experimental demonstrated PNN as well as by-produced bottleneck on chip size and energy consumption.

In this article, we propose a scalable architecture with reduced MZIs requirement least at O(Nlog₂ N) level. Other than regular mesh design containing N ² MZIs to construct two universal unitary matrixes U, V and one diagonal matrix Σ given by SVD algorithm to express arbitrary N*N real-value matrix, our scalable design, namely, pseudo-real MZIs mesh employs the real-part of an unitary mesh to learn the real-value matrix, (i.e., the mesh itself is truly programmed as a complex-value matrix, but works as a real-value matrix multiplier whose operation only depends on its real part). Theoretically, our method could construct 2 ^N available unitary matrixes whose real part is exactly the desired real-value matrix to learn. Hence, a nonuniversal mesh with few MZI cells might be good enough only if one of these 2 ^N available unitary matrixes can be expressed by this nonuniversal mesh. Such that the pseudo-real-value mesh could reduce required MZI cells (thus lower power and smaller chip size) to provide better scalability to high radix. In the applications like PNN, we use the most compact design consuming only 0.5*N*log₂ N MZIs, pseudo-real-value FFTUnitary MZI mesh, to show the feasibility to pay low cost on accuracy loss while apparently reduce the required photonic devices. Experimentally, we implement an 4 × 4 photonic neural chip and benchmark its performance in convolutional neural network for handwriting recognition task, which achieve the accuracy fully comparable to regular SVD-mesh as well as current digital computer.

2 Principles of pseudo-real-value PNN

The key thought of our design is that we do not try to find the expressions to two universal unitary matrix U,V and a diagonal matrix Σ separately as done in conventional SVD-based method to represent given real-value matrix M, but construct a matrix U whose real part fulfill that

and then the real part of the output E _out, which is produced from the amplitude-modulated input modes E _in multiply with U, would be equivalent to the target multiply-operation ME _in with only a difference of coefficient α, i.e.,

where the extraction of the real part of the output can be simply realized by interfering the output with a fixed reference source light. As such, infinitely many complex matrixes U satisfying Eq. (3) could be used for the target multiply-operation since the image part of U could be arbitrary. This is why we call our design as pseudo-real MZIs mesh (i.e., the designed U itself is a complex-value matrix unit, but truly enables a real-value matrix multiplier operation depending on its real part). Consequently, even a less-expressive mesh might be good enough only if it could effectively approach one of the (infinitely many) available matrixes.

Specified as shown Figure 1d, let us set the light source as two parts, one part with a ratio µ of the total intensity of the input light source I _in is used for excite the intensity modulated input modes as done in conventional SVD-mesh-based analog-matrix multiplication. The other part, i.e., (1 − μ)I _in is used as the reference sources to interfere with the output modes. As such the interference between the reference sources E _ref and output modes E _out produce homodyne signals as

where ϕ _ref and ϕ _out are the phase of reference sources and output modes. So the balanced detection to the homodyne signals allows us to extract the real part of the output if setting the initial phase of the reference sources as 0 (see SI-1 for further discussions):

i.e., when the input modes are purely amplitude-modulated, the real part of the output modes can be seen as the multiplication of the input with the real part of the matrix represented by the MZIs mesh other than the matrix itself as done in conventional methods [5, 8, 16, 17]. Hence no matter what the image part of the matrix expressed by the MZIs mesh is, if Re(U) is exactly the target real-value matrix M we would like to define, actually we have executed the desired matrix multiplication according to Eq. (3). Besides, unlike the regular intensity-detection scheme whose photonic matrix elements is always positive, here the pseudo-real MZI mesh architecture intrinsically support both positive- and negative-value calculation. Furthermore, since Im(U) is arbitrary, we can set the Im(U) as that fulfill the property of U = Re(U) + jIm(U) and at the same time U is a unitary matrix. Therefore, other than using two unitary matrixes and one diagonal matrix need in conventional SVD-based mesh design (Figure 1c), one unitary matrix is enough to implement any given matrix multiplication only if we have a general algorithm to construct the unitary matrixes whose real part are just the target real-value matrix. The desired algorithm is as following:

1. Decompose the target real-value matrix by SVD as M = βUΣV [17], here β could be a parameter depending on light power, mesh loss and detection gain, which is rational configured to ensure every singular value σ _ii (i = 1, 2, … n) in diagonal Σ is positive but less than 1, while U and V could be two real-value unitary matrix. Without the loss of the generality, we choose β = 1 in the following to simplify the discussion. Take the case of matrix radix N = 4 as an instance, we have

1. We mathematically construct a new matrix M′ = UΣ′V whose singular value σ _ii ′ (i = 1, 2, … 4) is correlated to the count part in M with ( σ i i ) 2 + ( σ i i ′ ) 2 = 1 .

2. The matrix M + iM′ would be a designed unitary matrix U _D with M = Re(U _D), if noting (M + iM′) (M + iM′) ^H = U(Σ + iΣ′)VV ^H (Σ + iΣ′) ^H U ^H  = I (here, I is the identity matrix). Hence the pseudo-real mesh design could achieve the minimal optical depth, requiring less than half the depth of the SVD-based design – other than three MZI mesh for U, Σ and V, only one unitary mesh U used to express M + iM′ is enough. This seems valuable for minimizing optical losses and reducing fabrication resources [16]. For example, based on the MZI unit cell shown in Figure 1c, the GridUnitary mesh (Figure 1e) has universal expressivity to any unitary matrix via Clement’s algorithm [5, 25]. Accordingly, when using pseudo real-value design with GridUnitary mesh to implement arbitrary real value matrix [17], the needed MZI cells could decrease from N ² in conventional SVD mesh design to N(N − 1)/2.

Furthermore, it is noteworthy that for each σ _ii and σ _ii ′, their sign could be either same or opposite. Therefore, for a given real value N*N matrix M, we can construct at least 2 ^N unitary matrixes whose real part equal M. That is to say, in principle, a nonuniversal mesh with few MZI cells, might actually have sufficient capability to represent any real value N*N matrix M when using pseudo-real mesh design only if one of these 2 ^N available unitary matrixes could be expressed by this nonuniversal mesh, e.g., a partial-GridUnitary mesh (P-GridUnitary, Figure 1f) built by removing the last stage of a standard (4 × 4) GridUnitary mesh or the FFTUnitary mesh (Figure 1g) proposed in [17, 26], while the performance degradation due to the usage of less-expressive unitary mesh in conventional SVD-based methods might be highly relieved [17, 25].

3 Learning-capability of the pseudo-real-value architecture

Matrix-vector-product (MVP) is the most frequently used and computationally expensive operations in the neural network algorithms [13, 15, 27]. Hence in this section, we study the real-value matrix expressivity of aforementioned pseudo-real-value design with universal and nonuniversal MZI mesh at first. Afterward, considering for the specific tasks, the primary figure of merit is the classification or detection accuracy but not the matrix-expressivity accuracy, we study their performances in the specific neural network models for different dataset to evaluate the learning capability loss to using pseudo-real-value nonuniversal unitary mesh for PNN tasks.

3.1 Real-value matrix expressivity

3.1.1 Numerical evaluations for small radix matrix

We numerically study the matrix expressivity of aforementioned pseudo-real-value design with universal and nonuniversal MZI mesh. Exactly, due to the lack of effective analytical or numerical decomposition algorithms for nonuniversal mesh like P-GridUnitary and/or FFTUnitary mesh to approach given unitary matrix, it is difficult to analytically obtain the global optimal for the pseudo-real nonuniversal mesh to express arbitrary random sampled real-value matrix [17]. Even though, for small matrix dimension N (N = 4), the mature methods in machine learning like stochastic gradient descent (SGD) still achieve satisfying local optimal. Specified, we generate 10,000 random arbitrary target 4 × 4 real matrices, and each real matrix could be the real part of at least 2⁴ = 16 kinds of unitary matrices, accordingly it would produce 160,000 available unitary matrices. Afterward, we use the GridUnitary, P-GridUnitary and FFTUnitary mesh to approach these 160,000 unitary matrices via SGD method (SI-2), and, respectively, founds 100, 95.2 and 89.1% average accuracy (defined as Tr(Re(U _tar*U _app ^H )), where U _tar is the available unitary matrix as the target, U _app is the obtained unitary matrix approached by SGD) as shown in Figure 2a–c. But, considering only one unitary-matrix whose real part is equal to the target real matrices is enough, we practically choose the best one from 16 corresponding approached unitary-matrices as the final solution. Such that, the statistical accuracy to approach the random arbitrary target real matrices by pseudo-real-value matrix with nonuniversal P-GridUnitary and FFTUnitary mesh design would increase to 98.1 and 97.3%, much better than the performance of P-GridUnitary and FFTUnitary mesh themselves on approaching unitary matrices (Figure 2d–f). In addition, as a baseline, these 10,000 random 4 × 4 real matrices can be also learned by conventional SVD architecture. Considering for regular SVD mesh, the diagonal matrix Σ can be always learned correctly, the learning-performances is fully dependent on the expressivity of the universal or nonuniversal unitary mesh on unitary matrix U and V. Accordingly, for an given matrix, we use Tr(Re(U _tar*U _app ^H )*Tr(Re(V _tar*V _app ^H ) to evaluate the performances of SVD architecture with universal GridUnitary, or nonuniversal P-GridUnitary and FFTUnitary mesh as shown in Figure 2g–i, which, respectively, produces 100, 89.1 and 79.0% average accuracy. That is to say, in the conventional SVD-architecture (Figure 1c), which contains two unitary mesh, substituting the universal unitary meshes to nonuniversal ones would highly impair the expressivity to the real-value matrix due to their remarkable lower fidelity. Whereas, for the pseudo-real-value matrix design, the accuracy-loss brought by using the nonuniversal unitary mesh to represent the real-value matrix is apparent less because of the much amplified feasible solution space. Moreover, considering the high error tolerance of ANN allowing inference at low precision of 4–5 bit [28], [29], [30], even for the 4 × 4 FFTunitary mesh with least MZIs (only 25% MZIs consumption comparing to universal 4 × 4 SVD mesh with 16 MZIs), the expressivity error of ∼2.7% is still lower than the quantization error. Besides, the nonuniversal unitary mesh has shallower optical depth, which is important to minimize optical losses and enhance the robustness to the fabrication imperfection [25], as well as reduce calibration complexity.

Figure 2:

The statistical accuracy distribution of expressing 4*4 unitary matrix by (a) GridUnitary mesh, (b) P-GridUnitary mesh, and (c) FFTUnitary mesh; the statistical accuracy distribution of expressing real-value matrix by real-part of (d) GridUnitary mesh, (e) P-GridUnitary mesh, and (f) FFTUnitary mesh. The statistical accuracy distribution of expressing real-value matrix by conventional SVD matrix with (g) GridUnitary mesh, (h) P-GridUnitary mesh, and (i) FFTUnitary mesh as the unitary matrix operator.

It seems promising of the pseudo-real-value architecture to reduce the MZIs requirement and accordingly improve the scalability while maintain satisfying learning capability for radix N = 4. To further check the convergence property of the proposed pseudo-real-value optimization approach, we explore the availability on larger radix. To simplify the discussion, we focus on the pseudo-real-value architecture with FFTUnitary mesh or its stacking (Stacked-FFTUnitary mesh), because FFTUnitary mesh consumes the minimal number of MZIs at O(N*log₂ N) level to make all input modes interact with each other and consequently might approach arbitrary unitary matrices [31]. Hence, the pseudo-real FFTUnitary mesh might be a promising candidate with most effective-use of the MZIs, while the intrinsic advantage of FFTUnitary mesh on fabrication imprecision tolerance maintains [17]. To ensure log₂ N is an integer, N should be 4, 8, 16, etc. Hence, as the extension to aforementioned study to N = 4 case, the expressivity of 8*8 FFTUnitary and Stacked-FFTUnitary consists of FFTUnitary multipliers stacked end-to-end 2 times are investigated as shown in Figure 3. These two meshes are both nonuniversal and consume MZIs less than the universal one. It found that 8*8 FFTUnitary and Stacked-FFTUnitary mesh could, respectively, achieve 74.6 and 93.1% accuracy on learning 25,600 unitary matrix constructed from 100 8*8 random real-value matrixes (Figure 3a and b), and the minima-error one picked from 256 candidates could optimize the statistical accuracy to 86.1 and 97.9% (Figure 3c and d, although the distribution profile is not very smooth due to limited 100 sample times). Therefore, the results of both 4*4 and 8*8 matrixes support the feasibility of using pseudo real-value nonuniversal mesh with O(N*log  N) MZIs to learn real-value matrix accurately. For both N = 4 and N = 8 cases, it is feasible to use reduced MZIs (4 cells and 24 cells, respectively) to achieve good-learning (e.g., accuracy > 97%) comparable to conventional SVD design (16 cells and 64 cells, respectively), indicating the improved scalability of the pseudo-real-value nonuniversal unitary mesh architecture with low matrix expressivity loss. However, the learning for cases of N = 16 or more becomes quite time-consuming on our computer (SI-2): as N increases, the learning time would grow exponentially since we need to train 2 ^N times to traverse the approaches of the given mesh to all available unitary matrices and range their representation error to pick out the minimal-error one. In future, it is quite desired to develop more efficient approaching methods.

Figure 3:

The statistical accuracy distribution of expressing 8*8 unitary matrix by (a) FFTUnitary mesh and (b) stacked-FFTUnitary mesh; the statistical accuracy distribution of expressing minimal-error one picked for real-value matrix based on pseudo-real-value (c) FFTUnitary mesh and (d) stacked-FFTUnitary.

Although the picking process consumes exponentially-increased training time and becomes not so practical for large N, the fidelity distribution for the mesh to learn arbitrary unitary matrix allow us to track the mathematic expectation of the minimum error under some rational assumptions. Here, we introduce a fidelity-dependent error E _F: E _F = 1 − fidelity. Then the picking process proposed above would be something like estimating the mathematic expectation of the minimum of the 2 ^N samples for the random variable following E _F distribution, whose profile can be tracked from numerically simulated fidelity distribution. When the mesh is with few layers (e.g., 8*8 FFTUnitary case shown in Figure 3a), the approaching is equivalent to embedding the N × N matrix into a (small) N*log₂   N subspace, hence the (large) ∼2 ^N sampling times imply that the distribution should be approximately Gaussian according to central-limit-theorem. But for the mesh with more layers (e.g., 8*8 Stacked-FFTUnitary case shown in Figure 3b), E _F distribution is asymmetric with a most probable value relative close to low-error (high fidelity) side. Therefore, the mathematic expectation of the minimum sampled value for such distribution should be lower than the counterpart calculated for Gaussian distribution. Even though, the error-expectation calculated for Gaussian distribution is still a good estimation to its upper bound. Likewise, the calculation for uniform distribution could restrict the lower-bound of the error-expectation (SI-3). This might be valuable for us to understand the expressivity of the mesh with very large N considering the difficult on numerical simulating the matrix-learning at high radix through existing minima-error picking route.

3.1.2 Discussions to the matrix expressivity at large-N limitation

Although it is challenge to numerical-study the matrix expressivity of pseudo-real value nonuniversal unitary mesh by aforementioned minima-error picking method at large radix N, the qualitative discussions to this issue are still of significance. More specified, how about the least MZIs needed for a nonuniversal unitary mesh to produce good enough expressivity to arbitrary dense (real-value) matrixes? And furthermore, since the weight kernels (matrixes) used in neural network applications concerned in this work usually tend to be sparse compare to random data and probable embedded to low-dimensional spaces [32], it is also of interest to see how about the least MZIs needed for a nonuniversal unitary mesh to express sparse matrix?

For the N*N matrix multiplication based on MZI mesh, it is physically implemented by feeding in N sequential 1 × N column vector into the mesh and detect the outputs. Accordingly, if the MZI used by the mesh is n _MZI, then the operation number to complete the matrix multiplication on optical domain should be on O(n _MZI*N) level. Obviously, on the viewpoint of the computation complexity, we have:

(7) O ( N 2 ) < O ( n MZI ∗ N ) < O ( N 3 )

where O(N ³) is given by the naive (‘schoolbook’) algorithm, and O(N ²) is based on the fact that any algorithm must output N ² entries. Consequently,

(8) O ( N ) < O ( n MZI ) < O ( N 2 )

It is straightforward that based on Reck and Clemamnts’s method, the pseudo-real-value architecture proposed here can use N(N − 1) MZIs to represent arbitrary dense (real-value) matrixes [15, 18, 33]. However, the advances on optimizing the computation complexity of the matrix multiplication may allow us see more. Very recently, the computational complexity of the multiplication between two N*N matrices was optimized to from O(N ³) to O(N ^2.3728595) by Alman and Williams [34], indicating the possibility to accordingly approach arbitrary dense matrix with sub-O(N ²) MZIs (e.g., we may achieve O(N ^1.3728595) circuit complexity via photonic mapping to such optimized algorithm of the matrix multiplication). Moreover, Ran Raz proved a lower-bound to the computational complexity of the matrix multiplication as O(N ²log  N) [35], but no one has developed the algorithm to reach this bound so far. However, if relax the requirement from dense matrix to sparse matrix, Likhosherstov et al., from Google demonstrate the feasible approaching [32]: They theoretically proved based on the Johnson-Lindenstrauss lemma [36], and experimental verified the log-dimension embedding for the expressive power of (sparse) self-attention matrices. As such, the sparse matrix might be approximated by mesh with the least MZIs at O(N*log  N) level. Based on these analysis, we further suggest the relationship between the matrix expressivity and the required MZIs of the pseudo-real-value mesh (SI-4), which may explain why the mesh with O(N*log  N) MZIs could approach arbitrary real-value matrix with small radix (e.g., N = 4 and 8) accurately as shown above (Figures 2f and 3d), but would degrade to just feasible for sparse matrix in the large-N limit as found in [32]. Anyway, further investigations to the matrix expressivity of pseudo-real-value nonuniversal unitary mesh are still highly desired.

4 Experimental implementation PNN on pseudo-real-value nonuniversal mesh

4.2 Photonic neural chip: experimental implementation and performance evaluation

The trained LeNet-5 network is implemented on an interference system based on silicon photonics chip including four-modulator array for (pure amplitude) data-input, pseudo-real-value matrix with P-GridUnitary mesh design and four balanced detector array to receive output-data. Such that all 3 × 3 and 4 × 4 kernels in convolution layers are performed on optical domain, while the nonlinear activations and full connection layers are executed in FPGA (Figure 6a–c). Experimentally, the interference system achieves 99.40% accuracy on MNIST test set including 10,000 instances (Figure 6d), while the MZI black dash marked in Figure 4c is configured to cross state, the P-GridUnitary mesh become equivalent to a FFTUnitary mesh, and then the test accuracy is 98.87% (Figure 6e). They are both very close to the training performance, indicating the low encoding error level of pseudo-real-value nonuniversal mesh for matrix expression. During the whole interference process, the classification accuracy is quite stable at high level without apparent deviation (see Video-1), whereas in conventional SVD-mesh design, much more consumed MZIs produce more phase-error and thermal crosstalk, resulting in apparent practical nonideal [5]. Additionally, we demonstrate the recognition of realistic hand-written digits captured by the camera (Figure 6f), further verifying the robustness of the PNN based on pseudo-real-value nonuniversal mesh when running in realistic world.

Figure 4:

The performances of PNN on LeNet-5 model for MNIST task using pseudo-real-value GridUnitary, P-GridUnitary and FFTUnitary mesh, as well as their comparision to SVD GridNet.

(a) The evolution of the MNIST accuracy of LeNet-5 model based on 4*4 pseudo-real GridUnitary (green), P-GridUnitary (red) and FFTUnitary mesh (black), as well as conventional SVD-GridNet mesh with 40 epoch training; confusion matrix of trained model on test set with 10,000 instances based on pseudo-real (b) GridUnitary, (c) P-GridUnitary and FFTUnitary mesh, as well as conventional SVD-GridNet mesh. Each column of the matrix represents the instances in predicted label, while each row represents the instances in true label.

Figure 5:

The performances of PNN on MobileNet model for Fashion-MNIST task using pseudo-real-value FFTUnitary mesh, as well as their comparision to the baseline uisng the reuglar universal real-value mesh.

(a) The evolution of the Fashion-MNIST accuracy of MobileNet model; Confusion matrixes of (a) baseline prediction and (b) pseudo-real-value FFTUnitary mesh.

Figure 6:

The system implementation of the PNN using pseudo-real-value MZI mesh.

(a) PNN inferences system based on LeNet-5 network implemented on (b) silicon photonics chip with pseudo-real-value nonuniversal MZI mesh design, where (c) a P-GridUnitary mesh where 10 thermal shifters is employed to program five MZIs within (shifter 1 and 2 are θ and ϕ for 1st MZI, likewise, 4 and 5 for 2nd, 6 and 7 for 3rd, 3 and 8 for 4th, 9 and 10 for 5th). Tested MNIST accuracy when the chip works as (d) P-GridUnitary mesh or (e) FFTUnitary mesh. The demonstration of recognition of realistic hand-written digits captured by the camera.

During the inferences, the voltage values used to configure the 102 kernels in the two convolutional layer of our train LeNet-5 model based on pseudo-real-value P-GridUnitary mesh and FFTUnitary mesh is shown in Figure 7a and b, respectively. We combine the applied voltage with the resistance values (∼1.25 kΩ, SI-5) of the thermal shifter to calculate the power needed to configure all kernels. Hence, statistically, the averaged power consumption for the matrix programming is 134 mW for P-GridUnitary mesh and 108 mW for FFTUnitary mesh (if not accounting the power of the phase shifter to fix the marked MZI at cross state) as shown in Figure 7c and d). To further evaluate the potential of our pseudo-real-value mesh for high speed processing, as shown in Figure 8a–c, we demonstrate the 4 × 4 pseudo-real mesh working as the photonic matrix multiplier at high time clock frequency of 25 GHz, corresponding to the computation capability of 0.8TOPS. Considering the obtained averaged matrix programming power (note it is independent on the time clock frequency), as well as the typical power of the driver chip (Tx ∼ 1030 mW), the TIA chip (Rx ∼ 130 mW/Channel) and the laser (∼9 mW), the experimental energy efficiency is ∼0.48 TOPS/W. Furthermore, noting the possible low limit of the electrical power on modulator, photodetector and laser could be less than 1 mW/Channel [30], as well as the typical energy efficiency of digital–analog-convertor (DAC) and analog–digital-convertor (ADC) for photonic computing at ∼5.5 mW/Gsps [44], the potential energy efficiency of pseudo-real-value PNN with nonuniversal FFTUnitary mesh design is promising to reach ∼1.3 TOPS/W or better, even using the energy-inefficient thermal phase shifter as the weight element.

Figure 7:

Voltage setting to configure the 102 kernels in the two convolutional layer of LeNet-5 model based on (a) pseudo-real-value P-GridUnitary mesh and (b) FFTUnitary mesh. Power consumed during configuring the kernel by pseudo-real-value (a) P-GridUnitary mesh and (b) FFTUnitary mesh.

Figure 8:

The feasibility of the pseudo-real-value MZI mesh to run matrix multiplication at high time-clock.

(a) Packaged active silicon photonic chip with pseudo-real-value matrix multiplier; (b) Microscopy of the chip highlight the modulators array, balanced photon detectors (BPD) and MZI mesh; (c) matrix multiplication at 25 GHz demonstrated via eye diagram test.

5 The scalability of the pseudo-real-value PNN

As shown in aforementioned sections, it is promising of our pseudo-real-value nonuniversal unitary mesh with MZIs least at O(N*log₂ N) levels for PNN applications. In some particular tasks, including LeNet-5 for MNIST dataset, and MobileNet for Fashion-MNIST dataset, a most compact design using 0.5N*log₂ N MZIs, pseudo-real-value FFTUnitary meshes, with either small or large radix N shows acceptable low classification accuracy loss. Also, we experimentally implement an inference system based on 4*4 pseudo-real-value nonuniversal unitary MZI mesh to show the feasibility of this architecture. Hence it is straightforward to substitute the conventional SVD MZI mesh with pseudo-real FFTUnitary mesh as the building block for weight matrix multipliers in the applications like PNN, then the computational power and chip area needed for mesh-construction as well as the calibration complexity are reduced remarkably, scaling with the radix of the matrix N as Nlog₂ N/2. As such, highly differing from the conventional electrical matrix multiplier which upgrades the performances via scaling-down of the transistor size and are facing performance-limitation due to the ending of Moore’s Law, the improvement of the photonic matrix multiplier for ANN applications can be beneficial from the scaling-up of the dimension N of the matrix represented by the pseudo-real nonuniversal unitary MZI mesh. Especially, there could be a rather high improvement space since the size of a die could be far larger than that of an MZI cell, allowing ultra large scale, e.g., pseudo-real FFTUnitary mesh. The scale laws of the chip size, optical loss, programming power and encoding error for PNN based on SVD-based GridNet mesh, pseudo-real-value universal pseudo-real GridUnitary mesh and nonuniversal pseudo-real FFTUnitary mesh are listed in Table 1 (SI-7). Quantitatively, the performances of these three kinds of mesh are evaluated and compared in Figure 9a–d assuming using state of art photonic components [17, 30, 45], [46], [47] (SI-7). It clearly present the all-round advantages of pseudo-real FFTUnitary mesh especially when the radix N increases. However, for the case of small radix N, the intrinsic large size of MZI (∼8000 μm²) make even the pseudo-real FFTUnitary mesh area-inefficient comparing to the photonics matrix multiplier based micro ring resonator (∼250 mm²) [29]. This implies that it is still quite significant to the optimize the photonic device, e.g., shrinking the size of MZI via optimized design as well as introducing the silicon-photonics-process compatible but more efficient phase shifter [48], besides the exploration on architecture-improvement. They co-open the room to upgrade the performances of the photonic chip for neural network applications.

Figure 9:

Comparison of the (a) mesh size, (b) insertion loss, (c) power consumption, (d) encoding error, among the pseudo-real-value nonuniversal FFTUnitary mesh (black) and universal GridUnitary mesh (red) as well as conventional SVD-GridNet mesh (green) as the functions of the radix N.

6 Conclusions

In conclusion, we propose the pseudo-real mesh to implement scalable and compact PNN, which utilizes the real part of the complex-value matrix defined by a nonuniversal unitary MZI mesh to express the real-value matrix. Qualitatively, this method allows nonuniversal unitary mesh consuming O(Nlog₂ N) level MZIs to achieve high accurate matrix expressivity close to conventional universal SVD mesh require O(N ²) MZIs at small radix N, while at large radix limit, it is also promising for the pseudo-real mesh with O(Nlog₂ N) MZIs to approach the weight kernels for PNN applications due to their low-dimension embedding tendency. We train the LeNet-5 model for MNIST dataset and MobileNet model for Fashion-MNIST dataset, verifying the compact pseudo-real-value FFTUnitary mesh design could apparent reduce the required photonic devices but pay acceptable low cost on learning capability loss. Experimentally, we implement an inference system based on 4*4 pseudo-real-value photonic neural chip, which run the LeNet-5 network for MNIST dataset with high accuracy to demonstrate the feasibility of our photonic computation architecture. On the view point of technical implementability, the pseudo-real mesh design shows all-round advantages over conventional SVD-based mesh design on chip size, insertion loss, power consumption and encoding error. Further improvement may be achieved by shrinking the size of MZI via optimized design as well as introducing the silicon-photonics-process compatible but more efficient phase shifter. Our results, presented in this paper, may inspire some further interdisciplinary explorations teaming up with the researchers in computer science and mathematics community on the exact potential of pseudo-real-value nonuniversal unitary mesh for scalable photonic processing.