Deep reinforcement learning empowers automated inverse design and optimization of photonic crystals for nanoscale laser cavities

1 Introduction

Inverse design of optical resonators [1, 2] is a crucial step in designing state-of-the-art nanoscale laser cavities [3] that realize classic photonic crystal lasers [4–6] finding broad applications in photonic integrated circuits, optical interconnects, and telecommunications. The inverse problem herein learns the hidden relationship between optical response and physical structure and typically involves retrieving a design topology for desired optical responses. Due to its seemingly “counter-intuitive” nature and a non-convex [7, 8] solution space, compared to the forward prediction problem, inverse design has been a major challenge [9] in the photonics community and been intensively studied in a multitude of disciplines [10–15].

Since the 90s, many approaches to photonics inverse design have been proposed, mainly including: first-principle methods [16, 17], finite element analysis (FEA) or finite difference time domain (FDTD)-based simulation solvers [18, 19], and evolutionary [20, 21] or gradient-based [22, 23] optimization algorithms. While these approaches have historically yielded satisfactory results, they tend to demand heavy human involvement due to an iterative or trial-and-error nature and thus have defied automation. With the advancement of deep learning since circa 2012 [24, 25], researchers found new hope in the intelligent inverse design and optimization of nanophotonics [2, 9]. However, existing supervised and semi-supervised learning methods (Figure 1(c)) ultimately suffer from three unresolved problems: (1) the need for a large pre-collected dataset for training and/or the need for a pre-trained Multi-layer Perceptrons (MLP) network in tandem network architectures; (2) the undesired one-to-many mapping that leads to non-unique design solutions for a given optical response; and (3) the inability to perform optimization of the photonic structure beyond inverse designing; in other words, the inverse designed structure simply mimics or replicates the initial design it started with and the optical features of the structure do not receive any improvement or optimization. Among the above, point 1 is the toughest to overcome primarily because deep learning is known to be highly data-hungry. Also, physics-based simulators solving Maxwell’s equations are generally time-consuming, thus generating large-scale datasets can become inefficient or even infeasible. On top of that, most nanophotonic devices are proprietary within individual research groups, making it difficult to release/access open-source datasets for the community. Points 2 and 3 pose a major challenge to the photonics community as well. Although a plethora of existing works (see Figure 1(c)) have attempted to use tandem MLP networks [26, 27], Generative Adversarial Networks (GANs) [28, 29], Variational Autoencoder (VAE) [30, 31], and iterative optimization with DNNs [32] to inverse design various photonic devices, these prior art have been limited by the above unresolved issues.

Figure 1:

PC nanobeam laser cavities used in this work for inverse design and optimization via deep reinforcement learning and an overview of existing deep learning-based inverse design models. (a) 3D visualization of a typical long PC nanobeam and its top view. (b) 3D visualization of a typical short L3 nanocavity nanobeam and its top view. Indium Phosphide (InP) is chosen as the semiconductor laser material (i.e., the gain material). Physical configurations and optical characteristics of both nanobeams are shown in Table 1. Both nanobeams possess perfect symmetry both laterally and longitudinally. x _i is the x-coordinate of the ith air hole counting from the inside in the taper region and x _m is the x-coordinate of the holes in the mirror region. (c) Existing (semi)-supervised models for photonics inverse design, which takes target optical responses as input and outputs retrieved design parameters. Left: a tandem neural network structure built with MLPs, where the boxed region represents the pre-trained forward network. Right: a GAN-based model that is able to distinguish predicted design parameters from ground truths. This method is unable to perform optimization of the photonic structure. G and D in the figure denote the generator and discriminator, respectively. This work will address the flaws inherent in these DL models shown here. A detailed comparison between the method proposed in this work and existing DL methods can be found in the Supporting Information.

Reinforcement Learning (RL) started out as a purely mathematical model over half a century ago [33] and in recent years has been successfully applied to solving increasingly more real-world tasks [34–45] with profound scientific and engineering significance. Google DeepMind’s AlphaGo Zero [46] and AlphaStar [47] are prominent hallmarks of RL’s ability to reach or even exceed the abilities of human beings. While conventional (semi)-supervised learning with deep neural networks (DNNs) has recently been applied in photonics forward [48–50] and particularly inverse designs [26], [27], [28], [29, 32, 51, 52], RL is still a relatively under-explored realm to date. Nonetheless, since RL requires NO pre-collected training data (and instead explores on the fly), is free of the non-uniqueness issue, and can provide optimizations of the photonic structure beyond inverse designing, it has a clear edge in the inverse problem. For example, recently, Sajedian et al. adopted deep RL to optimize dielectric nanostructures and high-transmission color filters [45, 53] in two separate works, marking the inception of RL in photonics design. More recently, Sui et al. inverse designed digital nano-materials using deep RL and demonstrated a robust convergence of the model [54]. In 2021, Mirhoseini et al. posed chip floorplanning as a deep RL problem to design the newest of Google’s tensor processing unit accelerators that surpassed strongest baseline models [38]. In 2022, Kuprikov et al. demonstrated a deep RL approach that was successfully used for the control of the generation of dissipative solitons in mode-locked fiber laser system [44]. However, there’s no existing literature that inverse design optical resonators or laser cavities using RL algorithms.

RL algorithms are best suited for solving complex high-dimensional (and possibly non-convex) optimization problems [55], especially when there’s a large design parameter space to explore. In our case, even one variable (e.g., picking the number of air holes in the nanobeam) has an enormous state – action space and degree of freedom. Thus, given the scale and difficulty of the photonics inverse problem, we developed a deep RL method that, through repeated episodes (sequences of states, actions and rewards), manages to explore a sizeable design parameter space looking for the optimal solution. We named the proposed method Learning to Design Optical-Resonators (L2DO). Because of the existence of a wide variety of RL algorithms, we opted to experiment with Deep Q-learning (DQN) [39] by Google DeepMind and Proximal Policy Optimization (PPO) [56] by OpenAI for L2DO’s implementation. To demonstrate the power of L2DO, two different photonic crystal (PC) nanobeam cavities [57, 58] are investigated in this work. In our experiments we show that, with a greater volume and variety of data samples accumulated during training, L2DO learns to be both faster and more stable at generating unique design parameters that satisfy given optical responses, thus solving the inverse design problem. In addition, L2DO can optimize good-to-have optical quantities that are also important for an ideal laser cavity. Finally, using several novel modifications to the model and training process, the sample efficiency (i.e., exploration rate) and convergence speed of L2DO were successfully improved by two and a half times, which helped alleviate the expensiveness of training the RL model. L2DO brings us closer to a future in which nanophotonic researchers are assisted by autonomous artificial agents with potentially vast optimization experience and superhuman efficiency. To the best of our knowledge, this work is the first to leverage deep RL for inverse designing and optimizing nanoscale laser cavities with a diversified array of tunable parameters.

2 Methods and results

Objective overview. To demonstrate L2DO’s effectiveness and efficiency in inverse designing different laser cavities, we chose to investigate a long PC nanobeam and a short L3 nanocavity nanobeam as schematically visualized in Figure 1(a) and (b). For simplicity, the former will be denoted as “long nanobeam” and the latter “L3 nanobeam” going forward in this text. Each nanobeam is divided into a Taper region and a Mirror region, respectively, according to their unique air hole size distributions. The x’s refer to the x-coordinate of each air hole, where x ₁, x ₂, … are the x-coordinate of the ith air hole counting from the inside in the Taper region and x _m is the x-coordinate of the holes in the Mirror region. Both nanobeams have Indium Phosphide (InP) as the gain (i.e., active) material. Nanobeam cavities are commonly used as laser sources in photonic integrated circuits (PICs), interconnects, and telecommunications when coupled to waveguides and optic fibers.

Both nanobeams’ initial specifications are summarized in Table 1, including their respective design parameters and optical responses. Design parameters are set with discretion according to the wafer specifications, fabrication capabilities, and special lasing requirements. In addition, both nanobeams are designed to be perfectly symmetric (i.e., left and right sides are mirror image of each other). Among others, a key difference is that air holes in the L3 nanobeam are elliptical, whereas those in the long nanobeam are circular. This design choice was made on a case-by-case basis with a special focus on maximizing the Q factor of each nanobeam. Finally, Q factors, resonance (operational) wavelengths λ, and modal volumes V are calculated by 3D FDTD simulations. Due to its more complex structure (i.e., hole shape, thickness etc.), the L3 nanobeam’s FDTD simulations take more time to finish compared to the long nanobeam, as shown in Table 1. The simulation time will play a major role in our subsequent design of the deep RL algorithm.

The inverse design problem defined. With the initial design parameters and optical responses introduced, the inverse design task to be fulfilled by the proposed L2DO is formally defined as follows:

1. Increase the Q factor of the long nanobeam and L3 nanobeam to a target maximum of 5e+7 and 3e+6, respectively, from their initial values and retrieve the corresponding unique design parameters (REQUIRED, the higher the better);

2. Maintain the modal volume V of both nanobeams at or below their initial levels (good-to-have, the smaller the better);

3. Constrain the resonance wavelength λ of both nanobeams to their initial values ± 50 nm of tolerance (good-to-have, the closer the better),

Deep RL algorithms and their elements. To address the challenging photonics inverse design problem, we developed a deep RL method capable of seamlessly generalizing across structures with increasingly more experience and higher efficiency accumulated over time. In other words, L2DO learns from its own experiences to inverse design a variety of optical resonators and laser cavities. Specifically, the RL problem can be formulated as a sequential Markov decision process (MDP) [33], because this allows us to more easily incorporate the inverse problem’s core attributes. We list five key MDP elements used in our L2DO:

1. States encode information about the structural (design) parameters, including the material’s refractive index, the spatial arrangement of components, the number, size and shape of each component, and the length, width, and thickness of the whole device, among others.

2. Actions are all possible changes in the value of the above state encodings, without violating any hard constraints on size or locations. Optimal actions are predicted by the RL agent’s policy neural network and are expected to traverse a large portion of the range of states.

3. Policy represents the key part of an RL agent that receives a state and makes decisions for what actions to take. A policy is generally parametrized by DNNs and can be considered as the “brain” of an RL model.

4. Environment is the FEA/FDTD-based simulation software solving Maxwell’s equations and it takes in an action and outputs the next state and the associated reward. Simulating photonic components routinely takes several minutes or even hours. To alleviate the computational burden induced by simulations, a DNN could be introduced to approximate the environment; more details about this approximation will be given later in this section.

5. Rewards are proportional to how close we get to the specified design targets, and good actions earn higher rewards. Design targets are defined by preset maxima (e.g., optical responses like the Q-factor, modal volume etc.) to reach, as well as other good-to-have features (e.g., resonance wavelength, bandgap width etc.) to optimize.

Full definitions of these elements are included in the Suppporting Information.

Implementation of L2DO by deep RL. Figure 2(a) schematically illustrates the full deep RL implementation of L2DO, containing the input and output, the policy network, the environment, actions, states, and rewards, etc. L2DO takes desired optical responses as input and yields the retrieved design parameters as output. Both DQN and PPO algorithms are completely integrated and realized with this model design. Here, state is also sometimes referred to as observation in the literature. The policy network, typically realized by DNNs, is taken to be an MLP neural network in our case. The policy represents the autonomous RL agent that decides what action to take during inverse designing and can be considered as the “brain” of an RL model. The goal of the policy is to gradually maximize the cumulative reward. One complete loop inside the box with dotted lines in Figure 2(a) constitutes one training step of L2DO, and multiple steps constitute a typical training episode in RL. In our case, we set the maximum number of steps within an episode (also known as the horizon) to be 250. Full model specs and hyperparameters of L2DO can be found in the extended data in the Supporting Information. Figure 2(b) pictorially illustrates the commonly practiced human-centered inverse design mythology, where a human expert (usually a scientist or researcher) devises a specific photonic design, enters it into a simulation solver, computes the output optical responses, evaluates the results by comparing them to desired targets, and revises the design if targets unsatisfied. This iterative approach, heavily relying on the knowledge and experience of the human expert, is highly labor-intensive, slow, and illy generalizable. We aspire to supersede this human-centered approach by our deep RL-empowered L2DO.

Figure 2:

Comparison between deep RL-enabled (this work) and conventional human-centered inverse design methodologies. (a) Full details of the deep RL-based implementation of L2DO for photonics inverse design. L2DO takes in the desired optical specifications of certain photonic devices and delivers the solved design parameters. Both DQN and PPO algorithms are seamlessly incorporated within L2DO. The policy (RL agent) is implemented with a 4-layer MLP network whereas the environment is realized by an FDTD-based simulation solver (or optionally a DNN). Data sequences are stored in a replay buffer for data sample reuse and breaking sample correlations. (b) Human-centered iterative inverse design approach, which is still the most common practice in existence today. This conventional approach, relying on the intuition and experience of the human expert, typically takes over a month or even months to yield a satisfactory photonic design.

Next, the expensiveness associated with training RL models is addressed. Although RL does not require any pre-collected training data, it still has to sample data and do explorations on the fly. This could cause issues like a low sample efficiency and difficult training convergence especially at the presence of a time-consuming (i.e., expensive) environment. In this work, we devised and propose a novel DNN/Simulator Alternating technique (DSA) (see Figure 3) that works in harmony with L2DO, where a surrogate DNN is introduced to approximate the FDTD simulator to reduce the computational cost of simulations and boost L2DO’s sample efficiency. This surrogate DNN essentially predicts rewards from given states. More details in the Supporting Information.

Figure 3:

The novel DNN/Simulator Alternating technique (DSA), where the surrogate DNN and the actual FDTD simulator take turns to function as the environment of L2DO. Input to the DNN is next states and output is rewards. The DNN is only activated after the first E episodes, which is set to be 10 for illustrative purposes in this diagram. Right before the activation, the DNN is trained on data samples collected from FDTD during the first E episodes. After the activation, the DNN and the simulator alternate between odd and even episodes, respectively. Samples during the episodes when the simulator is used are still stored to the replay buffer for continuously training the DNN down the road. DSA is expected to cut down training time by a large portion.

Learning curves of L2DO and the retrieved designs. Training results of L2DO are included in this part where studies on both the long and L3 nanobeams are demonstrated. The RL code was fully written in Python strictly following the algorithmic model illustrated in Figures 2 and 3, and popular machine learning libraries like Pytorch, gym, and Ray RLlib were extensively utilized in our program. Figure 4 showcases the learning curve plots (rewards vs. episodes) where the vertical axes are plotted in log-scale. Figure 4(a) shows the training convergence of using L2DO to inverse design the L3 nanobeam. Three separate trials were done and trial 3 (green) represents the optimal solution for the inverse design problem. Trial 1 (blue) and trial 2 (orange)’s training were terminated earlier because their respective rewards showed no improvement for over 50 consecutive steps. Similarly, Figure 4(b) shows the three trials for inverse designing the long nanobeam by L2DO-DSA and the optimal trial is also colored in green. The rewards in Figure 4 are calculated by Equation 6. Hyperparameters of the optimal runs are included in the extended data in the Supporting Information.

Figure 4:

Learning curves of L2DO, plotted as rewards versus episodes. Vertical axes are in log-scale. (a) Convergence trends of the L3 nanobeam trained with L2DO. Trial 1 and 2 were terminated earlier because the reward stopped growing for over 50 consecutive steps. Trial 3 represents the optimal learning curve trained for 58 episodes. (b) Convergence trends of the long nanobeam trained with L2DO-DSA, where trial 3 represents the optimal learning curve trained for 131 episodes. Early stopping is utilized as well. Choice of DSA for either the L3 or the long nanobeam is explained in the Supporting Information.

Based on the optimal curves in Figure 4, a unique set of design parameters retrieved for both nanobeams that successfully met the target maxima are summarized in Table 2. The set of solved design parameters corresponds to the state space defined in Table 2 in the Supporting Information. The physical structures of the inverse designed nanobeams are schematically illustrated in Figure 5(a), where black circles and the red dashed circles correspond to the solved and initial holes, respectively. We then verified that desired optical responses were indeed satisfied by L2DO by feeding the solved design parameters into the FDTD simulator and examining the output optical responses, which are also listed in Table 2 (check-marked). It is clearly observed that the FDTD-verified values (i.e., Q, V, and λ) are highly aligned across the board with the target values (i.e., ground truths) marked with stars and small differences are present but well under the margin of error. More importantly, the verified Q-factors (3.12 × 10⁶ and 5.04 × 10⁷, respectively) not only solved the inverse design problem by substantially improving from the initial values (8 × 10⁴ and 3 × 10⁶, respectively, as shown in Table 1) but also even exceeded the target maxima (3 × 10⁶ and 5 × 10⁷) by an appreciable amount (the higher, the better). Moreover, the verified V’s are calculated to be smaller than the target values (the smaller, the better) while the verified λ’s are shifted away from the target values by differences well under the tolerance of ±50 nm (the closer the better). These results mean that L2DO not only satisfied the REQUIRED and good-to-have tasks simultaneously, but also provided us extra bonuses we did not originally ask for. The target optical responses (starred) are extracted from Table 1 and the definition of the inverse problem presented earlier, whereas the verified values using the solved design parameters are accompanied by check marks in Table 2.

To better illustrate the advantage of L2DO, best Q-factors reported by a human expert [59] manually tuning the exact same nanobeams are tabulated in Tables 2 and 3 for comparison as a baseline. It is observed that the FDTD-verified Q’s definitively beat the human results by a large margin for both nanobeams (≈10 times and 5 times, respectively). λ and V reported by the human, on the other hand, are close to the L2DO-solved values in magnitude and thus are not listed (the desired λ and V are easier to achieve than Q because they do not require any major increase in magnitude (i.e., optimization)). Furthermore, human experts often need to spend over a month on the device optimization process as shown in Figure 2(b), whereas it only took us a few days to tune and train L2DO using hardware resources no more than a few workstations. Full specifications of the computing resources used are included in the Supporting Information. Finally, all solved design parameters fall within the bounds set by the state spaces in Table 2 in the Supporting Information, indicating a good tractability of the algorithm. Therefore we conclude that, without any specific expert knowledge or suffering from the one-to-many mapping issue, L2DO has indeed fulfilled the inverse design task defined before with above-expectation and superhuman performance.

We have also included best results from the literature [60–62] as baselines and have them tabulated along with our results for comparison in Table 3. Since the semiconductor materials/nanobeam dimensions used in existing literature are generally different from ours, their results might not be suitable for a direct comparison of metrics to ours; nonetheless, they can provide a perspective on how well our L2DO performed. For instance, Quan et al. [62] manually designed a nanobeam that reported a high Q of 1e+9, where the material is Si and the nanobeam has 40 holes on each side. McCutcheon et al. [61] manually designed a nanobeam that reported a high Q of 1e+6, where the material is SiN _x and the nanobeam has 24 holes on each side. Kim et al. [60] manually designed a nanobeam that reported a high Q of 1e+5, where the material is InGaAsP quantum wells and the nanobeam has around 20 holes on each side. In this work, we leveraged deep RL to design a long nanobeam reporting a high Q > 5e+7, where the material is InP and the nanobeam has 25 holes on each side. As we can see here, each work (including ours) used a different material (i.e., refractive index) and number of holes in their nanobeam, among other design choices, which resulted in varying Q factors. Compared to Kim et al. [60] which adopted a similar material and number of holes, our optimized Q is two orders of magnitude higher. It is important to note as well that existing literature reporting nanobeam designs are all coming from human’s manually tuned results while we are using deep RL to fulfill this task.

Analysis of training time and sample efficiency. With regards to the training time, the optimal trial of L2DO w/DQN was trained for 58 episodes in 88 h (shown in Figure 4(a) and Table 2) to inverse design the L3 nanobeam. On the other hand, the optimal trial of L2DO-DSA w/PPO was trained for 131 episodes in 152 h (also shown in Figure 4(b) and Table 2) to inverse design the long nanobeam. Training time was kept in control thanks to the early stopping technique we used, as pointed out in Figure 4. The total time spent on tuning the model was of course slightly longer because of the multiple trials we did and our limited parallel computing and CPU/GPU capacity. Critically, L2DO-DSA w/PPO utilized the DSA technique to increase sample efficiency and speed up training convergence. Without DSA, the training time is estimated to be over 300 h and the sample efficiency 2.5 times lower for the exact same hyperparameter settings in the same model. Furthermore, compared to the human expert and supervised deep learning, L2DO-DSA is estimated to have achieved over 200 times higher sample efficiency based on our calculations. A higher sample efficiency will play a bigger role when more complex photonic structures [4, 5] are inverse designed or larger state-action spaces are adopted in the future.

Generated optical responses and their meanings. Next, when verifying the solved designs in FDTD, we also calculated the resulting Electric field profiles and emission spectra, which are visualized in Figure 5(b) and (c). The corresponding FDTD-verified Q, V, and λ values are also labelled on the plots. According to the spectra in Figure 5(b), a single resonance peak can be located at the target wavelengths (950 and 1550 nm, respectively, for each nanobeam), indicating the existence of a single mode and the correctness of the electromagnetic (EM) modes given by the nanobeams designed by L2DO. y-axes of the spectra typically represent the intensity of the EM field measured experimentally or simulated by computer while x-axes correspond to wavelengths. For low-Q cavities, Q is inversely proportional to full width at half maximum (FWHM) of the spectrum. For high-Q cavities such as the L3 and long nanobeams, we cannot directly extract Q from the emission spectrum because the FWHM of each resonance in the spectrum is limited by the simulation time, T _sim, by FWHM ∼ 1/T _sim. Please see the Supporting Information for how Q is related to the spectrum in this case. According to the E _y profiles in Figure 5(c), we see that a fundamental or second order resonance mode is achieved by both nanobeam structures, which means that the design parameters solved by L2DO indeed gave rise to correct resonance mode profiles with respect to the spectra in Figure 5(b). These results guarantee a high-quality and precise laser for applications in interconnects on PICs and telecommunications. In future endeavors, experimentalists in our group will fabricate the solved designs demonstrated here into actual laser devices capable of performing real-world light-emitting tasks.

Figure 5:

Physical structures, spectra, and electric field profiles (E _y ) of the inverse designed nanobeams from Table 2. Left panel: L3 nanobeam’s solved structure, spectrum and electric field profile, respectively; right panel: long nanobeam’s solved structure, spectrum and electric field profile, respectively. In (a), red dashed circles correspond to the original holes, whereas black solid circles represent the resulting holes from L2DO’s inverse design solution. Results in (b) and (c) are generated by 3D FDTD simulations. y-axes of the spectra in (b) represent the intensity of the EM field whereas x-axes represent wavelengths. A single highest peak in a spectrum corresponds to a single resonance mode. E _y field plots in (c) display the electromagnetic resonance modes of the retrieved nanobeam structures, where the verified Q, V, and λ are also labelled. Resonance modes are necessary for lasing operations.

Discussion on fabrication uncertainties/tolerances. To account for potential fabrication uncertainties (such as during the ICP-RIE and EBL processes), the L2DO-retrieved design parameters were randomly perturbed within ± 10% of its neighborhood. Corresponding FDTD-calculated optical responses of the perturbed structures showed that Q-factors of about half of the structures stayed above the requested thresholds, thus demonstrating the robustness of the retrieved designs against fabrication uncertainties. For instance, when the solved x ₁ ∼ x ₆ and r of the long nanobeam were randomly fluctuated by ±10% on 20 independent trials, the calculated Q-factors of 9 cases remained higher than 5e+7; among the remaining 11 cases that dropped below 5e+7, the lowest and highest Q-factors were 3.24e+7 and 4.57e+7, respectively. The percentage error between 4.57e+7 and 5e+7 is calculated to be 8.6%, which is less than 10%. Therefore, it is observed that although perturbing design parameters could lead to degraded Q-factors in some of the cases, the highest Q-factor is still within the margin of error we expected. This insensitivity to fabrication errors will be crucial for future experimental verification of our inverse designed nanobeam cavities.

Discussion on the generalizability of L2DO. To optimize new designs without further training or with minimal training means to expand the generalizability of L2DO. In its current form, L2DO already possesses decent generalizability because if we were to slightly perturb the air hole radii such that it stays within a relatively close neighborhood of the current optimal solution, we could obtain optimized results for the new design by only training for a few iterations. This can be justified by the fact that RL can effectively learn from past cumulative experiences of high diversity in a large domain. As a result, designs with small changes in design parameters have most likely been encountered by the RL agent at some point during training and thus can be quickly identified by L2DO if needed. Nonetheless, in the current stage, we still need to re-train the whole algorithm for each new design. In order to obtain one model which is trained on setting A but can easily be adapted to target setting B or C, the model needs to have a good transfer ability. To the best of our knowledge, there are two paradigms in the field of machine learning that allow one model to achieve this goal, namely (1) the pre-train and fine-tune paradigm and (2) the meta-learning paradigm. In the pre-train and fine-tune paradigm, a model is pre-trained in a high-diversity dataset which contains samples pre-collected from different prior tasks, and then adapted to the target task by fine-tuning a part of the model or some additional parameters using a task-specific objective function. In our inverse-designed cavity problem, we have collected the dataset from previous RL trainings for each cavity setting, and have trained a general agent with those data. When faced with a new task, we can use this agent to perform some online interactions and update the policy to achieve adaptation. As for meta-learning [63], this new paradigm presents a greater potential to learn common information from various tasks and transfer to some previously unseen tasks similar to the pre-train method. However, due to the larger computational requirement, meta-trained models are usually only used for optimizing the same type of device with different design parameters (such as a long vs. a short nanobeam). In addition, we believe it is more advantageous to use the pre-trained method for optimizing across devices that are more differentiated or less similar in kind.

Comparison with particle swarm optimization (PSO). PSO is a meta-heuristic optimization algorithm that makes little assumptions about the problem being solved and can quickly search very large spaces of candidate solutions while offering no guarantees that an optimum be found. Basically, PSO trades optimality and accuracy for speed. PSO has been demonstrated in several prior works [64] concerning photonics optimization. We ran our nanobeam nanocavities with PSO and included the optimization results in the Supporting Information. As seen in Figure 1 in the Supporting Information, PSO did not match the level L2DO had reached. Nonetheless, we believe if given the right amount of computing power and time frame and under ideal circumstances, PSO may come close to L2DO’s results. Last but not least, since deep RL has been demonstrating groundbreaking performance [38, 39, 45, 47] in a variety of fields in recent years, we do believe L2DO will play a competitive role in photonics inverse design in the long run.

Comparative studies on tuning key hyperparameters. Finally, comparative studies were done on different hyperparameters in order to search for the optimal settings for L2DO(-DSA). Three of the studies are discussed here in detail. First, a comparative study on the impact different E’s have on the training of L2DO is presented. See Supporting Information for the definition of E. We chose E = 5, 10, 15 and ran three experiments with L2DO-DSA (using PPO) to inverse design the long nanobeam. All the other hyperparameters were kept constant throughout these experiments. Learning curves from these three runs are plotted in Figure 6(a) for a comparison of their respective convergences and rewards. Early stopping is utilized to curtail the training time. As seen in Figure 6(a), a maximum reward is achieved in the shortest time when E = 10, which indicates we should set E as 10 for DSA. E = 5 and 15 yielded poorer results by comparison. When E is 5, the surrogate DNN (refer to Figure 3) is illy trained and perhaps underfit due to insufficient training data accumulated in the first 5 episodes. When E is 15, the surrogate DNN was trained on sufficient data but it took too long till the DNN came into play, which inevitably hurt the training convergence and sample efficiency of the whole learning process. Therefore, E = 10 is proven to be the optimal choice for the DSA framework as it found a delicate balance between the amount of training data accumulated and the precise time when the DNN gets activated. This conclusion aligns well with the common dilemma between overfitting and underfitting in machine learning. Second, a comparative study on different learning rates lr set in L2DO-DSA with PPO is conducted, where three lr = 5e−5, 1e−4, and 1e−3 are chosen. The lr is used in updating the policy network with gradient descent. Because lr affects every single step of training, the three runs with different lr’s are plotted in terms of rewards versus steps in Figure 6(b) to better examine the fine details of the learning curves. As seen in Figure 6(b), lr = 1e−3 achieved the fastest and smoothest convergence with highest rewards, making itself the optimal lr in this case. The other two lr’s, 5e−5 and 1e−4, both displayed very noisy and spiky learning curves and thus are believed to be poor choices. According to further experiments, lr’s larger than 1e−3 could cause catastrophic divergence to the training process. The above phenomena could be explained by the fact that smaller lr’s can slow down training convergence while larger lr’s can cause training to diverge or skip the optimum. Third, a comparative study on different numbers of neurons in the middle two layers of the MLP policy network set in L2DO with DQN is done. Here, three different combinations are chosen: (60, 80), (80, 120), and (120, 180), where the first number in each pair corresponds to Layer1 and the second number Layer2 as shown in the inset in Figure 6(c). Figure 6(c) showcases the learning curves for these three different combinations of neuron quantities and the trial with (80, 120) beat the rest with a clear margin. The (60, 80) trial did poorly likely because it is a somewhat under-parameterized model and thus failed to learn an effective mapping. On the other hand, although the (120, 180) trial had enough parameters to fit the model, it also has a lot more parameters to train and thus made the whole convergence process slower. The (80, 120) trial, however, appears to have just the right amount of parameters the problem needs and was able to rapidly converge to the highest rewards in a short time. We subsequently chose (80, 120) as the network dimension for our policy. These experiments above demonstrate the importance of constantly tuning and finding the optimal hyperparameters when one attempts to train a deep RL model; without properly chosen hyperparameters, the model has a good chance of diverging to undesirable directions very quickly. The lower right corner of Figure 6 lists some additional hyperparameters tuned by us that did not improve the model performance by an appreciable degree; nonetheless, these are still important hyperparameters that one should carefully tune in an RL model.

Figure 6:

Comparative studies on tuning key hyperparameters in the process of finding the optimal settings for L2DO(-DSA). (a) Learning curves for different E values (i.e., 5, 10, and 15) set in the L2DO-DSA framework with PPO. The curve that reached the maximum converged reward in the shortest time possible has the optimal E we are looking for. (b) Learning curves for different learning rates lr (i.e., 5e−5, 1e−4, and 1e−3) set in L2DO-DSA with PPO for updating the policy network. Rewards versus steps is plotted in this case to more closely examine the effects of different lr’s. (c) Learning curves for different numbers of neurons in the middle 2 layers of the MLP policy network set in L2DO with DQN. Three combinations are chosen: (60, 80), (80, 120), and (120, 180). Inset: middle 2 layers of the policy network. Lower right corner lists some additional hyperparameters tuned by us that did not lead to a significant improvement in training.

3 Discussion

In summary, this work features the first deep RL model, L2DO, applied to inverse designing and optimizing optical resonators playing a crucial role in nanoscale laser cavities. The inverse problem concerns retrieving a design topology that satisfies certain optical specifications of the photonic device. L2DO successfully addressed three major challenges faced by state-of-the-art deep learning-enabled inverse design methods: (1) a severe shortage of available training data and/or the need for a pre-trained neural network model; (2) the issue of one-to-many mapping or non-unique solutions; (3) the inability to perform optimization of the photonic structure beyond inverse designing. Specifically, L2DO incorporates two advanced RL algorithms, DQN and PPO, to inverse design photonic crystal nanobeam cavities. According to our studies, L2DO not only met the required maxima of certain optical responses (e.g., the Q factor > 50 million) but also optimized some good-to-have features (e.g., modal volume and wavelength) that are conducive to a high-quality laser. The solved values were then verified manually in FDTD and their correctness was confirmed by checking the generated optical responses. We can conclude that L2DO successfully improved state-of-the-art results in the literature by over 2 orders of magnitude and dominated a human expert by delivering Q factors that exceed the human’s results by over 10 times, while requiring only a fraction of the time and effort. In addition, to tackle issues like low sample efficiency and time-consuming simulations, a novel DNN/Simulator Alternating technique was introduced to L2DO and demonstrated very effective and consistent performance across different scenarios. L2DO-DSA is estimated to have reached over 200 times higher sample efficiency based on our calculations and reduced training time by a substantial amount. In the last few paragraphs, we discussed the connection between our optimized results and fabrication tolerances and compared L2DO to existing optimization algorithms such as the PSO algorithm. In the last part of this paper, we conducted several comparative studies on the effects different hyperparameters have on the training process and demonstrated how we arrived at the optimal network settings for L2DO. We then offered some advice on what hyperparameters need special attention when one attempts to tune a deep RL model. All in all, our inverse designed laser cavities can find broad applications in modern PICs, interconnects, and telecommunications.

Looking forward, L2DO will be expanded and strengthened in the following domains. (1) More advanced versions of deep Q-learning such as DDQN [65], Dueling DQN [66], ApeX DQN [67], and rainbow [68] etc. will succeed the current DQN; (2) The alternating mechanism in DSA (adopting an odd/even approach) will be succeeded by more sophisticated alternating mechanisms. Moreover, the following interesting thrusts will be explored as well. First, we look to incorporate in L2DO a light-weight neural network (NN) model [69] with high interpretability, which features a small model depth, limited parameters, and fast training. This NN can replace the current MLP policy network and the surrogate DNN in DSA. Next, we could introduce Gaussian processes (GP) [70] into the state space of L2DO-DSA that calculates an uncertainty; when the uncertainty goes over a threshold, DSA will switch to the FDTD simulation and when the uncertainty drops, DSA will switch back to the surrogate DNN. This method can provide an attractive alternative to the current odd/even switching mechanism. Third, a new paradigm of RL, coined offline RL [71], has recently emerged as a promising path towards effective real-world RL applications. Offline RL requires learning knowledge solely from previously collected datasets, without any active real-time environment interactions. Offline RL can improve the generalizability and speed of RL models by an appreciable degree. Last, our group will fabricate the inverse designed nanobeams and experimentally verify their validity.

Our methodology in this work is inspired by the famous marriage of AI and Electronic Design Automation (EDA) [38] over the last 5–8 years. For future prospects, this work paves the way for applying deep RL to the rapid multi-objective inverse design and optimization of nanophotonic devices without the need for pre-collecting any data or resorting to human-centered trial-and-error iterations. Through our efforts, we mainly aim to empower the rise of fully automated photonic design because the current state of Photonic Design Automation (PDA) is still largely lacking. Our time and efforts in the subsequent stage will be chiefly directed towards developing software that brings AI and PDA together into one unity.

Supporting information

The Supporting Information is available at the online version of this article and the repository: https://github.com/Arcadianlee/Photonics-FDTD-DRL.git. Correspondence and requests for materials should be addressed to Z.Z.