A Research on Aero-engine Control Based on Deep Q Learning
-
Qiangang Zheng
,
Zhihua Xi
Abstract
For improving the response performance of engine, a novel aero-engine control method based on Deep Q Learning (DQL) is proposed. The engine controller based on DQL has been designed. The model free algorithm – Q learning, which can be performed online, is adopted to calculate the action value function. To improve the learning capacity of DQL, the deep learning algorithm – On Line Sliding Window Deep Neural Network (OL-SW-DNN), is adopted to estimate the action value function. For reducing the sensitivity to the noise of training data, OL-SW-DNN selects nearest point data of certain length as training data. Finally, the engine acceleration simulations of DQR and the Proportion Integration Differentiation (PID) which is the most commonly used as engine controller algorithm in industry are both conducted to verify the validity of the proposed method. The results show that the acceleration time of the proposed method decreased by 1.475 second while satisfied all of engine limits compared with the tradition controller.
Funding statement: This study was supported in part by National Natural Science Foundation of China under Grant 51906102, in part by National Science and Technology Major Project under Grant 2017-V-0004-0054, in part by Research on the Basic Problem of Intelligent Aero-engine under Grant 2017-JCJQ-ZD-047-21, in part by China Postdoctoral Science Foundation Funded Project under Grant 2019M661835, in part by Aeronautics Power Foundation under Grant 6141B09050385, in part by the Fundamental Research Funds for the Central Universities under Grant NT2019004.
Nomenclature
| Symbol | Explanation | Symbol | Explanation |
|---|---|---|---|
| H | Height | T41 | High pressure turbine inlet temperature |
| Ma | Mach number | Nf | Fan rotor speed |
| PLA | Power level angle | Nc | Compressor rotor speed |
| Wfb | Fuel flow | Smf | Fan surge margin |
| F | Engine thrust | Smc | Compressor surge marge |
Copyright Reminder
If any material that is included in a paper is under copyright, the authors of the paper are responsible for obtaining copyright permissions, including any information required by the copyright holder, and be able to produce such permissions upon request.
References
1. Skira CA, Agnello M. Control system for the next century’s fighter engines. Trans ASME J Eng Gas Turbines Power. 1992;114:749–54.10.1115/91-GT-278Search in Google Scholar
2. Zheng Q, Miao L, Zhang H, Ye Z. On-board real-time optimization control for turbofan engine thrust under flight emergency condition. Proc Inst Mech Eng Part I: J Syst Control Eng. 2017;231:554–66.10.1177/0959651817710127Search in Google Scholar
3. Zheng Q, Zhang H, Miao L, Sun F. On-board real-time optimization control for turbo-fan engine life extending. Int J Turbo Jet-Engines. 2017;34:321–32.10.1515/tjj-2015-0066Search in Google Scholar
4. Iii HAS, Brown H. Control of jet engines. Control Eng Pract. 1999;7:1043–59.10.1016/S0967-0661(99)00078-7Search in Google Scholar
5. Jaw LC, Mattingly JD. Aircraft engine controls, design, system analysis, and health monitoring. Virginia: American Institute of Aeronautics and Astronautics, Inc.; 2009.10.2514/4.867057Search in Google Scholar
6. Tang W, Wang L, Gu J, Gu Y. Single neural adaptive PID control for small UAV micro-turbojet engine. Sensors. 2020;20:345.10.3390/s20020345Search in Google Scholar PubMed PubMed Central
7. Achiam J, Knight E, Abbeel P Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894, 2019.Search in Google Scholar
8. Botvinick M, Ritter S, Wang JX, Kurth-Nelson Z, Blundell C, Hassabis D. Reinforcement learning, fast and slow. Trends Cogn Sci. 2019;5:408–22.10.1016/j.tics.2019.02.006Search in Google Scholar PubMed
9. Schuitema E, Hobbelen DG, Jonker PP, Wisse M. Using a controller based on reinforcement learning for a passive dynamic walking robot. Humanoid Robots, 2005 5th IEEE-RAS International Conference on. IEEE, 2005:232–7.Search in Google Scholar
10. Wang S, Braaksma J, Babuska R, Hobbelen D. Reinforcement learning control for biped robot walking on uneven surfaces. Neural Networks, 2006. IJCNN’06. International Joint Conference on. IEEE, 2006:4173–8.Search in Google Scholar
11. Ziqiang P, Gang P, Ling Y. Learning biped locomotion based on Q-learning and neural networks. Adv Autom Rob. 2011;1, Springer, Berlin, Heidelberg:313–21.10.1007/978-3-642-25553-3_39Search in Google Scholar
12. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.Search in Google Scholar
13. Gu S, Lillicrap T, Sutskever I, Levine S. Continuous deep q-learning with model-based acceleration. International Conference on Machine Learning. 2016:2829–38.Search in Google Scholar
14. Zhang M, McCarthy Z, Finn C, Levine S, Abbeel P. Learning deep neural network policies with continuous memory states. Robotics and Automation (ICRA), 2016 IEEE International Conference on. IEEE, 2016:520–7.10.1109/ICRA.2016.7487174Search in Google Scholar
15. Lenz I, Knepper R, Saxena A DeepMPC: learning deep latent features for model predictive control. Robotics: Science and Systems, 2015.10.15607/RSS.2015.XI.012Search in Google Scholar
16. Oh J, Chockalingam V, Singh S, Lee H. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605.09128, 2016.Search in Google Scholar
17. Jaderberg M, Czarnecki WM, Dunning I, Marris L. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science. 2019;364:859–65.10.1126/science.aau6249Search in Google Scholar PubMed
18. Foerster JN, Assael YM, de Freitas N, Whiteson M. Learning to communicate to solve riddles with deep distributed recurrent q-network. arXiv preprint arXiv:1602.02672, 2016.Search in Google Scholar
19. Silver D, Huang A, Maddison CJ, Guez A. Mastering the game of Go with deep neural networks and tree search. Nature. 2016;529:484–9.10.1038/nature16961Search in Google Scholar PubMed
20. Luong NC, Hoang DT, Gong S, Niyato D, Wang P, Liang Y, Kim D. Applications of deep reinforcement learning in communications and networking: A survey. IEEE Commun Surv Tutorials. 2019;21:3133–74.10.1109/COMST.2019.2916583Search in Google Scholar
21. Zheng QG, Zhang HB, Li Y, Hu Z. Aero-engine on-board dynamic adaptive MGD neural network model within a large flight envelope. IEEE Access, 2018;6:45755–61.10.1109/ACCESS.2018.2789935Search in Google Scholar
22. Zheng Q, Du Z, Fu D, Hu Z, Zhang H. Direct thrust inverse control of aero-engine based on deep neural network. Int J Turbo Jet-Engines. 2019. https://doi.org/10.1515/tjj-2018-0049.10.1515/tjj-2018-0049Search in Google Scholar
23. Zheng QG, Zhang HB, Ye ZF, Miao L. Acceleration process optimization control of turbofan engine based on variable guide vane adjustment. J Aerosp Power. 2016;31:2801–8.Search in Google Scholar
© 2020 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Original Research Articles
- Novel Closed-Form Equation for Critical Pressure and Optimum Pressure Ratio for Turbojet Engines
- Development of Combustor for a Hybrid Turbofan Engine
- A Parametric Study of Hydrogen Fuel Effects on Exergetic, Exergoeconomic and Exergoenvironmental Cost Performances of an Aircraft Turbojet Engine
- A Model Reference Adaptive Sliding Mode Control Method for Aero-Engine
- Numerical Study of Mixing Enhancement in 2D Supersonic Ejector
- Flow Development Through An S-Shaped Diffusing Duct
- Nozzle Geometry Effect on Twin Jet Flow Characteristics
- Application of Proper Orthogonal Decomposition Method in Unsteady Flow Field Analysis of Axial High Bypass Fan
- A Research on Aero-engine Control Based on Deep Q Learning
- Blade Number Selection for a Splittered Mixed-Flow Compressor Impeller Using Improved Loss Model
- Study of Correctly Expanded Sonic Jet with Air Tabs
- Large-Eddy Simulation of Shaped Hole Film Cooling with the Influence of Cross Flow
- Modal Analysis of the Axial Compressor Blade: Advanced Time-Dependent Electronic Interferometry and Finite Element Method
- Exergy Analysis of a Turboprop Engine at Different Flight Altitude and Speeds Using Novel Consideration
- Thermal Optimization Design of Heat Exchanger in Supersonic Engine with Parameters’ Fluctuation
Articles in the same Issue
- Frontmatter
- Original Research Articles
- Novel Closed-Form Equation for Critical Pressure and Optimum Pressure Ratio for Turbojet Engines
- Development of Combustor for a Hybrid Turbofan Engine
- A Parametric Study of Hydrogen Fuel Effects on Exergetic, Exergoeconomic and Exergoenvironmental Cost Performances of an Aircraft Turbojet Engine
- A Model Reference Adaptive Sliding Mode Control Method for Aero-Engine
- Numerical Study of Mixing Enhancement in 2D Supersonic Ejector
- Flow Development Through An S-Shaped Diffusing Duct
- Nozzle Geometry Effect on Twin Jet Flow Characteristics
- Application of Proper Orthogonal Decomposition Method in Unsteady Flow Field Analysis of Axial High Bypass Fan
- A Research on Aero-engine Control Based on Deep Q Learning
- Blade Number Selection for a Splittered Mixed-Flow Compressor Impeller Using Improved Loss Model
- Study of Correctly Expanded Sonic Jet with Air Tabs
- Large-Eddy Simulation of Shaped Hole Film Cooling with the Influence of Cross Flow
- Modal Analysis of the Axial Compressor Blade: Advanced Time-Dependent Electronic Interferometry and Finite Element Method
- Exergy Analysis of a Turboprop Engine at Different Flight Altitude and Speeds Using Novel Consideration
- Thermal Optimization Design of Heat Exchanger in Supersonic Engine with Parameters’ Fluctuation
