High-precision fault management in optical fiber communication systems based on artificial intelligence
-
Baydaa Hadi Saoudi
Abstract
The delivery of high-speed internet services heavily depends on stable and secure data transmission provided by optical fiber infrastructures. Therefore, promptly addressing fiber anomalies – whether deliberate actions like optical tapping or physical issues such as fiber cuts – is essential to ensure service continuity and network robustness. If left unresolved, these disruptions can compromise network stability, cause substantial financial damage, expose sensitive data, and progressively degrade performance. To mitigate these risks, there is a pressing need for intelligent systems capable of autonomously detecting, identifying, and pinpointing faults without relying on manual inspection of OTDR (optical time-domain reflectometry) traces. This study introduces an advanced machine learning-based solution for analyzing OTDR signals during fault recovery. It integrates a semi-supervised anomaly detection ensemble to uncover both previously known and novel faults and employs a multitask BiLSTM architecture enhanced with an attention mechanism to accurately locate and classify the type of fiber issues. The semi-supervised model ensures robust performance even with limited labeled samples. Tests conducted on real OTDR datasets demonstrate high effectiveness, achieving an anomaly detection F1-score of 98.78 % and fault classification accuracy of 98.62 %, along with precise localization. Implementing this approach can greatly enhance the fault tolerance of optical networks, reducing downtime and preserving data security in the face of both attacks and accidental damages.
Acknowledgment
The authors would like to thank their institutes for their support and assistance during the course of this study.
-
Research ethics: This study was conducted in accordance with the ethical standards.
-
Informed consent: Not applicable.
-
Author contributions: Author 1 and 2: Conceptualization, Methodology, Writing – Original Draft. [Author 3]: Data curation, Formal analysis, Writing – Review & Editing. [Author 4]: Supervision, Project administration. All authors read and approved the final manuscript.
-
Use of Large Language Models, AI and Machine Learning Tools: The authors declare that no generative AI or machine learning tools were used for the generation of data, figures, or analysis in this study.
-
Conflict of interest: The authors declare that they have no conflict of interest.
-
Research funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
-
Data availability: The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
1. Nouioua, M, Fournier-Viger, P, He, G, Nouioua, F, Min, Z. A survey of machine learning for network fault management. In: Chiroma, H, Abdulhamid, Si. M, Fournier-Viger, P, Garcia, NM, editors. Machine learning and data mining for emerging trend in cyber dynamics: theories and applications. Cham: Springer International Publishing; 2021:1–27 pp.10.1007/978-3-030-66288-2_1Suche in Google Scholar
2. Borland, J. Analyzing the internet collapse. MIT Technol Rev 2008.Suche in Google Scholar
3. Lee, W, Myong, SI, Lee, JC, Lee, S. Identification method of non-reflective faults based on index distribution of optical fibers. Opt Express 2014;22:325–37. https://doi.org/10.1364/oe.22.000325.Suche in Google Scholar PubMed
4. Thyagaturu, AS, Mercian, A, McGarry, MP, Reisslein, M, Kellerer, W. Software defined optical networks (SDONs): a comprehensive survey. IEEE Commu Sur Tuto 2016;18:2738–86. https://doi.org/10.1109/comst.2016.2586999.Suche in Google Scholar
5. Musumeci, F, Rottondi, C, Nag, A, Macaluso, I, Zibar, D, Ruffini, M, et al.. An overview on application of machine learning techniques in optical networks. IEEE Commu Sur Tuto 2019;21:1383–408. https://doi.org/10.1109/comst.2018.2880039.Suche in Google Scholar
6. Khan, FN, Fan, Q, Lu, C, Lau, APT. An optical communication’s perspective on machine learning and its applications. J Lightwave Technol 2019;37:493–516. https://doi.org/10.1109/jlt.2019.2897313.Suche in Google Scholar
7. Rafique, D, Velasco, L. Machine learning for network automation: overview, architecture, and applications [Invited Tutorial]. J Opt Commun Netw 2018;10:D126–43. https://doi.org/10.1364/jocn.10.00d126.Suche in Google Scholar
8. Musumeci, F, Rottondi, C, Corani, G, Shahkarami, S, Cugini, F, Tornatore, M, et al.. A tutorial on machine learning for failure management in optical networks. J Lightwave Technol 2019;37:4125–39. https://doi.org/10.1109/jlt.2019.2922586.Suche in Google Scholar
9. Abdelli, K, Grießer, H, Tropschug, C, Pachnicke, S. Optical fiber fault detection and localization in a noisy OTDR trace based on denoising convolutional autoencoder and bidirectional long short-term memory. J Lightwave Technol 2022;40:2254–64. https://doi.org/10.1109/jlt.2021.3138268.Suche in Google Scholar
10. Bakar, AAA, Jamaludin, MZ, Abdullah, F, Yaacob, MH, Mahdi, MA, Abdullah, MK, et al.. A new technique of real-time monitoring of fiber optic cable networks transmission. Opt Laser Eng 2007;45:126–30. https://doi.org/10.1016/j.optlaseng.2006.03.009.Suche in Google Scholar
11. Chun-Kit, C, Tong, F, Lian-Kuan, C, Keang-Po, H, Lam, D. Fiber-fault identification for branched access networks using a wavelength-sweeping monitoring source. IEEE Photon Technol Lett 1999;11:614–16. https://doi.org/10.1109/68.759416.Suche in Google Scholar
12. Abdelli, K, Cho, JY, Azendorf, F, Griesser, H, Tropschug, C, Pachnicke, S, et al.. Machine-learning-based anomaly detection in optical fiber monitoring. J Opt Commun Netw 2022;14:365–75. https://doi.org/10.1364/jocn.451289.Suche in Google Scholar
13. Shaneman, K, Gray, S. Optical network security: technical analysis of fiber tapping mechanisms and methods for detection & prevention. IEEE MILCOM 2004. Military Commun Confer 2004;2:711–16.10.1109/MILCOM.2004.1494884Suche in Google Scholar
14. Nassif, AB, Talib, MA, Nasir, Q, Dakalbab, FM. Machine learning for anomaly detection: a systematic review. IEEE Access 2021;9:78658–700. https://doi.org/10.1109/access.2021.3083060.Suche in Google Scholar
15. Wang, H, Bah, MJ, Hammad, M. Progress in outlier detection techniques: a survey. IEEE Access 2019;7:107964–8000. https://doi.org/10.1109/access.2019.2932769.Suche in Google Scholar
16. Hodge, VJ, Austin, J. A survey of outlier detection methodologies. Artif Intell Rev 2004;22:85–126. https://doi.org/10.1007/s10462-004-4304-y.Suche in Google Scholar
17. Djenouri, Y, Belhadi, A, Lin, JCW, Djenouri, D, Cano, A. A survey on urban traffic anomalies detection algorithms. IEEE Access 2019;7:12192–205. https://doi.org/10.1109/access.2019.2893124.Suche in Google Scholar
18. Breunig, MM, Kriegel, HP, Ng, RT, Sander, J. LOF: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD international conference on Management of data; 2000:93–104 pp.10.1145/342009.335388Suche in Google Scholar
19. Alghushairy, O, Alsini, R, Soule, T, Ma, X. A review of local outlier factor algorithms for outlier detection in big data streams. Big Data and Cognitive Computing 2020;5. https://doi.org/10.3390/bdcc5010001.Suche in Google Scholar
20. Chabchoub, Y, Togbe, MU, Boly, A, Chiky, R. An In-Depth study and improvement of isolation forest. IEEE Access 2022;10:10219–37. https://doi.org/10.1109/access.2022.3144425.Suche in Google Scholar
21. Kriegel, H-P, Schubert, M, Zimek, A. Angle-based outlier detection in high-dimensional data. In: Presented at the proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining. Nevada, USA: Las Vegas; 2008.10.1145/1401890.1401946Suche in Google Scholar
22. Shin, HJ, Eom, D-H, Kim, S-S. One-class support vector machines–an application in machine fault detection and classification. Comput Ind Eng 2005;48:395–408. https://doi.org/10.1016/j.cie.2005.01.009.Suche in Google Scholar
23. Zhu, F, Yang, J, Gao, C, Xu, S, Ye, N, Yin, T, et al.. A weighted one-class support vector machine. Neurocomputing 2016;189:1–10. https://doi.org/10.1016/j.neucom.2015.10.097.Suche in Google Scholar
24. Schölkopf, B, Platt, JC, Shawe-Taylor, J, Smola, AJ, Williamson, RC. Estimating the support of a high-dimensional distribution. Neural Comput 2001;13:1443–71. https://doi.org/10.1162/089976601750264965.Suche in Google Scholar PubMed
25. Hardin, J, Rocke, DM. Outlier detection in the multiple cluster setting using the minimum covariance determinant estimator. Comput Stat Data Anal 2004;44:625–38. https://doi.org/10.1016/s0167-9473(02)00280-3.Suche in Google Scholar
26. Rousseeuw, PJ, Driessen, KV. A fast algorithm for the minimum covariance determinant estimator. Technometrics 1999;41:212–23. https://doi.org/10.2307/1270566.Suche in Google Scholar
27. Sherstinsky, A. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Phys Nonlinear Phenom 2020;404:132306. https://doi.org/10.1016/j.physd.2019.132306.Suche in Google Scholar
28. Mitra, H, Jalali, M. A survey of long short-term memory (LSTM) algorithm and its application. kiaeee 2021;8:70–7.Suche in Google Scholar
29. Lindemann, B, Müller, T, Vietz, H, Jazdi, N, Weyrich, M. A survey on long short-term memory networks for time series prediction. Proced CIRP 2021;99:650–5. https://doi.org/10.1016/j.procir.2021.03.088.Suche in Google Scholar
30. Thakur, D. LSTM and its equations, google summer of code; 2019. https://medium.com/@divyanshu132/lstm-and-its-equations-5ee9246d04af [Accessed 20 Dec 2022].Suche in Google Scholar
31. Shahkarami, S, Musumeci, F, Cugini, F, Tornatore, M. Machine-learning-based soft-failure detection and identification in optical networks. In: 2018 Optical Fiber Communications Conference and Exposition (OFC); 2018.10.1364/OFC.2018.M3A.5Suche in Google Scholar
32. Boitier, F. Proactive fiber damage detection in real-time coherent receiver. In: 2017 European Conference on Optical Communication (ECOC); 2017.10.1109/ECOC.2017.8346077Suche in Google Scholar
33. Panayiotou, T, Chatzis, SP, Ellinas, G. Leveraging statistical machine learning to address failure localization in optical networks. J Opt Commun Netw 2018;10:16. https://doi.org/10.1364/jocn.10.000162.Suche in Google Scholar
34. Shu, L, Yu, Z, Wan, Z, Zhang, J, Hu, S, Xu, K, et al.. Dual-stage soft failure detection and identification for low-margin elastic optical network by exploiting digital spectrum information. J Lightwave Technol 2020;38:2669–79. https://doi.org/10.1109/jlt.2019.2947562.Suche in Google Scholar
35. Furdek, M, Natalino, C, Giglio, AD, Schiano, M. Optical network security management: requirements, architecture, and efficient machine learning models for detection of evolving threats [invited]. J Opt Commun Netw 2021;13:A144–55. https://doi.org/10.1364/jocn.402884.Suche in Google Scholar
36. Lun, H, Fu, M, Liu, X, Wu, Y, Yi, L, Hu, W, et al.. Soft failure identification for long-haul optical communication systems based on one- dimensional convolutional neural network. J Lightwave Technol 2020;38:2992–9. https://doi.org/10.1109/jlt.2020.2989153.Suche in Google Scholar
37. Zhang, C, Wang, D, Jia, J, Wang, L, Chen, K, Guan, L, et al.. Potential failure cause identification for optical networks using deep learning with an attention mechanism. J Opt Commun Netw 2022;14:A122–33. https://doi.org/10.1364/jocn.438900.Suche in Google Scholar
38. Babbar, J, Triki, A, Ayassi, R, Laye, M. Machine learning models for alarm classification and failure localization in optical transport networks. J Opt Commun Netw 2022;14:621–8. https://doi.org/10.1364/jocn.457687.Suche in Google Scholar
39. Karandin, O, Ayoub, O, Musumeci, F, Hirota, Y, Awaji, Y, Tornatore, M, et al.. If not here, there. Explaining machine learning models for fault localization in optical networks. In: 2022 International Conference on Optical Network Design and Modeling (ONDM); 2022.10.23919/ONDM54585.2022.9782859Suche in Google Scholar
40. F Musumeci, Venkata, VG, Hirota, Y, Awaji, Y, Xu, S, Shiraiwa, M, et al.. Domain adaptation and transfer learning for failure detection and failure-cause identification in optical networks across different lightpaths [invited]. J Opt Commun Netw 2022;14:A91-100 https://doi.org/10.1364/jocn.438269.Suche in Google Scholar
41. Vela, AP, Shariati, B, Ruiz, M, Cugini, F, Castro, A, Lu, H, et al.. Soft failure localization during commissioning testing and lightpath operation. J Opt Commun Netw 2018;10:A27–36. https://doi.org/10.1364/jocn.10.000a27.Suche in Google Scholar
42. Yang, Z, Hong, D, Feng, X, Xie, J. A novel event detection method for OTDR trace with high sensitivity based on machine learning. In: 2021 2nd Information Communication Technologies Conference (ICTC) 2021.10.1109/ICTC51749.2021.9441614Suche in Google Scholar
43. Abdelli, K, Griesser, H, Ehrle, P, Tropschug, C, Pachnicke, S. Reflective fiber fault detection and characterization using long short-term memory. J Opt Commun Netw 2021;13:E32–41. https://doi.org/10.1364/jocn.423625.Suche in Google Scholar
44. Abdelli, K, Grießer, H, Tropschug, C, Pachnicke, S. A BiLSTM-CNN based multitask learning approach for fiber fault diagnosis. In: 2021 Optical Fiber Communications Conference and Exhibition (OFC); 2021.10.1364/OFC.2021.M3C.7Suche in Google Scholar
45. Abdelli, K, Griesser, H, Pachnicke, S. Convolutional neural networks for reflective event detection and characterization in fiber optical links given noisy OTDR signals. In: Photonic Networks; 22th ITG Symposium. VDE; 2021:1–5 pp.Suche in Google Scholar
46. Reis, I, Baron, D, Shahaf, S. Probabilistic random forest: a machine learning algorithm for noisy data sets. Astron J 2019;157:16. https://doi.org/10.3847/1538-3881/aaf101.Suche in Google Scholar
47. Yang, L, Meng, X, Karniadakis, GE. B-PINNs: bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. J Comput Phys 2021;425:109913. https://doi.org/10.1016/j.jcp.2020.109913.Suche in Google Scholar
48. Abdelli, K, Azendorf, F, Tropschug, C, Griesser, H, Pachnicke, S, Choo, J, et al.. Dataset for optical fiber faults. IEEE Dataport 2022.https://doi.org/10.21227/pdpn-1b78.Suche in Google Scholar
49. Aggarwal, CC, Sathe, S. An introduction to outlier ensembles. In: Aggarwal, CC, Sathe, S, editors. Outlier Ensembles: An Introduction. Cham: Springer International Publishing; 2017.10.1007/978-3-319-54765-7Suche in Google Scholar
50. Zimek, A, Campello, RJGB, Sander, J. Ensembles for unsupervised outlier detection: challenges and research questions a position paper. SIGKDD Explor Newsl. 2014;15:11–22. https://doi.org/10.1145/2594473.2594476.Suche in Google Scholar
51. Zhao, Y. SUOD: accelerating large-scale unsupervised heterogeneous outlier detection. In: Conference on machine learning and systems (MLSys); 2021.Suche in Google Scholar
52. Y Zhao, X Ding, J Yang, H Bai. SUOD: toward scalable unsupervised outlier detection. arXiv preprint arXiv:2002.03222 2020.Suche in Google Scholar
53. Achlioptas, D. Database-friendly random projections. In: Presented at the proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on principles of database systems. Santa Barbara, California, USA; 2001:274–81 pp.10.1145/375551.375608Suche in Google Scholar
54. Shyu, M-L, Chen, S-C, Sarinnapakorn, K, Chang, L. A novel anomaly detection scheme based on principal component classifier. In: Proceedings of the IEEE foundations and new directions of data mining workshop 2003.Suche in Google Scholar
55. Keller, F, Muller, E, Bohm, K. HiCS: high contrast subspaces for density-based outlier ranking. In: 2012 IEEE 28th International Conference on Data Engineering 2012:1037–48 pp.10.1109/ICDE.2012.88Suche in Google Scholar
56. Schubert, E, Zimek, A, Kriegel, H-P. Fast and scalable outlier detection with approximate nearest neighbor ensembles. In: Renz, M, Shahabi, C, Zhou, X, Cheema, MA, editors. Database Systems for Advanced Applications. Cham: Springer International Publishing; 2015:19–36 pp.10.1007/978-3-319-18123-3_2Suche in Google Scholar
57. Bahdanau, D. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 2014.Suche in Google Scholar
58. Pedregosa, F. Scikit-learn: machine learning in python. J Mach Learn Res 2011;12:2825–30.Suche in Google Scholar
59. Y Zhao, Z Nasrullah, Z Li. Pyod: a python toolbox for scalable outlier detection. arXiv preprint arXiv:1901.01588 2019.Suche in Google Scholar
60. Achtert, E, Kriegel, H-P, Reichert, L, Schubert, E, Wojdanowski, R, Zimek, A, et al.. Visual evaluation of outlier detection models. In: Kitagawa, H, Ishikawa, Y, Li, Q, Watanabe, C, editors. Database Systems for Advanced Applications. Berlin, Heidelberg: Springer Berlin Heidelberg; 2010:396–9 pp.10.1007/978-3-642-12098-5_34Suche in Google Scholar
61. Cruzes, S. Failure management overview in optical networks. IEEE Access 2024. https://doi.org/10.1109/access.2024.3498704.Suche in Google Scholar
62. Wang, D, Zhang, C, Chen, W, Yang, H, Zhang, M, Tao Lau, AP, et al.. A review of machine learning-based failure management in optical networks. Sci China Inf Sci 2022;65:211302. https://doi.org/10.1007/s11432-022-3557-9.Suche in Google Scholar
63. Zhang, C, Zhang, M, Liu, S, Liu, Z, Wang, D. Covert fault detection with imbalanced data using an improved autoencoder for optical networks. J Opt Commun Netw 2023;15:913–24. https://doi.org/10.1364/jocn.502937.Suche in Google Scholar
© 2025 Walter de Gruyter GmbH, Berlin/Boston
