Comparison of different ML methods concerning prediction quality, domain adaptation and robustness

Payman Goodarzi; Andreas Schütze; Tizian Schneider

doi:10.1515/teme-2021-0129

Article

Comparison of different ML methods concerning prediction quality, domain adaptation and robustness

Payman Goodarzi

Payman Goodarzi studied Embedded Systems at Saarland University and received his Master of Science degree in March 2020 with a thesis on the interpretability of neural networks. Since that time, he has been working at the Lab for Measurement Technology (LMT) of Saarland University and at the Centre for Mechatronics and Automation Technology (ZeMA) as a scientific researcher. His research interests include ML and deep learning for condition monitoring of technical systems.
, Andreas Schütze

Andreas Schütze received his diploma in physics from RWTH Aachen in 1990 and his doctorate in Applied Physics from Justus-Liebig-Universität in Gießen in 1994 with a thesis on microsensors and sensor systems for the detection of reducing and oxidizing gases. From 1994 until 1998 he worked for VDI/VDE-IT, Teltow, Germany, mainly in the fields of microsystems technology. From 1998 until 2000 he was professor for Sensors and Microsystem Technology at the University of Applied Sciences in Krefeld, Germany. Since April 2000 he is professor for Measurement Technology in the Department Systems Engineering at Saarland University, Saarbrücken, Germany and head of the Laboratory for Measurement Technology (LMT). His research interests include smart gas sensor systems as well as data engineering methods for industrial applications.
and Tizian Schneider

Tizian Schneider studied Microtechnologies and Nanostructures at Saarland University and received his Master of Science degree in January 2016. Since that time, he has been working at the Lab for Measurement Technology (LMT) of Saarland University and at the Centre for Mechatronics and Automation Technology (ZeMA) leading the research group Data Engineering & Smart Sensors. His research interests include ML methods for condition monitoring of technical systems, automatic ML model building and interpretable AI.

Published/Copyright: February 25, 2022

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal tm - Technisches Messen Volume 89 Issue 4

Abstract

Nowadays machine learning methods and data-driven models have been used widely in different fields including computer vision, biomedicine, and condition monitoring. However, these models show performance degradation when meeting real-life situations. Domain or dataset shift or out-of-distribution (OOD) prediction is mentioned as the reason for this problem. Especially in industrial condition monitoring, it is not clear when we should be concerned about domain shift and which methods are more robust against this problem. In this paper prediction results are compared for a conventional machine learning workflow based on feature extraction, selection, and classification/regression (FESC/R) and deep neural networks on two publicly available industrial datasets. We show that it is possible to visualize the possible shift in domain using feature extraction and principal component analysis. Also, experimental competition shows that the cross-domain validated results of FESC/R are comparable to the reported state-of-the-art methods. Finally, we show that the results for simple randomly selected validation sets do not correctly represent the model performance in real-world applications.

Zusammenfassung

Machine Learning und datenbasierte Modelle sind in der Literatur zu Computer Vision, Biomedizin oder Zustandsüberwachung weit verbreitet. Allerdings zeigen diese Methoden oft Schwächen in der realen Anwendung. Domain Shift oder Vorhersagen außerhalb der Verteilung der Trainingsdaten werden häufig als Ursache benannt. Besonders bei industrieller Zustandsüberwachung ist unklar, wann diese Probleme auftreten und welche Algorithmen robust dagegen sind. In diesem Beitrag werden die Ergebnisse einer klassischen ML-Auswertekette bestehend aus Merkmalsextraktion, Merkmalsselektion und Klassifikation bzw. Regression (FESC/R) mit jenen von mehrschichtigen neuronalen Netzen auf zwei öffentlich verfügbaren Datensätzen verglichen. Es wird gezeigt, dass mögliche Datenverschiebungen mittels Merkmalsextraktion und Hauptkomponentenanalyse sichtbar gemacht werden können. Weiterhin wird gezeigt, dass die mit FESC/R auf Domain Shift Problemen erreichten Ergebnisse gleichwertig zu denen von mehrschichtigen neuronalen Netzen sind. Letztlich wird gezeigt, dass eine zufällige Kreuzvalidierung die in einer realen Anwendung zu erwartende Genauigkeit eines ML-Modells nicht hinreichend abbilden kann.

Keywords: Machine learning; condition monitoring; domain adaptation; neural network

Schlagwörter: maschinelles Lernen; Zustandsüberwachung; Domänenadaption; neuronale Netze

Funding source: European Regional Development Fund

Award Identifier / Grant number: SE-ProEng

Funding source: Bundesministerium für Bildung und Frauen

Award Identifier / Grant number: 16ME0077

Funding statement: This work was in part supported by the European Regional Development Fund (Europäischer Fonds für regionale Entwicklung, EFRE) within the project „SE-ProEng“. This work was in part supported by the German ministry for education and research (BMBF) within the project „KI-MUSIK4.0“ under code 16ME0077.

About the authors

Payman Goodarzi

Payman Goodarzi studied Embedded Systems at Saarland University and received his Master of Science degree in March 2020 with a thesis on the interpretability of neural networks. Since that time, he has been working at the Lab for Measurement Technology (LMT) of Saarland University and at the Centre for Mechatronics and Automation Technology (ZeMA) as a scientific researcher. His research interests include ML and deep learning for condition monitoring of technical systems.

Andreas Schütze

Andreas Schütze received his diploma in physics from RWTH Aachen in 1990 and his doctorate in Applied Physics from Justus-Liebig-Universität in Gießen in 1994 with a thesis on microsensors and sensor systems for the detection of reducing and oxidizing gases. From 1994 until 1998 he worked for VDI/VDE-IT, Teltow, Germany, mainly in the fields of microsystems technology. From 1998 until 2000 he was professor for Sensors and Microsystem Technology at the University of Applied Sciences in Krefeld, Germany. Since April 2000 he is professor for Measurement Technology in the Department Systems Engineering at Saarland University, Saarbrücken, Germany and head of the Laboratory for Measurement Technology (LMT). His research interests include smart gas sensor systems as well as data engineering methods for industrial applications.

Tizian Schneider

Tizian Schneider studied Microtechnologies and Nanostructures at Saarland University and received his Master of Science degree in January 2016. Since that time, he has been working at the Lab for Measurement Technology (LMT) of Saarland University and at the Centre for Mechatronics and Automation Technology (ZeMA) leading the research group Data Engineering & Smart Sensors. His research interests include ML methods for condition monitoring of technical systems, automatic ML model building and interpretable AI.

References

1. R. K. Mobley, An introduction to predictive maintenance. Elsevier, 2002.10.1016/B978-075067531-4/50006-3Search in Google Scholar

2. A. Schütze, N. Helwig, and T. Schneider, “Sensors 4.0 – smart sensors and measurement technology enable Industry 4.0,” Journal of Sensors and Sensor systems, vol. 7, no. 1, pp. 359–371, 2018.10.5194/jsss-7-359-2018Search in Google Scholar

3. D. C. Montgomery, Design and analysis of experiments. John Wiley & Sons, 2017.Search in Google Scholar

4. P. W. Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” International Conference on Machine Learning, pp. 5637–5664, Dec. 2021.Search in Google Scholar

5. J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. v. Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” Pattern Recognition, vol. 45, no. 1, pp. 521–530, Jan. 2012, doi: 10.1016/j.patcog.2011.06.019.Search in Google Scholar

6. G. Widmer and M. Kubat, “Learning in the presence of concept drift and hidden contexts,” Machine Learning, vol. 23, no. 1, pp. 69–101, Apr. 1996, doi: 10.1007/BF00116900.Search in Google Scholar

7. M. G. Kelly, D. J. Hand, N. M. Adams, “The impact of changing populations on classifier performance,” Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 367–371, 1999, doi: 10.1145/312129.312285.Search in Google Scholar

8. H. Shimodaira, “Improving predictive inference under covariate shift by weighting the log-likelihood function,” Journal of Statistical Planning and Inference, vol. 90, no. 2, pp. 227–244, Oct. 2000, doi: 10.1016/S0378-3758(00)00115-4.Search in Google Scholar

9. D. A. Cieslak and N. v. Chawla, “A framework for monitoring classifiers’ performance: when and why failure occurs?,” Knowledge and Information Systems, vol. 18, no. 1, pp. 83–108, Jan. 2009, doi: 10.1007/s10115-008-0139-1.Search in Google Scholar

10. R. Alaiz-Rodríguez, A. Guerrero-Curieses, and J. Cid-Sueiro, “Minimax regret classifier for imprecise class distributions,” Journal of Machine Learning Research, vol. 8, pp. 103–130, 2007.Search in Google Scholar

11. R. Caruana, “Multitask Learning,” Machine Learning, vol. 28, no. 1, pp. 41–75, 1997, doi: 10.1023/A:1007379606734.Search in Google Scholar

12. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A Kernel Two-Sample Test,” Journal of Machine Learning Research, vol. 13, no. 25, pp. 723–773, 2012.Search in Google Scholar

13. K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting Visual Category Models to New Domains,” European conference on computer vision, pp. 213–226, 2010, doi: 10.1007/978-3-642-15561-1_16.Search in Google Scholar

14. Y. Ganin and V. Lempitsky, “Unsupervised Domain Adaptation by Backpropagation,” ICML’15: Proceedings of the 32^nd International Conference on Machine Learning – Volume 37, pp. 1180–1189, Jul. 2015.Search in Google Scholar

15. Y. Ganin et al., “Domain-adversarial training of neural networks,” The journal of machine learning research, vol. 17, no. 1, pp. 2030–2096, 2016.10.1007/978-3-319-58347-1_10Search in Google Scholar

16. E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial Discriminative Domain Adaptation,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2962–2971, Jul. 2017, doi: 10.1109/CVPR.2017.316.Search in Google Scholar

17. X. Li, W. Zhang, Q. Ding, and J.-Q. Sun, “Multi-Layer domain adaptation method for rolling bearing fault diagnosis,” Signal Processing, vol. 157, pp. 180–197, Apr. 2019, doi: 10.1016/j.sigpro.2018.12.005.Search in Google Scholar

18. W.-G. Chang, T. You, S. Seo, S. Kwak, and B. Han, “Domain-Specific Batch Normalization for Unsupervised Domain Adaptation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2019.10.1109/CVPR.2019.00753Search in Google Scholar

19. Z. Lu, Y. Yang, X. Zhu, C. Liu, Y.-Z. Song, and T. Xiang, “Stochastic Classifiers for Unsupervised Domain Adaptation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020.10.1109/CVPR42600.2020.00913Search in Google Scholar

20. W. Li, Z. Yuan, W. Sun, and Y. Liu, “Domain Adaptation for Intelligent Fault Diagnosis under Different Working Conditions,” MATEC Web of Conferences, vol. 319, p. 03001, Sep. 2020, doi: 10.1051/matecconf/202031903001.Search in Google Scholar

21. X. Wang, F. Liu, and D. Zhao, “Cross-Machine Fault Diagnosis with Semi-Supervised Discriminative Adversarial Domain Adaptation,” Sensors, vol. 20, no. 13, p. 3753, Jul. 2020, doi: 10.3390/s20133753.Search in Google Scholar PubMed PubMed Central

22. W. Zhang, G. Peng, C. Li, Y. Chen, and Z. Zhang, “A New Deep Learning Model for Fault Diagnosis with Good Anti-Noise and Domain Adaptation Ability on Raw Vibration Signals,” Sensors, vol. 17, no. 2, p. 425, Feb. 2017, doi: 10.3390/s17020425.Search in Google Scholar PubMed PubMed Central

23. N. Helwig, E. Pignanelli, and A. Schütze, “Condition monitoring of a complex hydraulic system using multivariate statistics,” 2015 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Proceedings, pp. 210–215, May 2015, doi: 10.1109/I2MTC.2015.7151267.Search in Google Scholar

24. T. Schneider, S. Klein, and M. Bastuck, “Condition monitoring of hydraulic systems Data Set at ZeMA,” Zenodo, Apr. 2018, doi: 10.5281/ZENODO.1323611.Search in Google Scholar

25. “Case Western Reserve University Bearing Data Set,” Case Western Reserve University Bearing Data Center. https://engineering.case.edu/bearingdatacenter.Search in Google Scholar

26. S. Zhang, S. Zhang, B. Wang, and T. G. Habetler, “Deep learning algorithms for bearing fault diagnostics – A comprehensive review,” IEEE Access, vol. 8, pp. 29857–29881, 2020.10.1109/ACCESS.2020.2972859Search in Google Scholar

27. T. Schneider, N. Helwig, and A. Schütze, “Industrial condition monitoring with smart sensors using automated feature extraction and selection,” Measurement Science and Technology, vol. 29, no. 9, p. 94002, Aug. 2018, doi: 10.1088/1361-6501/aad1d4.Search in Google Scholar

28. A. Widodo and B.-S. Yang, “Support vector machine in machine condition monitoring and fault diagnosis,” Mechanical Systems and Signal Processing, vol. 21, no. 6, pp. 2560–2574, Aug. 2007, doi: 10.1016/j.ymssp.2006.12.007.Search in Google Scholar

29. D.-T. Hoang and H.-J. Kang, “A survey on Deep Learning based bearing fault diagnosis,” Neurocomputing, vol. 335, pp. 327–335, Mar. 2019, doi: 10.1016/j.neucom.2018.06.078.Search in Google Scholar

30. M. S. Mahdavinejad, M. Rezvan, M. Barekatain, P. Adibi, P. Barnaghi, and A. P. Sheth, “Machine learning for internet of things data analysis: a survey,” Digital Communications and Networks, vol. 4, no. 3, pp. 161–175, 2018, doi: 10.1016/j.dcan.2017.10.002.Search in Google Scholar

31. W. Zhang, D. Yang, and H. Wang, “Data-Driven Methods for Predictive Maintenance of Industrial Equipment: A Survey,” IEEE Systems Journal, vol. 13, no. 3, pp. 2213–2227, Sep. 2019, doi: 10.1109/JSYST.2019.2905565.Search in Google Scholar

32. A. Preece, D. Harborne, D. Braines, R. Tomsett, and S. Chakraborty, “Stakeholders in Explainable AI,” arXiv preprint arXiv:1810.00184, Sep. 2018.Search in Google Scholar

33. C. Schorr, P. Goodarzi, F. Chen, and T. Dahmen, “Neuroscope: An Explainable AI Toolbox for Semantic Segmentation and Image Classification of Convolutional Neural Nets,” Applied Sciences, vol. 11, no. 5, 2021, doi: 10.3390/app11052199.Search in Google Scholar

34. T. Dorst, Y. Robin, S. Eichstädt, A. Schütze, and T. Schneider, “Influence of synchronization within a sensor network on machine learning results,” Journal of Sensors and Sensor Systems, vol. 10, no. 2, pp. 233–245, Aug. 2021, doi: 10.5194/jsss-10-233-2021.Search in Google Scholar

35. Y. Robin, P. Goodarzi, T. Baur, C. Schultealbert, A. Schütze, and T. Schneider, “Machine Learning based calibration time reduction for Gas Sensors in Temperature Cycled Operation,” 2021 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), pp. 1–6, May 2021, doi: 10.1109/I2MTC50364.2021.9459919.Search in Google Scholar

36. T. Schneider, S. Klein, and A. Schütze, “Machine learning in industrial measurement technology for detection of known and unknown faults of equipment and sensors,” tm – Technisches Messen, vol. 86, no. 11, pp. 706–718, Nov. 2019, doi: 10.1515/teme-2019-0086.Search in Google Scholar

37. Y. Robin et al., “High-Performance VOC Quantification for IAQ Monitoring Using Advanced Sensor Systems and Deep Learning,” Atmosphere, vol. 12, no. 11, p. 1487, Nov. 2021, doi: 10.3390/atmos12111487.Search in Google Scholar

38. I. Kononenko, E. Šimec, and M. Robnik-Šikonja, “Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF,” Applied Intelligence, vol. 7, no. 1, pp. 39–55, 1997, doi: 10.1023/A:1008280620621.Search in Google Scholar

39. R. A. Fisher, “The Use of Multiple Measurements in Taxonomic Problems,” Annals of Eugenics, vol. 7, no. 2, pp. 179–188, Sep. 1936, doi: 10.1111/j.1469-1809.1936.tb02137.x.Search in Google Scholar

40. S. Wold, M. Sjöström, and L. Eriksson, “PLS-regression: a basic tool of chemometrics,” Chemometrics and Intelligent Laboratory Systems, vol. 58, no. 2, pp. 109–130, Oct. 2001, doi: 10.1016/S0169-7439(01)00155-1.Search in Google Scholar

41. T. Brown et al., “Language Models are Few-Shot Learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. 2020.Search in Google Scholar

42. Z. Dai, H. Liu, Q. v. Le, and M. Tan, “CoAtNet: Marrying Convolution and Attention for All Data Sizes,” arXiv preprint arXiv:2106.04803, June 2021.Search in Google Scholar

43. P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-Aware Minimization for Efficiently Improving Generalization,” arXiv preprint arXiv:2010.01412, Oct. 2020.Search in Google Scholar

44. B. Zoph and Q. v. Le, “Neural Architecture Search with Reinforcement Learning,” arXiv preprint arXiv:1611.01578, Nov. 2016.Search in Google Scholar

45. T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” The Journal of Machine Learning Research, vol. 20, no. 1, pp. 1997–2017, 2019.10.1007/978-3-030-05318-5_11Search in Google Scholar

46. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016.10.1109/CVPR.2016.90Search in Google Scholar

47. A. van den Oord et al., “WaveNet: A Generative Model for Raw Audio,” arXiv preprint arXiv:1609.03499, Sep. 2016.Search in Google Scholar

48. J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” Advances in neural information processing systems, vol. 25, Jun. 2012.Search in Google Scholar

49. M. Holschneider, R. Kronland-Martinet, J. Morlet, and Ph. Tchamitchian, “A Real-Time Algorithm for Signal Analysis with the Help of the Wavelet Transform,” in Wavelets, Springer, Berlin, Heidelberg, 1990, pp. 286–297, doi: 10.1007/978-3-642-75988-8_28.Search in Google Scholar

50. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, Mar. 2021, doi: 10.1145/3446776.Search in Google Scholar

51. R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt, “Measuring Robustness to Natural Distribution Shifts in Image Classification,” arXiv preprint arXiv:2007.00644, Jul. 2020.Search in Google Scholar

52. D. N. Perkins, G. Salomon, et al., “Transfer of learning,” International encyclopedia of education, vol. 2, pp. 6452–6457, 1992.Search in Google Scholar

53. S. Bozinovski, “Reminder of the First Paper on Transfer Learning in Neural Networks, 1976,” Informatica (Slovenia), vol. 44, 2020.10.31449/inf.v44i3.2828Search in Google Scholar

54. I. Gulrajani and D. Lopez-Paz, “In Search of Lost Domain Generalization,” arXiv preprint arXiv:2007.01434, Jul. 2020.Search in Google Scholar

55. S. Youssef, “Einsatz maschineller Lernalgorithmen zur mikromagnetischen Materialcharakterisierung,” dissertation, Saarland University, 2021.Search in Google Scholar

Received: 2021-12-14

Accepted: 2022-02-03

Published Online: 2022-02-25

Published in Print: 2022-04-30

You are currently not able to access this content.

Articles in the same Issue

https://doi.org/10.1515/teme-2021-0129

Keywords for this article

Machine learning; condition monitoring; domain adaptation; neural network