Developing and Qualifying an ML Application for MRO Assistance

Introduction

In recent years, machine learning (ML) has gained increasing prominence within the industry, particularly in enabling the capabilities of Industry 4.0 [1]. Integrating ML into industrial processes is expected to yield benefits such as enhanced performance [2]. Specifically, in maintenance management, a transition from reactive and periodic maintenance to predictive and proactive maintenance is anticipated [3]. Maintenance management is often referred to as Maintenance, Repair, and Overhaul (MRO), which encompasses many cost-efficient procedures. The MRO process is commonly applied to overhaul gas turbines to minimize life cycle costs and optimize component performance. The availability and efficiency of gas turbine units are two key concerns for operators regarding performance optimization [4]. Many components have a limited service life relative to the overall operational lifespan of the gas turbine system [5], making maintenance and repair a longstanding focus for operators.

Traditionally, maintenance has been conducted through static repair process chains, in which the same value-adding steps are repeatedly executed. However, this manual and static approach is often inefficient. Typically, engineers consult the Repair Specification Manual (RSM) to identify appropriate repair actions or subsequent inspection processes for observed material defects. This process is monotonous, prone to errors, and incompatible with the real-time capabilities required for documentation management in Industry 4.0 [6], as the RSM documentation is searched manually and updated infrequently. To address these challenges, using ML applications is becoming an increasingly viable solution [7].

Engineering Case

In this paper, a turbine blade from a gas turbine is considered as the technical system under study. The turbine blades undergo an MRO process that initially involves stripping, cleaning, and inspection. The inspection results determine the scope of repair and individual activities before the repair process begins. This paper examines two subprocesses in more detail: damage detection and repair estimation.

Damage detection primarily focuses on inspecting surface cracks, spallation, and material loss. Traditionally, after stripping, the blades were manually inspected for cracks by inspectors, who documented them in 2D drawings. To automate this process, cameras were employed to capture image data of the turbine blades, and a neural network was trained to detect cracks from the images and map them onto a 3D model of the blade. Repair estimation involves deciding whether the turbine blade should be repaired, left as is, or discarded. This step begins once all inspection processes have been completed and the findings are entered into a system. Inspectors then review the findings to assess whether they conform to the RSM, derive the necessary repair steps, or identify deviations from the RSM. Engineers manually review the findings in cases of deviations and determine the appropriate dispositions. A decision tree model was implemented to estimate dispositions for damages not defined in the RSM, utilizing historical data from previous decisions related to such damages.

Each ML application was evaluated as a standalone solution using the precision, accuracy, and F1 score metrics. The ML model for damage detection achieved different accuracies depending on the specific damage. An accuracy of 80,5 percent was achieved for spallation, an accuracy of 74,1 percent for material loss, and an accuracy of 54,2 percent for surface crack [8]. For the estimation of dispositions for damages not defined in the RSM, the developed ML model achieved an F1 score of 88.5 percent [7].

Problem Statement

For industrial applicability, however, integrating the developed ML models into existing processes and qualification of the resulting new processes is crucial. Beyond the numerical validation of the accuracy of ML solutions, it is equally important to build trust among engineers in the results provided by these models. Hybrid processes arise when engineers, supported by ML models, carry out tasks. This support can range from simple feedback through assistance to full automation. The higher the level of support, the more critical the engineers’ trust in the information provided or in the tasks automatically performed by the ML model becomes [9]. The qualification process aims to identify potential risks associated with hybrid processes and enhance confidence in the outcomes generated by the ML models. This raises the question of how ML models can be effectively integrated into business processes and what the qualification process for their implementation should entail.

State of the Art

Several ML processes have been established in academia and industry, outlining the necessary steps for implementing and deploying ML models. However, these frameworks differ in their level of detail. One of the most widely used methodologies is CRISP-DM [10]. Although initially developed for data management, CRISP-DM also applies to ML projects. Nevertheless, it has limitations regarding the selection and verification of ML algorithms. The industrialization steps are addressed at the deployment stage, but the specific details of the industrialization process remain vague, as only broad deployment steps are described. Other ML processes are usually related to CRISP-DM since ML is a data-driven analytics field; however, these frameworks differ in certain phases relevant to ML or supplement existing ones [11].

Ashmore et al. (2019) propose a life cycle focusing on ML models operating in safety-critical systems. As with any technical artifact, assurance can only be achieved by understanding the complex processes involved. Therefore, it is crucial to comprehend the ML life cycle to evaluate the ML components effectively. Their approach emphasizes the essential role of data in ML models, and accordingly, they highlight this in their description. They specify the desired properties of datasets and outline methods to achieve these properties [12]. Xiang et al. (2018) address the verification of neural networks, focusing on formal verification. However, their approach does not address assurance during the data management and model training phases [13]. Focusing on reinforcement learning, Garcia and Fernandez (2015) explored methods for reducing risk metrics, but their research covers only the model training phase [14]. Salay and Czarnecki review existing methods for ensuring the safety of supervised learning throughout the ML life cycle, but they do not connect these methods to industrialization processes [15]. Examining the ML life cycle proposed by Oracle, the software company places greater emphasis on business processes. Oracle‘s approach begins with a typical ML life cycle comprising business needs, data, model development, model training, model testing, model deployment, integration with business applications, and monitoring & optimization [16]. Subsequently, an operationalization perspective is introduced, suggesting that after an ML model is developed, a Minimum Viable Product (MVP) should simplify this step [17]. In both perspectives, the business need is strongly emphasized. Detailed attention is given to defining the use case and how it can be represented through an MVP, aiming to understand the minimal requirements and the gaps between the experimental phase and these requirements. The integration of new ML solutions with existing business applications is also examined.

In conclusion, all existing ML processes or life cycles provide initial guidance for implementing and operationalizing ML solutions. However, most lack a qualification aspect. Process qualification is a risk mitigation measure in industrial sectors, and its absence in ML-based processes limits trust in these solutions. This paper‘s contribution is the development of an ML process that incorporates qualification aspects, extending from data identification to ML training and, ultimately, to the monitoring of ML solutions.

Approach

As outlined in the previous chapter, several studies have introduced ML applications in industrial contexts. Alongside these developments, various standards and norms have been established to guide ML system development in general software engineering and ML-specific aspects. This research aims to create a practical guide grounded in scientific standards while reflecting industrial realities. To achieve this, existing norms were systematically reviewed. Based on ISO/IEC FDIS 5338:2023 (Information technology – Artificial intelligence – AI system life cycle processes), the AI technical life cycle structure served as the foundation for this analysis. Each phase of the life cycle was carefully examined, and relevant activities, milestones, and associated standards were outlined.

The key standards that informed the guide include:

1. ISO/IEC FDIS 5338:2023 – Provided the overarching framework for the AI life cycle.

2. ISO/IEC 27000 & ISO/IEC 27002 – Addressed information security management and IT security for AI systems.

3. ISO/IEC 25010 & ISO/IEC 25059 – Defined quality metrics for both software and AI systems.

4. ISO/IEC TR 24368 – Focused on ethical and societal concerns for AI systems.

5. EU Ethics Guidelines for Trustworthy AI & EU AI Act – Informed a regulatory framework for responsible AI.

Using these standards as a foundation, the guide was applied to develop two ML applications for the case studies described earlier with Siemens Energy. Based on ISO/IEC FDIS 5338:2023 (Information technology – Artificial intelligence – AI system life cycle processes), the AI technical life cycle structure served as the foundation for this analysis. This standard was significant for structuring the process phases and defining activities and milestones. ISO/IEC 25010 and ISO/IEC 25059 provided essential quality metrics guiding software and AI system evaluations. On the other hand, standards like ISO/IEC TR 24368 and IEEE 7000-2021 were considered secondary, offering valuable but less critical insights into ethical concerns and system architecture design. Documents such as the EU Ethics Guidelines for Trustworthy AI and the EU AI Act also influenced the development process, primarily for embedding Responsible AI principles. The development process involved collaboration between domain experts from the company and external ML developers. Technical steps and milestones were analyzed throughout the project, along with potential tools and methods used. A primary focus was placed on the sequential qualification of ML system components and their implementation in industrial environments. Additionally, particular attention was paid to integrating principles of Responsible AI.

A combination of deductive and inductive approaches was employed to build the guide. Initially, the guide was created deductively based on the standards and subsequently refined through observations of actual industrial implementation. The deductive approach helped establish a strong, standards-based structure, while the inductive approach provided practical insights and allowed for adjustments needed for real-world applications.

Qualifying an ML Application for MRO Assistance

To ensure that ML systems can be effectively utilized, they must undergo an internal qualification process similar to that used for other tools in industrial settings. This process verifies that the ML system meets the requirements, but the qualification specifics depend heavily on the application and context. In this case, the ML system was designed to support the MRO process as an assistance tool, with final decision-making in human operators’ hands. The qualification process followed a sequential approach aligned with the ML system’s development, incorporating four key quality gates: data and ML model, IT system, frontend, and business process. Unlike traditional software systems, ML models can be qualified based on performance metrics such as precision, recall, and F1 score. Typically, human baseline testing is used to establish thresholds. Still, in this case, human decisions (assumed to be correct) served as the benchmark due to the high risks associated with incorrect repair assessments. Given the novelty of ML development in many industrial contexts, experience-based thresholds were not yet established. Therefore, an initial qualification threshold of 80 percent was set, with iterative reviews involving users to reassess and potentially refine these thresholds.

A specific challenge in qualifying this machine learning (ML) system, which applied computer vision, was ensuring the reliability of the data used for model training. Inspectors labeled defects on turbine blades, and these labels were compared to a “Golden Dataset” to verify accuracy. In this context, the “Golden Dataset” refers to a set of images or data samples meticulously and consistently labeled by domain experts, serving as a reference standard against which the accuracy of newly labeled data is assessed. Inspectors were authorized to label production data only after completing a qualification test. To ensure ongoing labeling accuracy, periodic audits were conducted.

The resulting qualification process is structured as follows:

1. Data and Model Evaluation

   Data collection and labeling were carefully monitored, with an 80 percent threshold for performance metrics (precision, recall, and F1 score). Inspectors underwent specific training and testing to ensure labeling accuracy, with ongoing verification against Golden Data.

2. IT System

   The system infrastructure was verified to meet safety and performance requirements, ensuring real-time predictions and operational stability. Security measures were implemented, including penetration testing and rolebased access control.

3. Frontend (User Interface & Experience)

   Usability tests ensured that the interface allowed users to interact effectively with the system and provide feedback on ML predictions, which was then integrated into the retraining process.

4. Business Process

   The ML system was integrated into the existing MRO process, with clear role definitions for human operators. A continuous improvement process was established to monitor performance and gather stakeholder feedback.

Process for Development and Qualification of ML Applications

An interactive guide was developed based on the analyzed standards and the development and qualification processes undertaken to support domain experts and ML developers in building qualified ML systems. The entire process description is accessible as a PDF at [18]. The process is structured as a BPMN diagram, highlighting the key phases in ML system development (Figure 1). These phases include Understanding current processes and business goals, Identifying domain knowledge, Defining requirements and selecting metrics, Determining reference values and thresholds, Defining data management and Selecting/Generating data, Model selection and training, System development/deployment, Frontend development, Defining new business process and monitoring processes, final Reporting, Rolling-out, continuous Monitoring, and Improving.

Figure 1

Process chart for the development of qualified ML applications

Each phase is assigned specific responsibilities to the ML developer or the domain expert. The qualification steps (quality gates, QG) are conducted after the stages of “Model selection and training”, “System development and deployment”, “Frontend development”, and “Define new business process and monitoring processes”. Notably, continuous monitoring begins at the start of the usage phase. As outlined in the qualification process, ML systems must be designed to ensure their performance is consistently monitored, thereby facilitating ongoing qualification of their effectiveness.

The guide is developed as a clickable application, where each phase provides an overview of tasks, objectives, and potential methods for both roles. Furthermore, the guide emphasizes considerations for responsible AI. An example of this is shown in Figure 2, where the “Data management and data selection” phase is viewed from the perspective of the domain expert.

Figure 2

Exemplary view of tasks for the domain expert during “data management and data selection”

In this phase, the domain expert is responsible for developing the data generation qualification process and ensuring data quality relevant to the use case and other tasks. Specific objectives include ensuring that all labelers are qualified, labels are representative, and the ML developer fully comprehends and documents the data‘s meaning. The guide suggests methods such as the SemDaServ method [19] and highlights responsible ML considerations. For instance, it emphasizes that data sharing must not violate privacy laws or include sensitive personal data. Additionally, it calls for clear documentation of shared data and transparent communication with relevant stakeholders.

In the engineering case for damage detection, the “Data Management and Data Selection” phase began with establishing a data labeling process, as no pre-labeled images were initially available for the ML model. The process was structured as follows: Cameras installed around the turbine blade captured images under real lighting conditions and camera angles to simulate the actual environment. These images were then presented to engineers to define the labels. The labeling strategy was designed to ensure that quality metrics, including “quantity of occurrence”, “distribution across product types”, and “uniformity of labels”, were maintained. The uniqueness of the labeling process in this project lies in its approach to ensuring high-quality training data for the computer vision application. Initially, training sessions were conducted to familiarize labelers with the specific requirements and methods of the labeling process. Subsequently, the labelers were required to pass a test where their annotations were compared against a “Golden Dataset”, a reference created by experienced engineers. Only after passing this test were labelers authorized to annotate new images. To further ensure data quality, a continuous validation mechanism was conceptualized, wherein a randomly selected image from the Golden Dataset was presented for review after a certain number of labeling tasks. Subsequently, an ML model was trained, with a performance threshold set at a minimum of 80 percent. Whenever the model’s performance fell below this threshold, experts labeled additional images to improve model accuracy and robustness.

In addition to the development phases, the qualification steps for the final reporting of the AI application are presented in a generalized format, which must then be tailored to the specific use case. For example, after the data and model evaluation phase, quality gate 1 starts with the requirement: “Define assisted engineering activity, the system under investigation, system specifications, and any necessary restrictions, as well as the intended user. Ensure thorough documentation.” The specific use case is refined: “A recommendation system was developed to assess repair needs for turbine blades. The qualification applies to blade type 2SGT5-4000F and covers the identified defect types, such as surface crack, spallation, and material loss inspection. The intended users of the AI application are quality inspection engineers.”

Conclusion

This study introduced a structured approach for developing and qualifying ML systems within Maintenance, Repair, and Overhaul (MRO) processes. The proposed qualification methodology employs a sequential framework that addresses key quality gates: data and ML model validation, IT infrastructure, frontend design, and business processes. Establishing initial performance thresholds and implementing continuous monitoring ensures consistent evaluation and improvement of ML systems throughout their lifecycle. This systematic approach is critical in industrial settings, where precision and reliability are essential to minimize risks and optimize decision-making processes.

The scientific contribution of this research lies in integrating a comprehensive qualification framework tailored to ML systems within industrial applications, bridging the gap between existing ML development frameworks and the specific demands of industrialization and risk management. Unlike traditional approaches that often treat ML solutions as “black boxes”, the proposed methodology emphasizes transparency and trust by embedding iterative validation checkpoints throughout the lifecycle. This continuous qualification strategy differentiates itself by aligning technical development with business requirements while incorporating principles of Responsible AI derived from established standards such as ISO/IEC 5338:2023 and EU Ethics Guidelines for Trustworthy AI. Additionally, this work contributes novel methods for ensuring labeling accuracy through applying “Golden Datasets” and qualification testing for inspectors, offering a replicable model for enhancing data quality in safety-critical environments. The guide’s interactive design provides a unique tool that operationalizes theoretical insights into practical steps, making it accessible for ML developers and domain experts.

Although this study focused on supervised learning models, the proposed framework is adaptable to other domains and supervised learning applications. The structured approach includes core steps like understanding business goals, identifying domain knowledge, and defining new business processes, which ensure that the ML implementation aligns with specific use cases and organizational needs. This alignment improves the system’s effectiveness and enhances its integration within existing workflows, leading to smoother transitions from prototype to deployment in real-world scenarios.

Future research will expand this guide to other industrial sectors and explore emerging standards, particularly those addressing generative AI and explainability. Integrating these standards will be critical as AI technologies continue to evolve and diversify in application. Ongoing validation through real-world case studies and iterative refinement will be essential to align the guide with evolving norms and industrial needs, ensuring it remains relevant and applicable across different industries and technological landscapes.