Model-driven latency analysis of distributed skills in automation networks with robots in the AI.Factory

2.1 Domain-specific definition of skills

The definition of a robotic Skill from [7] is shown in Figure 1, and is visualized and adapted so that the two definitions can be directly compared. The Skill consists of a SkillInterface for the Preconditions and Parameters. The Preconditions are input from the CurrentState (State of the system) and the Parameters from the overlying Tasks. In the center of the robot Skill is the Execution, which is parametric. That means the same Execution occurs every time the Skill is executed, and only some parameters can be adjusted, affecting characteristics, but not the behavior of the Skill. A simple example would be the drilling of a hole. The Execution of the Skill drilling would stay the same while, e.g., the depth can be adjusted by an input Parameter, resulting in the same Execution. A unique feature of the robotic Skills is constantly monitoring the Skill’s Execution. The Execution is constantly evaluated (ContinousEvaluation) and checked to determine whether Pre and Postconditions and the Prediction are fulfilled. The Skill’s Execution results in a StateChange. In the robotic domain, a Skill is composed of DevicePrimitives, and Tasks are composed of Skills.

Similar to the Skill definition from the robotics domain, a Skill in the automation domain consists of two different inputs, a StateMachine, and Parameters, which are, in this case, exposed by a defined interface (cp. Figure 2). At first glance, differences like the lack of post- and precondition checks and continuous evaluation can be seen. These points are not as explicitly defined for industrial automation Skills, as they are for the robotics domain. One of the biggest differences compared to the robotics domain, however, is the integration of Skills into the surrounding architectures. While in the robotics domain, the Skill merely leads to a not further specified StateChange, the industrial automation Skill focuses directly on the Process. In this case, the Skill, which is provided by a specific resource, controls a technical process. There are further definitions of industrial automation, such as that of Köcher et al. [8], providing a definition of Capability, Skills, and Resources that is linked to a number of industry standards. Likewise, comparisons have already been made by da Silva et al. [9] based on this concept with the Skill descriptions of heterogeneous robot systems. However, to achieve modeling and analysis of distributed heterogeneous control networks with collaborating robots, a uniform way to define both robotic and industrial automation Skills needs to be defined, containing information to create a matching between software and controller platform.

2.2 Hybrid use case description

A use case is designed to demonstrate the communication and information exchange between systems and serve as the application example for the modeling and analysis approach to be developed and tested. The use case was chosen because of its hybrid nature of combining a six-axis robot with a typical example of a production plant. It comprises two scenarios: normal behavior and fault recovery. The system operates as expected in normal behavior scenarios, with no interruptions or errors. If a fault occurs in the error scenarios, a central agent intervenes by rescheduling tasks and reallocating responsibilities. This ensures the continuity of the process and highlights the system’s resilience and adaptability. The scenarios include a typical industrial automation use case combined with a six-axis robotic arm, focusing on material handling operations.

The use case is a yogurt production plant demonstrator that consists of a process and an intralogistics part. The investigation focuses on the intralogistics of the plant, which can be seen in Figure 3 (Conveyors-C1 to C4 and Switches-S1 to S3). The resources included are four conveyors: a light gate at the beginning and one at the end of each conveyor, and three switches that can move 360° to bring yogurt bottles to a specific target. The resources are hierarchically ordered in three different Conveyor Groups (CG). CG1 consists of conveyor C1 and switch S1, CG10 consists of conveyors C2 and C3 and switch S2, and CG9 consists of switch S3 and conveyor C4. The skills involved in this use case can be found in Table 1, and are further described in Section 4. An empty bottle will enter the system at the beginning of C1 and is detected by the first light barrier. The bottle is then transported to the Switch S1, where the skill DirectionControl_CG1 decides in which direction the bottle should be delivered. After the switch is turned in this specific direction, it continues to move through the transportation system in the same way until it reaches the output slot at the end of C4. This is the normal operation. The fault recovery case within the hybrid system is a broken switch in S3. The bottle would either get stuck or be redirected. This use case additionally involves a robot, handling yogurt bottles from the yogurt plant’s storage and placing them into a logistics system, cp. Figure 3. Three agents are involved in handling the process: Central.AI, the MyJoghurt plant, which consists of the storage and logistics system, and a Franka Emika robot. The scenarios (normal behavior and fault recovery) are described in the sequence diagram in Figure 4. The first ten messages in this scenario are the same for the normal behavior, and the fault recovery scenario. The Central.AI provides the recipe to the MyJoghurt plant and the coordinates necessary for the Franka robot to know the insertion and clearance point as the interaction point. Afterward, the MyJoghurt plant and the Franka robot communicate directly, resulting in the Franka providing a bottle at the entrance.

Figure 3:

Yogurt production with Franka and logistics system of MyJoghurt.

Figure 4:

Sequence diagram of the hybrid use case, including normal behavior and fault recovery scenario.

In the normal scenario (Figure 4, alt: [Scenario: normal]), the Franka robot retrieves empty yogurt bottles from the MyJoghurt bottle storage area. It places them onto the logistics system’s input conveyor (C1). The bottles are transported along a predefined conveyor path, C1 → S1 → C2 → S2 → C3 → S3 → C4, which is controlled by switches at various points. As the process technology part belongs to the not considered part of the plant, the filling of yogurt into the bottles is assumed to take place between S1 and S2. Once the bottles reach the output conveyor (C4), the robot transfers the filled bottles to the finished product storage area for completed products. Central.AI, which operates on a separate computer, manages the data from the other two agents, including layout and real-time sensor information.

In the fault recovery scenario (cp. Figure 4, alt: [Scenario: mitigation of failed switch]), switch S3 is defective and unable to transport the yogurt bottles to C4. The Central.AI is informed of the fault by MyJoghurt and orchestrates a mitigation plan, instructing the robot to retrieve the bottles from C3 and place them in the finished product storage area, ensuring that the production process continues without interruption. The unique feature here is that the communication between the MyJoghurt system and the Franka robot can continue as usual. The Central.AI sends the Franka robot new fixed coordinates for the pick-up point of the bottle, which is now at the light barrier at the end of C3. Central.AI uses the information from MyJoghurt, i.e. the bottle position at the end of C3, to calculate a six-dimensional pose for the pick-up point of the Franka robot.

3.1 Modeling of timing behavior in distributed CPS and robots

An example of modeling and analyzing real-time systems is the MARTE [10] standard developed by the Object Management Group (OMG). MARTE extends the Unified Modeling Language (UML) to provide broad-spectrum coverage of real-time system modeling. In the automotive domain, Mubeen et al. [11] have introduced an approach to evaluate end-to-end timing behavior in control networks. By developing plug-ins for the AUTOSAR standard, they enable early-stage design analysis. Nevertheless, due to the high degree of standardization and cloud connectivity in automotive systems, this approach has limitations that hinder its direct applicability to industrial automation. A timing-aware, model-driven development approach for automotive software is presented by Bucaioni et al. [12]. This method integrates design and implementation by establishing a detailed relationship between EAST-ADL [13] and a commercial development tool. It supports early-phase timing analysis through the automatic allocation of software to hardware and facilitates the phases of design, deployment, and verification. Expanding on this, Marinescu and Enoiu [14] extend the EAST-ADL approach to include the modeling of functions, execution timing, and resource usage. In another effort, Eder et al. [15] explore automated hardware design solutions by deploying software tasks through a DSL. Their methodology incorporates three key perspectives: modeling, language design, and exploration automation. These viewpoints address the heterogeneous technologies inherent in modern automation systems, highlighting the need for a more cohesive framework to bridge design and implementation in industrial contexts. These works collectively highlight advancements in timing-aware modeling and analysis across various domains. However, the transition of these methodologies from automotive to industrial automation systems remains an ongoing challenge, primarily due to differences in system architectures and domain-specific requirements.

3.2 Modeling of skills in industrial automation and robotics

The definition of skills is a well-researched field and was defined especially for the robotic domain and for different use cases. A brief overlook of the skill definition was given in Section 2. Two of the approaches on which this approach is based are the CSS model definition from [6] and the definition of robotic skills in manufacturing in [7]. Köcher et al. [8] link the definition of capability and skills to a number of industry standards. Likewise, comparisons have already been made by da Silva et al. [9], based on this concept with the skill descriptions of heterogeneous robotic systems. A formalized description of skills is presented in Lesire et al. [16] to derive operational models. The focus of this work is specifically on robotic skills. In the field of Cyber Physical Production-Systems (CPPS) a skill definition is given by Gonnermann et al. [17]. The approach focuses on the sensorial skills to enable the automated setup of process monitoring. A multitude of works exist that address skills. However, they lack a software-oriented focus on skills in heterogeneous systems that enable a timing analysis for controller and communication load.

3.3 Latency analysis in distributed controlled networks of robot systems and CPS

Latency analysis for software can be conducted on differing granularity levels, depending on the level of detail required. Coarse-grained approaches measure execution times at the function level, while fine-grained methods can capture the execution time of individual code statements [18], [19]. During runtime, time monitoring can be categorized into non-intrusive and intrusive methods [19]. Non-intrusive methods primarily focus on network-related parameters such as congestion, while intrusive methods involve embedding measurement instructions directly into the source code, enabling real-time monitoring of software states [19]. Tools like Code-TEST [20] provide time-monitoring functionalities in addition to the internal timing mechanisms of devices. However, such tools are often vendor- or operating-system-specific, which restricts their portability [21]. In the domain of machine learning (ML) and edge computing benchmarks, existing approaches like Tiny Benchmark [22], AIBench [23], and AIoT Bench [24] predominantly rely on operating system-based timing functions. For Programmable Logic Controllers (PLCs), Iqbal et al. [25] propose a benchmarking methodology aligned with the IEC 61131-3 standard. This approach leverages vendor-specific system counters to measure program execution times, capturing metrics, such as last, minimum, and maximum PLC scan times. Some approaches estimate Worst-Case Execution Times (WCETs) using probability distributions derived from collected measurements [26]. An approach of a vendor-independent and generalizable benchmark among different devices was developed by Rupprecht et al. [27]. The measurement of the latency between agents in the context of automation and robotic cooperation was carried out by Vogel-Heuser et al. in [28] and [29]. In summary, most benchmarks employ specific time measurement methods without thoroughly examining their characteristics or ensuring their portability across different devices. Furthermore, the details of the measurement techniques are often not explicitly documented, and they typically exclude PLCs, limiting compatibility across benchmarks.

4.1 Consolidated skill concept

A consolidated representation of skills is to be developed to enable the efficient modeling of heterogeneous systems and to standardize the representation of code snippets for uniform analysis. To achieve this, two code examples – one from the field of robotics and another from industrial automation – will first be analyzed and compared with the two previously outlined skill models. Two types of skills are considered in this context to ensure generalizability. Since, in most cases, skills are not merely arranged in a purely sequential manner but can also be represented hierarchically, both a basic skill and a composite skill are examined.

A basic skill from the industrial automation use case is a motor driver for one of the actuators in the MyJoghurt system. This skill represents the most elementary level of operation and directly controls the motor outputs. In the structure of the function block (cp. Figure 5), the states serving as the interface for the skill are evident. These states are INIT, RUN_pos, RUN_vel, and STOP. The commonly encountered functions INIT and STOP align with the definition of the OMAC state machine [30], as specified in PackML, which is widely utilized in industrial automation. However, it becomes apparent that the state machine is not consistently applied across all levels, as the states RUN_pos and RUN_vel are customized states. This discrepancy does not play a significant role in the abstract level of skill representation, as it is clear that these non-standardized code snippets can still be associated with the interface state machine. The interface variables and the code modules within the states RUN_pos and RUN_vel, control the motor either based on target position or target velocity. Parameters are passed to these states in compliance with the definition of the parameter interface. A more granular analysis reveals that additional state machines exist within the skill, which process individual operations step-by-step. These state machines implement a form of pre- and postcondition verification through their transition conditions. This description applies to both robotic skills and industrial automation skills. However, these internal state machines do not impact the modeling of latency times within the system. For this reason, this part is represented as the execution layer and is not further considered regarding pre-/post-conditions or additional skill evaluation. If it becomes necessary to distribute the continuous evaluation of the skill to other skills, this can be easily exposed through the interface and optionally modeled as a validation interface. In this case, a prediction state is not present. In the context of industrial automation, such a state could ideally be invoked externally and, in the sense of a digital twin, facilitate the comparison between the predicted and the actual execution of the skill.

Figure 5:

Control code of Driver_JVL skill from the use case.

A hierarchically higher-level skill, such as controlling an entire conveyor group in industrial automation, serves as an example and is stated in Figure 6. The interfaces also contain a state machine (cp. Figure 6, CS_Interface: Mode), but the primary distinction from the basic skill lies in the parameters. The I/O connections for sensor and actuator integration are defined at this level. The state machine perspective is primarily used as a parameter-passing mechanism rather than a fully implemented interface. Therefore, the definition that a state machine must exist at this level is only partially accurate. Instead, it is more appropriate to say that the individual skills operate within the execution layer, while the higher-level skill only indirectly features a state machine. The parameters passed to this skill, in addition to the current state, include initialization conditions for all units of the transport system, the bottle number being requested, and three arrays for each bottle present. These arrays specify the target for the bottle, the position of the bottle, and the desired route for the bottle. Specific variables are defined for communication with the robotic system. For instance, Group1_ready indicates whether the MyJoghurt system is ready to receive a bottle, and G10_request signals whether a bottle is ready for pickup by the robot. If the bottle is removed from the robot, the clearance can be communicated by the variable bottleRemoved. The remaining parameters are utilized for a unique state where the system is initially cleared of bottles. In this case, these parameters have no further impact on the operating state. It can be stated that the hierarchically higher-level skill, although composed of basic skills that are state machines and thus inherently possess pre- and postconditions, does not explicitly reflect these conditions at the higher skill level. For this reason, it cannot be assumed that higher-level skills necessarily implement pre- and postcondition checks or adhere to a state machine interface. Nevertheless, when the skill is displayed in an expanded representation, it can serve as a container with defined inputs. If it is presented in a collapsed form, the state machine and parameters should be inherited in a specific manner.

Figure 6:

Control code of the Conveyor_Group skill from the industrial automation example.

In contrast to the IEC 61131-3 conform implementations of the automation skills, the skills of the robotic arm are written in C++ and are differently structured. Figure 7 shows an example of a basic skill for the Franka robot. The skill depicted here is used to move the robot within a Cartesian coordinate system and is equivalent to a function. Input parameters include the variables targetCoordinates, provided as a vector, and movementTime. Unlike explicit state machine control, the interface is not directly available but is invoked by the higher-level code within a specific state. The concept of continuous monitoring during the execution of a skill is generally applicable to the Franka robot. However, it is implemented in our example only at the lower-level skills, which control the drivers of the Franka robot. Therefore, the representation of continuous monitoring can be achieved by inheriting the properties of the lower-level skills into the higher-level ones or by modeling this feature as optional in the system design. However, a check for a postcondition is implemented here, where the timer variable – and thus the execution time of the skill – is monitored. Upon completing the skill, the state change is communicated back, ensuring proper synchronization. Due to copyright reasons, displaying the individual driver libraries here is impossible, as these are not custom-written drivers in contrast to our industrial automation scenario. However, the structure differs slightly from that of PLC programming. In this context, an output stream is defined for communication with the lower-level robot controller, whereas in PLC programming, variables are directly mapped to the controller outputs. Nevertheless, the basic skills can be assigned here so that they are clearly defined when a skill actuates an actuator or reads a sensor. This abstraction layer can thus be effectively utilized for system modeling.

Figure 7:

Base skill MoveCartesian of the Franka robot.

Figure 8 proposes a consolidated version of the skill descriptions combining both approaches from [6], [7]. The illustration is based on the CSS definition from [6], with all deviations from this standard highlighted in blue. The CSS framework was utilized to maintain the relationship between Resource, Capability, and Technical Process, as these elements are of significant importance in the combined consideration of industrial automation and robotics. The goal of this consolidation is not to redefine skills for implementation within generalized programming frameworks or concepts, but rather to abstract the definition in such a way that, irrespective of the actual implementation – such as those depicted in Figures 5–7 – the classification of control code within the analysis concept can be achieved.

Figure 8:

Class diagram of the consolidated skill description.

From this objective, the first modification of the model emerges: the introduction of an interface linking the skill definition to the analysis. This interface enables the inclusion of skill parameters that can be utilized not only during the operation phase but also for analysis, automated deployment, or other applications. The Analysis is enabled by some of the elements of the skills, including the Parameter that are inputted from a Property, the newly introduced class Metrics, and the also newly introduced I/O-Connections. The I/O-Connection class was introduced to address the unclear interaction (controls) between skills and processes. While the I/Os are generally determined by their assignment to a resource, they are not yet explicitly defined enough to enable a distributed analysis. For instance, if a skill is located on a controller that is not directly connected to the I/O variable, additional latency arises due to the communication required with the corresponding resource. The Metric class was introduced to analyze the code embedded within a skill, focusing on properties such as execution time, memory usage, and other characteristics. Metrics can be defined and extended as needed for the specific application under analysis. Additionally, any number of metrics can be associated with a single execution, allowing for a flexible and comprehensive evaluation. The Execution concept was adopted from the robotic skill domain, recognizing that, particularly in robotics, not every skill is associated with a StateMachine. For this reason, the StateMachine is implemented as an optional component. While optional, the State Machine can still serve as an interface and be used to support the continuous evaluation of robotic skills – an aspect not explicitly defined in CSS. It can also be employed for evaluating specific conditions, such as preconditions, postconditions, and predictions, thereby enhancing the flexibility and applicability of the framework. For this purpose, the preconditions, postconditions, and predictions defined in the robotic skill are transformed into StateChangeConditions. This ensures that the evaluation of skill execution remains consistently described and seamlessly integrated into the framework. The final modification lies in the composition of skills. As demonstrated in the example of IEC 61131-3 code, an overarching skill can be composed of multiple skills. Given that in PLC programming, the definitions of tasks and primitives have slightly different meanings, it was introduced that a skill can be defined as a composition of multiple skills. This adaptation ensures compatibility and reflects the hierarchical structure commonly observed in industrial automation scenarios.

Based on the definition presented in Figure 8, Table 1 identifies and categorizes the existing skills for the use case. Column 1 also includes the number of skills of each type, as a skill can appear multiple times – as instances without any code modifications. This is particularly evident in the example of a driver skill. The skill Driver_JVL (cp. Table 1 ) always performs the same task: controlling the motor based on the inputs from the skills CE_Conveyor or CE_Switch. The only differences are the inputs, outputs, and interface parameters. The table includes skill names, descriptions, interfaces, and I/Os. The state change conditions are not explicitly listed, as they do not play a significant role in the analysis of timing behavior in this specific case. A small example can, however, be found within the Franka basic skill MoveCartesian, where a continuous time query is performed.

4.2 Integration of the consolidated skill concept

Based on the presented consolidated skill concept, an integration of the existing approach by Hujo et al. [5] is to be carried out. Figure 9 provides an overview of the key symbols used to represent software properties, particularly with regard to timing behavior in PLCs and other cyclically operating controllers. The DSL consists of skills, controller boxes, a communication annotation element, and three different types of connectors that can interlink the skills. The idea here is to represent the distribution of skills across different controllers and their communication with one another, whether within a single controller or between two controllers, which introduces additional latency. Figure 10 shows the revised skill symbol, incorporating all elements listed in Figure 8 (derived from the skill description given in Table 1), which can be used for modeling. It is important to note that some fields, as described in the abstract definition of the skill, are optional. An example of this is the ChangeCondition, which can be assigned to a state. An excerpt of the skill implementation for the use case, based on the symbols, is presented in the following section, explaining how it is used.

Figure 9:

List of elements for the DSL from [5].

Figure 10:

Skill element based on the proposed concept.