Investigating the graphical IEC 61131-3 language impact on test case design and evaluation of mechatronic apprentices

2.1 Code coverage assessment in IEC 61131-3 control software

Code coverage assessment is a white-box test strategy to assess the test adequacy using the control code of the system under test [3], [10]. The parts of the control code activated during test execution are tracked to detect software parts not covered by the available test cases. The analysis of the code coverage of existing test cases is used in computer science to assess the adequacy of these test cases [16], as a basis to design new test cases for non-tested software parts [17] or to select test cases based on their coverage of changed software parts [6]. Statement and branch coverage are the most common code coverage criteria in test adequacy assessment. Whereas statement coverage focuses on executing “every statement in the program at least once” [16], the mightier branch coverage additionally focuses on executing all (decision) branches within the program [5].

The control code of aPS is mainly realized in one of the five programming languages of the international standard IEC 61131-3 [18]. IEC 61131-3 defines two text-based languages (Structured Text [ST] and Instruction List [IL]) and three graphical languages (Function Block Diagram [FBD], Sequential Function Chart [SFC], and Ladder Diagram [LD]). Some approaches exist to apply code coverage criteria from computer science to the IEC 61131-3 languages. Ulewicz et al. [6] estimate code coverage for ST and SFC in aPS by inserting trackers into the respective control code. Hao et al. [19] generated test cases for structured text using a control flow graph of the software, which allowed them to apply branch coverage criteria from computer science. Bohlender et al. [4] generated test cases for ST using symbolic execution. To assess code coverage for the graphical programming language FBD, the researchers either transformed the FBD control code to C code [20], Java code (for IEC 61499 FBD [21]), or to timed automata [5] to apply computer science metrics. Jee et al. [6] introduced d-path-coverage as a criterion for code coverage assessment for FBDs without transforming them. With the d-path criterion, every data flow path within the FBD should be tested at least once to obtain full code coverage. This approach is similar to applying branch coverage to control flow charts in computer science. As the code structure of SFC is similar to control flow graphs, branch coverage can be applied similarly. Each control flow path within the SFC must be executed at least once to obtain full code coverage. This paper uses the d-path criterion for FBD and the control flow path assessment for SFC for better comparability in the following.

2.2 Code coverage teaching and assistance in testing

Systematic testing approaches are introduced in research to train or assist engineers in quality assurance. For example, testers are assisted in test adequacy assessment during start-up and maintenance with a visualization of the code coverage achieved by executed test cases. Aniche et al. [17] observed in a study with 84 professional software developers that they strongly rely on the source code and code coverage to design new test cases and manually check their test adequacy. They suggested (visually) connecting test cases to the code statements they cover to assist testers in test case selection for changed code statements. In a study with 30 professional software developers, Lawrence et al. [22] highlighted the code covered by test cases in C# code. However, this measure did not result in more test cases or faults detected. The developers tended to misjudge their level of test adequacy, meaning they focused more on achieving a high code coverage instead of creating test cases able to detect distinct fault types. On the contrary, Berner et al. [23] showed with an 8-person team that code coverage visualization increases the system’s robustness and is beneficial, especially for novices, and when testers are aware of possible misjudging. Rahmani et al. [24] used control flow graphs (CFGs) to visually represent the code and highlight covered code segments, thus achieving increased productivity of their 32 study participants. For IEC control code, experts in the automation domain rated the visual representation of the code segments covered in the graphical language FBD (Tool FBDTestMeasurer) as useful in achieving systematic and adequate testing [7].

To support engineers in the long term, testing and code coverage should be taught early during their education [17]. According to Bandura [9], a person’s ability and willingness to use learned skills does not only correspond to their real competence but also their self-perceived competence to succeed in a specific task (self-efficacy or perceived competence) [10]. Ribeiro et al. [25] investigated the self-efficacy concept in software engineering and recommended including it in future teaching strategies. In computer science, testing is included in the university curricula as a “key skill”, and teaching various testing techniques is proposed. For example, teaching programming and testing jointly is suggested to teach students code validation using code coverage early on alongside programming [26]. Code coverage visualization and tool support assist students in reaching high code coverage levels for their test cases [27]. Early test education or even test-driven development is further suggested to improve the programming style of students [28], leading to a more efficient testing phase and less effort for testers.

The aPS domain lacks concrete teaching concepts for code coverage assessment or test case generation based on code coverage, as introduced in the state of the art. In this paper, factors that influence education, as well as the testing strategies of vocational school apprentices, shall be investigated. As professional software developers strongly rely on source code and code coverage to design new test cases [17], the impact of the programming language, which is used as a basis for code coverage education, shall be analyzed.

3.1 Hypotheses to investigate apprentices’ learning success

Apprentices’ learning success is measured on two levels of competence: the apprentices’ perceived and real competence. Perceived competence [10] can be decomposed into several factors such as “comprehension” (1), “self-assessment of performance” (2), and “explanatory ability”, meaning the self-confidence to explain a given subject (3). The three factors of perceived competence are evaluated based on the fulfillment of the following hypotheses.

As Vogel-Heuser et al. showed [8], mechanical engineering students prefer to model the behavioral perspective of a system. SFC clearly depicts the procedure of the program and provides its functional view. On the contrary, FBD is a data-flow language that shows the logic combination of single elements on the hardware level, like Boolean algebra. Assuming that apprentices also prefer the behavioral perspective, the first hypothesis H1 is as follows:

1. Mechatronic apprentices rate SFC to be more comprehensible than FBD.

Apprentices’ comprehension (1) is hereby assessed in three areas:

• 1. Basic understanding of test case design based on code coverage assessment (H1.1: Mechatronic apprentices are basically able to derive test cases based on SFC and FBD code)

  2. Ability to critically examine their test cases (H1.2: Mechatronic apprentices are able to critically reflect on the test cases they derived based on SFC and FBD code)

  3. Classification of the two languages regarding their informativeness (H1.3: Mechatronic apprentices rate SFC program code of equal functionality as more informative than FBD program code.)

The evaluation of the basic understanding (1.1) furthermore validates the suitability of the two graphical IEC 61131-3 programming languages for code-based test case design. Based on the results of the comparison of the apprentices’ self-assessment of hypotheses H1.1 and H1.2, it can be determined whether there is a difference in the apprentices’ comprehension (1) depending on the language SFC or FBD (cf. RQ1).

After learning and applying code-based test case design, apprentices are supposed to self-assess their performance (2) as part of their perceived competence. As apprentices are assumed to comprehend SFC better than FBD (H1), they are expected to self-assess that they perform better using SFC. Performance means here that they can derive more non-equivalent test cases and achieve higher code coverage using SFC than FBD in a given time, resulting in hypothesis H2:

1. Mechatronic apprentices self-perceive their performance to be better using SFC instead of FBD code.

The last factor for perceived competence is the apprentices’ “explanatory ability” (3), which requires comprehension of the subject to be explained, but also the confidence and ability to adjust the explanation depending on the addressee and to adapt the complexity of the linguistic level respectively [10]. Apprentices are expected to feel more confident in one of the two languages and thus favor SFC or FBD when explaining the subject to different addressees, here a fellow apprentice and an experienced technician. This leads to the following hypothesis H3:

1. Mechatronic apprentices have a language preference when explaining their test case design.

To evaluate the real competence of the apprentices in test case design and evaluation using SFC and FBD, the results of the task they are supposed to work on during the experiment are analyzed. Due to the better comprehension expected (cf. H1) and the strong mutually dependent relationship between perceived and real competence assumed in literature [10], apprentices are expected to perform better with SFC than with FBD despite their prior knowledge of FBD, leading to the following hypothesis H4:

1. Mechatronic apprentices derive more non-equivalent, adequate test cases in a limited time using SFC instead of FBD code.

3.2 Experiment implementation

The experimental focus was on a preliminary, qualitative study of the hypotheses regarding the IEC 61131-3 language impact to enable the derivation of a teaching strategy for a future empirical study. For this preliminary, qualitative study, obtaining a homogeneous sample of apprentices was crucial. To ensure this, the experiment was conducted as a one-day workshop with a typical class size of 20 apprentices from a mechatronic vocational school. All 20 apprentices were male, between 16 and 20 years old, and about half had an Abitur (German high school diploma). All apprentices were from the same class in the 2nd year of apprenticeship, so an equal level of mechatronic education and knowledge can be assumed. They already learned and practiced the graphical IEC 61131-3 programming language FBD at school in a 2-week programming course. According to the apprentices’ teachers, none of the apprentices had prior knowledge of programming in general, SFC programming, modeling, or testing outside of vocational school.

To avoid bias due to the order in which test case design is taught (starting either with SFC or FBD), the group is divided into two equally sized subgroups, “FBD subgroup” and “SFC subgroup” before the experiment (cf. Figure 2). Two of the apprentices’ teachers, who trained them in the last two years of apprenticeship, divided them into equal subgroups regarding prior knowledge (e.g. type of their dual training company and whether they use FBD there), performance (e.g. grade), degree of education (e.g. Abitur), and motivation (subjective teacher assessment based on usual classroom collaboration and interest in software-related topics) in this order. As these factors could, of course, bias the self-efficacy results at the end of the experiment. The FBD subgroup is taught testing and code coverage with FBD first and then with SFC (cf. upper part of Figure 2). Conversely, the SFC subgroup starts with SFC and then learns to test with FBD (cf. lower part of Figure 2). Immediately before learning testing and code coverage in the previously unknown language (here: SFC), both subgroups are introduced to that language and its programming. All teaching units comprised a 20-minute PowerPoint lecture followed by a 70-minute exercise. Each exercise consisted of two tasks: First, the apprentices received test cases and the task: “Determine the code coverage of the test cases given”, followed by the second task: “Create all test cases required to achieve 100 % code coverage”. The exercise results were solved on paper, collected, and then discussed. In the “learn SFC programming” unit, the 70-minute exercise was a practical programming exercise on small laboratory plants normally used for IEC 61131-3 programming education. The small sample size of a typical vocational school class allows individual counseling to ensure that everyone understands the topic before moving on to the next phase of the experiment. The table at the bottom of Figure 2 depicts the expected knowledge per subgroup after each experiment phase. Assuming both subgroups have the same prior knowledge in FBD, SFC and testing using both languages, after the separate experiment phases, the subgroups are reunited for an assessment on code coverage and test case design in FBD and SFC and a final questionnaire. The questionnaire addresses the hypotheses introduced in Section 3.1. The final evaluation of results (cf. Figure 1, center box) is based on the tasks and questionnaires completed during the separate experiment phases, as well as the final assessment and the final questionnaires.

Figure 2:

Execution of the experiment with knowledge differences between the two subgroups (FBD and SFC) after each experiment phase (cf. bottom table).

To answer hypotheses H1–H3, questionnaires are used because they are suitable methods for subjective assessments such as self-perceived competence. The apprentices’ self-assessed performance and comprehension (cf. Section 3.1) are assessed with paper questionnaires consisting of questions Q1–Q3 for both programming languages, SFC and FBD:

• Q1: How confident do you feel that your test cases are sufficient to test the SFC/FBD code given?

• Q2: How confident do you feel about creating at least one suitable test case using SFC/FBD?

• Q3: How confident do you feel about finding all test cases required for 100 % code coverage using SFC/FBD?

The questions are answered on a 5-point Likert scale with response options ranging from 1 – “not at all confident” to 5 – “extremely confident”. Question Q2 tests the apprentices’ basic understanding of whether they are able to create basic test cases using SFC and FBD. Question Q3 aims at apprentices’ self-confidence in their test case design skills to achieve complete coverage based on the respective IEC 61131-3 programming language. The results of the questionnaire (self-assessed performance and comprehension) are compared with the real performance of the apprentices, which is determined by the number of adequate, non-equivalent test cases they created in SFC and FBD, respectively.

Preferences for either SFC or FBD in terms of comprehension and self-confidence to explain the test case design are assessed with a paper questionnaire consisting of questions Q4–Q6. For each question, the apprentices had to choose one of the three answer options (Equal, SFC, FBD) and briefly reason their choice.

• Q4: Which language would you rate more informative for test case design and evaluation?

• Q5: Which language would you choose to explain code-based test case design to a fellow apprentice?

• Q6: Which language would you choose to explain code-based test case design to an experienced technician?

3.3 Mechatronic use case

As a mechatronic use case (cf. C2) for the apprentices’ code coverage training, a screw gripper is chosen, which is part of a pick-and-sorting unit for different screw types. The gripper has three fingers to grip screws (cf. Figure 3, left). Four different shapes of screw heads are to be picked: Triangle, hexagon, dodecagon, and circle (cf. Figure 3, center). For secure gripping, the screws must be gripped by their screw heads along the edges and not at the corners (cf. Figure 3, right). For the sake of simplicity, it is assumed that the screw gripper can only rotate 30° and must therefore rotate a certain number of 30° rotation steps depending on its initial position to be in the correct position to grip the respective screw. The screw gripper use case represents a typical mechatronic application that is easily understandable within the given time but is sufficiently complex that not all test cases are immediately obvious. The different screw types and possible initial rotation angles require the apprentices to consider multiple conditions and testing scenarios to achieve thorough testing and to develop comprehensive test cases.

Figure 3:

Screw gripper example – (a) side view (left), (b) top view (middle), (c) rotation required for gripping (right).

The number of rotations required is determined by an IEC 61131-3 control code part, realized in FBD and SFC (cf. Figure 4). The screw types (triangle, hexagon, dodecagon, and circle) are defined with the numbers 0 to 3. Dodecagon and circle do not require the gripper to rotate as they can be gripped from any angle. The hexagon-shaped screw needs the gripper to be rotated at most once, whereas the triangle-shaped screw requires at most three rotations based on the gripper’s initial rotation angle. While it is intuitive to test each screw type, considering all possible initial rotation angles is not immediately evident while creating possible test cases solely based on the text description.

Figure 4:

Excerpt of control code for gripper rotation in FBD (left) and SFC (right).

The code excerpts (cf. Figure 4) were designed so that the program paths and corresponding test cases are equally difficult to detect from a testing perspective. SFC appears to be more intuitive at first glance, owing to its structured and sequential nature. On the contrary, possible input parameters and paths based on the initial angle (0, 30, 60, 90) are more clearly visible in FBD, attributed to the use of the modulo operator. Additionally, the apprentices lack familiarity with the SFC language whereas they are acquainted with FBD and can thus handle more complex examples in FBD.

5.1 Hypotheses assessment

1. Mechatronic apprentices rate SFC to be more comprehensible than FBD.

   1. Basic Understanding

Apprentices were, in general, able to derive test cases based on SFC and FBD code (H1.1, cf. Figure 5 left) and rated their confidence in finding at least one test case in both languages at least as medium (cf. Figure 6, left), thus showing their basic understanding as well as the general suitability of the two languages for code-based test case design. The comparison of apprentices’ average perception regarding SFC and FBD showed, in general, a slight trend towards SFC, as apprentices were, on average, “confident” to find at least one test case. In contrast, the perceived competence in FBD was only “medium”. Considering the previous subgroups separately, the FBD subgroup was less confident in finding at least one test case using SFC than the SFC subgroup. The missing confidence could be explained through the less routine and exercise they had with code coverage in SFC. On the other hand, no positive effect of more exercise could be observed for FBD. Both subgroups rated their confidence to find at least one test case within FBD as similarly poor, even though the apprentices of the FBD subgroup were expected to be more familiar with FBD code coverage assessment than the SFC subgroup. The FBD subgroup did not show a preference for FBD in the final questionnaire even though they first learned code coverage with FBD and thus exercised it more than the SFC subgroup.

• 1. Critical Evaluation

Apprentices were able to evaluate their test cases (H1.2, cf. Figure 6 right). In the early experiment phases, apprentices were already slightly more confident in the sufficiency of their test cases with SFC than with FBD (cf. Figure 5 right). Regarding apprentices’ confidence in finding all test cases required for 100 % code coverage using SFC/FBD (cf. Figure 6, right), apprentices were, on average, more confident with SFC than with FBD. The results show that apprentices were able to reflect their test cases critically in both languages (H1.2). However, a preference towards SFC is visible in the results (cf. Figure 6 right).

• 1. Classification of the two languages regarding their informativeness

As expected in H1.3, apprentices of both subgroups rated SFC as more informative than FBD (cf. Figure 7, left). Apprentices named the clearer code structure as the main reason for preferring SFC.

Since all three comprehension areas (1.1–1.3) were confirmed and showed a preference towards SFC instead of FBD control code for test case design and evaluation, hypothesis H1 is considered confirmed.

1. Mechatronic apprentices self-perceive their performance to be better using SFC instead of FBD code.

As assumed in hypothesis H2, the SFC subgroup using SFC self-perceived their performance (cf. Figure 5 right) better than the FBD subgroup using FBD. Despite being more experienced in FBD programming, the FBD subgroup was, on average, “not confident”, whereas the SFC subgroup was, on average, “neutral”. In conclusion, a preference for SFC in test case design was already visible, comparing the subgroups’ subjective ratings when they were still separated.

1. Mechatronic apprentices have a language preference when explaining their test case design.

The final questionnaire examined whether apprentices have a language preference to explain their test case design to (a) a fellow apprentice or (b) an experienced technician. The results of both questions show a clear preference for SFC, thus confirming hypothesis H3 that there is a preference and further highlighting the perceived competence as better using SFC.

In conclusion, the evaluation of the perceived competence shows that apprentices were more confident using SFC than FBD for their code-based test case design, thus deriving a subjective preference for SFC as finding F1. Next, the objective preference regarding the apprentices’ real competence is discussed.

1. Mechatronic apprentices derive more non-equivalent, adequate test cases in a limited time using SFC instead of FBD code.

Regarding their real competence (cf. Figure 5), the apprentices were able to derive more non-equivalent, adequate test cases using SFC than FBD (finding F2), thus confirming hypothesis H4 . Even though only a slight trend is visible, it matches the perceived competence of the apprentices (finding F3). Due to this match in finding F3, the research questions RQ1 and RQ2 are considered answerable. The results indicate a difference in test case design and evaluation depending on the graphical IEC 61131-3 programming language (SFC or FBD) used by the apprentices (RQ1). Due to the apprentices’ preferences and statements in favor of SFC and the better performance results, SFC is recommended for teaching testing to apprentices (RQ2).

Despite not having prior knowledge of SFC programming, the 1.5-h block on SFC introduction and programming was sufficient for the apprentices to understand the control code and to apply code-based test case design and evaluation. A new programming language does not hinder testers, such as maintenance personnel, from learning and applying new testing techniques. According to the apprentices’ reasons for preferring SFC, a clear code structure is crucial to assess the code coverage and to derive new test cases faster. The results (cf. Figures 6 and 7) also confirm that a clear code structure facilitates finding more or even all test cases. Due to this feedback, a stronger focus on code structure while programming is promising to improve the subsequent testing process. Thus, it is recommended to teach testing and code coverage while teaching programming so that developers become aware of the impact of their programming style on the later testing phase and personnel responsible for testing. Concluding, the experiment results show a trend toward SFC in apprentices’ perceived and real competence. Further, apprentices are more likely to test intuitively rather than systematically when applying code coverage.

5.2 Threat to validity

This section discusses the potential threat to the validity of the study. The paper proposes a method to assess language impact on test case design and evaluation, using perceived and real competence to derive the experiment design. The structure, initially independent of the application, (cf. Figure 1) enhances transferability to similar studies. However, generalizability is potentially compromised as the questionnaires used to measure the self-efficacy are task- and domain-specific, following Bandura’s [9] recommendation. Additionally, follow-up studies with larger sample sizes would improve the generalizability of the results obtained. Considering the construct validity, meaning the validity of the measures taken to answer the research questions, the tasks within each group phase were identical in both content and structure, except for IEC language, to enhance comparability. The uniformity of the example for both use cases introduces a potential bias, as different languages in IEC are more suitable for specific use cases (e.g. SFC for procedural programs and FBD for logic combination [8]). The code excerpts of the use case were chosen to be equally difficult from a testing perspective (cf. Section 3.3). Nevertheless, as the use case itself reflects a sequential decision process, a potential residual bias remains, due to which a follow-up investigation of the impact of the IEC 61131-3 language based on the code construct and problem scenario (e.g. static and/or sequential process) at a granular level is recommended. For results reliability, meaning the researcher’s impact on the data and the analysis, the final questionnaires and task results of all apprentices were blindly evaluated without knowing their respective groups. The link of an apprentices’ evaluation result to their group was recovered afterward using unique apprentice identifiers. The division of apprentices into two subgroups was conducted not by the researcher but by the apprentices’ teachers. This division was based on three objective criteria such as degree of education, along with the teachers’ subjective assessment of the apprentices’ motivation as fourth criterion (cf. Section 3.2). The subjective assessment served only as the final tiebreaker in group allocation in case of uniformity in previous categories. However, there remains the possibility of misjudgment by the teachers, potentially resulting in an uneven distribution of groups. Additionally, apprentices’ condition on that day may also impact the comparability of the groups. These threats are hardly evitable and have a greater impact on small sample sizes than on large sample sizes. In a future empirical study with a large sample size, the impact of such effects could be investigated. Regarding the accuracy of the conclusions (internal validity), it should be noted that this paper presents the results of a preliminary study, deliberately only reflecting qualitative tendencies, which will be used to derive a teaching strategy on testing to be investigated regarding its statistical significance in empirical follow-up studies.