Enhancing DevOps Engineering Education Through System-Based Learning Approach

2.1 DevOps Software Development Process

DevOps engineering has emerged as a critical approach to software development that emphasizes collaboration, automation, and continuous improvement of new software versions while guaranteeing their correctness and reliability (Bass, Weber, & Zhu, 2015; Leite et al., 2019; Zarour, Alhammad, Alenezi, & Alsarayrah, 2020). It involves the integration of development and operations teams to automate and streamline the software development process while ensuring high-quality software delivery (Hornbeek, 2019). As such, it has become an essential skill for software engineers seeking to succeed in today’s fast-paced technology industry (Gurcan & Cagiltay, 2019). Nowadays, the field of DevOps engineering has rapidly evolved, transforming the landscape of software development and deployment by integrating the principles of CI, continuous delivery, and automation (Buttar et al., 2023). As a result, there is an increasing demand for skilled professionals who can effectively navigate and apply these principles in real-world contexts (Amaro, Pereira, & da Silva, 2022) and understand the culture and mindset of DevOps (Jha et al., 2023). In academia, unfortunately, academics are not motivated to learn or adopt DevOps, and no strong evidence exists of academics teaching DevOps (Pang, Hindle, & Barbosa, 2020). Few trials are reported in the literature that discuss the experience of offering a dedicated DevOps course at different universities (see, e.g., Demchenko et al., 2019; Hobeck et al., 2021; Jennings & Gannod, 2019; Paez & Fontela, 2023; Radenković, Popović, & Mitrović, 2022). Hence, to meet the increasing demand for DevOps professionals and to enrich the literature with more reports on offering DevOps courses, we have introduced DevOps engineering as an elective course in our software engineering program.

The development and operations teams create, deploy, and manage both the infrastructure (environments) and deployment (CI/CD) pipelines in a way that drives the DevOps culture, values, and practices (Díaz et al., 2024). A typical DevOps CI/CD pipeline consists of four main phases, as depicted in Figure 1. The source code repository is where the code for the application is stored and managed using a version control management system. It can be a public or private repository, such as GitHub or BitBucket. The CI stage takes the code from the repository and builds it into a binary artifact. This stage focuses on early and frequent integration of code changes. The resulted artifact is typically a Docker image or a Kubernetes pod. The continuous delivery/continuous deployment (CD) stage then tests the binary artifact and deploys it to a staging environment. The continuous delivery (CD) and continuous deployment (CD) are both DevOps practices aimed at streamlining the software development and release process. Continuous Delivery: Ensures software is always release-ready. Continuous Deployment: Automates the entire release process, including deployment to production. This allows the developers to test the changes in a production-like environment before deploying them to production. The continuous deployment stage then automatically deploys the binary artifact to the production environment. Although continuous delivery and continuous deployment are closely related concepts in DevOps, they differ in their final step of delivering changes to production.

Figure 1

Typical DevOps CI/CD pipeline.

The CI/CD pipeline can be implemented using a variety of tools and technologies (Alnamlah, Alshathry, Alkassim, & Jamail, 2021; Kamath, Vignesh, & Darshan, 2023). Some popular CI/CD tools include Jenkins, Travis CI, and CircleCI. Some popular CD tools include Docker and Kubernetes. The CI/CD pipeline is a critical part of the DevOps workflow. It helps to ensure that code is always up-to-date and that changes are deployed to production in a safe and controlled manner. The CI/CD pipeline can be customized to fit the specific needs of the application. For example, the pipeline can be configured to run different tests at different stages. The pipeline can also be configured to deploy the application to different environments, such as staging, production, and development. Figure 2 illustrates how the CI/CD pipeline intersects with the development and operation processes.

Figure 2

DevOps CI/CD pipeline intersection with development and operation processes.

Hence, adopting a successful DevOps process in software development needs more effort from both academicians and professionals in the industry to better understand and implement DevOps practices and uncover neglected or untouched areas yet, for instance, controlling cross-functional teams in the dynamic and iterative nature of the control in DevOps process (Wiedemann, Wiesche, Gewald, & Krcmar, 2023).

2.2 SBL

SBL involves the integration of theory, practice, and reflection in a problem-based learning environment (Harden et al., 1997; Matinho et al., 2022; Syeed, Shihavuddin, Uddin, Hasan, & Khan, 2022). SBL is not uncommon in graduate education; it is being used in medicine to understand the interconnectedness of bodily systems, which is crucial for doctors and nurses. For example: Zhong, Huang, and Lin (2023) found that the application of the OBL (organ system-based learning) practice teaching model in the digestive system department is conducive to improving the theoretical knowledge of nursing students and improving the core competence and clinical teaching satisfaction. Similarly, Lu, Li, Cao, and Min (2022) found that the OBL model in undergraduate clinical practice teaching of anesthesiology has significantly improved the education quality, theoretical achievement, and comprehensive ability of interns. SBL emphasizes the use of real-world scenarios and hands-on experiences to help students develop the knowledge and skills required to solve complex problems (Namasivayam, Fouladi, Tien, & Moganakrishnan, 2019). By simulating a real-world environment, students can practice problem-solving skills and apply theoretical concepts to practical scenarios. Garrubba, Donkers, Daniel, and Ennulat (2015) found that both system-based and problem-based are both effective teaching methods for teaching physical diagnosis curriculum. Atta and AlQahtani (2018) found that teaching pathology, by traditional medical schools that are in the process of shifting toward an integrated SBL, needs to be more documented and addressed. As a result, the system-based practice has become increasingly common in medical education (Bhate et al., 2023; Castillo et al., 2020).

Accordingly, SBL, which emphasizes the integration of knowledge across disciplines and the development of practical problem-solving skills, offers a more holistic and immersive learning experience well-suited to meet the demands of the fast-paced and interconnected world of DevOps engineering.

While SBL and project-based learning (PBL) share similarities, such as their student-centered and real-world problem-focused nature, they also exhibit notable distinctions (Li & Zhu, 2023; Radenković et al., 2022). SBL emphasizes the comprehension of complex systems (Purao, Vaishnavi, Welke, & Lenze, 2009), adopts an interdisciplinary approach, and places greater emphasis on collaborative learning (Raj et al., 2021), e.g., focuses on system-thinking perspective (Ahlgren, 2013; Finn, Avalos, & Dunne, 2014; Kim & Senge, 1994; Mobus & Kalton, 2015; Spain, 2019). On the other hand, PBL primarily focuses on resolving specific problems and may exhibit a narrower scope compared to SBL. SBL projects tend to be more extensive and intricate, allowing for diverse interpretations and solutions, while PBL projects often have more defined parameters. In the context of software engineering, SBL offers a promising approach to teaching intricate subjects such as DevOps engineering. DevOps engineering involves the integration of software development and operations teams to automate and streamline the software development process while ensuring the delivery of high-quality software. Proficiency in this field requires a range of skills, including knowledge of CI/CD pipelines, containerization, and IaC. Table 1 summarizes the benefits and challenges of various teaching strategies that are used in various domains including software engineering and can be adopted to teach DevOps courses. In this research work, we are investigating the adoption of the SBL strategy in developing software applications from the requirements engineering and analysis to the testing and then delivery. Further research is needed to evaluate the effectiveness of other approaches and identify best practices for preparing students with the knowledge and skills necessary to succeed in the DevOps field.

3.1 Preparation

1. Define the research questions: The first step is to define the research questions. These research questions are aligned with the principles of SBL. The main research questions to tackle in this case study are as follows:

   1. RQ1: How can an SBL approach be implemented in a DevOps course?

   2. RQ2: How does this approach impact student learning outcomes compared to the traditional delivery method?

   3. RQ3: What are the perceived benefits and challenges of using SBL for teaching DevOps concepts?

2. Identify the learning objectives and key concepts: The main objective of this case study is to teach students the DevOps process in a way that they perceive the software development process as a whole system, rather than as a collection of independent tasks. The key concepts that students need to learn include communication and collaboration, CI, continuous delivery, and the importance of automation.

3. Design the course: Design the course includes

1. Select a specific course to deliver the DevOps material. In our case, we have selected an elective course in the software engineering curricula for senior students (400-level courses) to offer the DevOps course.

2. Design learning activities: The next step is to design learning activities that will help students to learn the key concepts. These activities should be hands-on and should encourage students to work together. The main activity focuses on involving students in projects to develop a software system using DevOps principles.

3.2 Implementation

1. Assess the current state of knowledge: It is important to assess the current state of knowledge related to the main DevOps concepts before starting the course. This will help you to identify any gaps in their knowledge and to tailor the course to their specific needs. In our case, and during the first class, the instructor asked the students to discuss their understanding of the DevOps process, the CI/CD concept, and the main automation tools. The students knowledge of these concepts was almost null. Few students mentioned that they had heard of the term DevOps but had no clue about how it works.

2. Deliver the course: At this phase, the DevOps course, which has been prepared, see next section, by the authors of this article, is delivered to students.

3. Provide feedback: It is important to provide feedback to students throughout the course. This will help them to track their progress and to identify any areas where they need additional help.

4. Update course as needed: As you teach the course, you may need to modify the learning activities or the assessment methods to ensure that the course is meeting the needs of the students.

3.3 Evaluation

1. Assess student learning: Finally, assess student learning to ensure that they have achieved the learning objectives. This assessment can be done through a variety of methods, such as quizzes, exams, and projects.

2. Reflect on the course: It is important to reflect on the course after it is finished. This will help you to identify what worked well and what could be improved for future courses.

4.1 Step 1: Preparation

The course “DevOps Engineering” is meticulously designed to equip students with the requisite knowledge and skills to implement DevOps as a method for delivering systems to enterprises with enhanced quality and velocity. To implement an SBL approach in a DevOps course, the following steps are taken:

1. Define what the System is? Developing a software system using the DevOps development approach is an ideal candidate for SBL because it inherently functions as a system. The system in the DevOps process consists of:

   1. Development (coding, testing, building, and version control).

   2. Operations (infrastructure and configuration management).

   3. Automation (test automation, CI/CD, and monitoring automation).

2. Design Curriculum and Learning Activities: The course is structured into ten modules, each focusing on a unique aspect of DevOps engineering. Each module includes succinct videos (2–5 min) that elaborate on each concept in detail. Moreover, a minimum of two detailed real-world case studies per module are incorporated, offering practical perspectives related to the module’s concepts. In the tenth week of the semester, two guest lecturers from the industry are invited to give a lecture (one for each of them) for students and answer their questions about the practical use of DevOps in real environments. The course adopts PBL where students work in teams ( 4–5 students each) to deliver their final project where they work together to build and deploy a software application using DevOps principles. Students in each project select an active open-source project and build a pipeline for it. The project was divided into three phases to make it more manageable and achievable for students. The course’s grade distribution is based on assignments and labs (20%), presentations (5%), a midterm exam (15%), a course project (20%), and a final exam (40%). During the course delivery, three assignments and nine labs are practiced by students. The labs practically guide students through the construction of a complete pipeline, from version control systems to exploring KubeCTL. They cover an array of topics such as version control systems, CI, containerization, configuration management, and continuous monitoring. The course targets six specific learning outcomes (CLOs), namely:

   1. CLO 1: Recognize DevOps principles, methods, and practices.

   2. CLO 2: Appraise DevOps principles and practices including integration, delivery, and testing.

   3. CLO 3: Illustrate how to apply DevOps tools used for monitoring, alerting, and reporting.

   4. CLO 4: Apply DevOps concepts in an enterprise environment by automating processes using tools.

   5. CLO 5: Select the best strategy to deploy software applications utilizing CI/CD pipelines.

   6. CLO 6: Apply the principles of DevOps to a software development project.

3. Promote collaboration and run feedback loops. The project and assignments’ activities encourage collaboration and communication among students. This reflects the real-world scenario where students play roles like developers, operations team members, and automation specialists and work together effectively. Students also practice the feedback loops where, for example, they analyze how a code change in development triggers the CI/CD pipeline to respond to that code change.

By achieving the specified outcomes and getting involved in the various activities and project work, students will be well-equipped with the necessary knowledge and skills to proficiently implement DevOps practices and principles in a real-world software development environment. Defining course objectives, course learning outcomes, course material, assignments, assessments, and project design that involves adopting the DevOps process to develop and release a whole system based on the SBL approach, all these activities provide an answer for the first research question RQ1: How can an SBL approach be implemented in a DevOps course?

4.2 Step 2: Implementation

The DevOps engineering course has been delivered twice in two consecutive semesters: Fall 2022 and Spring 2023. The section size ranges from 20 to 25 students in each semester. As presented in the preparation step, the course material and labs are delivered in 15 weeks per semester. As planned, the project runs through three main phases as follows:

1. In the first phase, students familiarize themselves with the chosen project’s specifics, including understanding its architecture and design. They provide the GitHub link to the project for transparency and accessibility.

2. The second phase shifts focus to the practical application of DevOps principles. Students undertake tasks such as local testing, code pushing into production, setting up a CI/CD pipeline, monitoring, and application deployment, thereby providing a hands-on introduction to the critical components of DevOps engineering.

3. The third phase requires students to execute another cycle of their pipeline, incorporating a new deployment with a new feature or a bug fix. This allows them to gain experience in iterative development and CI, essential facets of DevOps. They also compile a final project report and presentation, enhancing their communication and documentation skills.

The project was delivered on a dedicated GitHub public repository, which contained the project’s source code, DevOps setup related to the project, and project report (ideally as a set of linked markdown documents starting from the repository README). The project report described the project rationale and contained all artifacts related to the problem analysis and the design phases.

At the project’s culmination, students defend their design and implementation in an oral examination. Each student is allocated a 5-min presentation slot, with a maximum total presentation time of 20 min per group. The examination is an opportunity for students to articulate their understanding of the project, defend their design decisions, and demonstrate their practical skills. The evaluation, as discussed in step 3 below, considers the quality of the presentation, the project report, and the developed code.

To provide a comprehensive overview of the student’s experience and the quality of their work, we will delve into two exemplary projects completed by the students. The first project, known as IEEE CMS, entails a content management solution designed to facilitate the management of various types of contents by university clubs. This particular version focuses on catering to the needs of event planners and organizers. The project aims to streamline the process of managing and updating content on the chapter’s website and utilizes a CI/CD pipeline to automate the process of building, testing, and deploying the application. The project was undertaken by a team of five engineers, all serving as DevOps engineers who have worked together to build the CI/CD pipeline. Figure 4 showcases the CI/CD pipeline implemented for their project, and Figure 5 illustrates the deployment diagram associated with their project. The system consists of a container for a web server, another container for the application server, and a third container for the database server. The system also includes a CI/CD pipeline, which automates the process of building, testing, and deploying the application.

Figure 4

IEEE CMS CI/CD pipeline.

Figure 5

IEEE CMS deployment diagram.

The second project centers on the development of a Python AI assistant. The Python AI assistant aims to create a chatbot that can assist with a variety of tasks such as scheduling meetings, answering questions, and providing information. The project entailed the creation of “Jarvis,” a voice-command assistant service designed for Python 3.8. The team responsible for this project consisted of four engineers, all serving as DevOps engineers who have worked together to build the CI/CD pipeline. The project utilizes a CI/CD pipeline to automate the process of building, testing, and deploying the chatbot. Figure 6 portrays the CI/CD pipeline implemented for their project. Furthermore, Figure 7 provides a dashboard that showcases the status of each build. It provides a visual representation of the build status of the application, allowing the development team to quickly identify any issues or errors that may have occurred during the build process. The dashboard displays various metrics such as the build status, test coverage, and deployment status.

Figure 6

Python AI CI/CD pipeline.

Figure 7

Python AI build status dashboard.

4.3 Step 3: Evaluation

The course exit survey is an important tool to gather valuable feedback from students about their academic experience. It is designed to evaluate the level of student satisfaction with the curriculum and course being taught, as well as their confidence in achieving the course learning outcomes. Students were asked to use a five-category scale, ranging from “strongly disagree” (1) to “strongly agree” (5), to evaluate statements on their achievement level of each CLO defined for this course. Their feedback is an indicator of whether the adopted SBL teaching approach was effective or not. The students, who filled out the survey, were of the same level (senior students), and all of them had no experience with SBL teaching approach and that they had not practiced it before. Therefore, we can assume that the SBL was fairly new educational approach for all the students in this study. The average age for the students was 22 years old and all of them were male.

The course exit survey was conducted twice. The total number of students’ feedback for the two semesters was 45 responses (n = 45), and they were studied as one sample to analyze their responses and see how effective was the SBL teaching strategy, see Figure 8 for details about the overall achievement per CLO in each semester.

Figure 8

Course learning outcomes – students’ achievements.

The summative descriptive statistics are shown in Table 2; more than 70% of the students indicated that they were “Adequately Satisfied” or “Fully Satisfied” with their accomplishment of each course learning objective. This suggests that the learning process was adequately successful. CLO 1 and CLO 2 have the highest percentage of students (87 and 80%, respectively) reporting “Fully Satisfied.” This suggests that the course effectively equipped students with core DevOps skills. We noticed that there’s a slight decrease in the percentage of “Fully Satisfied” responses as we move from CLO 1 and 2 to CLO 3–6 (ranging from 73 to 78%). This might indicate a need for further refinement in how these later learning outcomes are addressed in the course. The low percentages (0–4%) in the “Not Satisfied” and “Barely Satisfied” categories suggest that most students felt the course adequately addressed the learning objectives.

Note that the standard deviation values for all CLOs are relatively low (between 0.27 and 0.34). This suggests a certain level of consistency in student responses, meaning the results are not overly skewed towards one satisfaction level or another. Overall, these results suggest a successful DevOps course with a strong foundation in core skills (CLO 1 and 2) and a generally positive student experience. However, there might be room for improvement in how the course addresses learning outcomes (CLO 3–6) to maintain the same level of high student satisfaction. The number of students who passed the course compared to the number who registered for the course is shown in Figure 9. The passing rate per semester is shown in Figure 10, and the detailed grade distribution of all students is also shown in Figure 11. It is worth mentioning that while preparing this article, a well-known local organization interviewed our graduates who have taken the DevOps course and were totally satisfied with their DevOps knowledge and skills and the job interviews resulted in hiring all those graduates. The discussions above answer the second research question: How does this approach impact student learning outcomes compared to a traditional delivery method?

Figure 9

Passing rate of registered students.

Figure 10

Passing rate per semester.

Figure 11

Course learning outcomes – students’ achievements.

4.4 Educational Recommendations

The results of the conducted study suggest that SBL can be applied to facilitate DevOps teaching courses. Based on our findings, the learning of important but rather complex topics, such as DevOps, can be fostered when students are able to apply and practice learning objectives by implementing the whole software development process as one system.

The key lessons learned and presented in the following section will help to understand and guide the design of an SBL solution to educate project management phenomena and practices. We assume that these lessons are not specific to teaching project management methods, like DevOps, but can be applied to the design and implementation of other learning solutions outside of the scope of project management as well. Key lessons learned are:

1. Ensure that the DevOps theoretical bases and SBL learning approach are explained clearly through lectures, reading materials, videos, and case studies.

2. Analyze the course content and delivery methods for CLO 3–6 to identify areas for improvement.

3. Explain DevOps pipeline phases and development requirements, and rules before start using the tools and practice building the pipeline so that the learning focus will be on the technical aspects, not the theoretical aspects.

4. Build repetitions into various DevOps projects and solutions: Repetition helps students memorize and learn.

5. Conduct follow-up interviews with students (especially those who reported lower satisfaction) to gain more specific feedback about their learning experience and refine the DevOps course to ensure that all learning outcomes are effectively addressed.

6. We suggest not waiting until the end of the course to gather feedback. In the future iterations of DevOps course delivery, we plan to conduct mid-course surveys or interviews to identify areas where students might need additional support.