A formative evaluation: co-designing tools to prepare vulnerable young people for participating in technology development

3.1.1 Phase 1: first evaluation and Re-design

For the first prototype evaluation in August 2022, we evaluated the toolkit with professionals in the context of media education. For this, we spoke to four professionals in three institutions – including two social workers, a media pedagogy and an engineer who volunteers in a makerspace. They got to know the toolkit and evaluated all modules without any additional hints or information. In the process, they were observed. They were able to ask questions during this process and in case of problems, experts provided sufficient information for further use. Afterwards, we held a reflection discussion based on guiding questions. This was recorded as a participatory observation by one of the researchers. Based on the evaluation results, problem areas and optimization needs were identified. This was followed by a redesign process.

3.1.2 Phase 2: second evaluation and Re-design

The second phase evaluated the redesign process and tested the re-designed set of tools in real-world application contexts. Among others, these were two youth residential care institution with residents. The evaluation phase took place six months later, in February and March 2023. It was conducted with all involved participants: vulnerable young people as well as the professionals. These sessions were accompanied by participatory observation with professionals talking out loud with experts and researchers present.

3.3.1 Data collection: participant observation

The evaluation was conducted using the qualitative research method of participant observation. This research process based on the specifications of qualitative empirical research. Qualitative empirical research methods make it possible to respond individually to experiences and feedback in the form of social interaction. ²⁸ The evaluation can be designed to be as open to findings as possible. Lamnek’s argumentation promises that qualitative research methods are better suited to evaluate social situations and human actions. Given this perspective, the following work will pursue a logic of discovery. An approach that is deeply rooted in qualitative social research. ²⁹ Participant observation represents a form of qualitative research perspective and is anchored in field research. It is therefore the study of a social or cultural group in a natural environment. Observation means selective, systematic, and attentive perception. Systematicity is shown in the targeted search for impressions that have been previously discovered or describe a hypothesis from theory. The method thus goes through a process from passive perception to active observation. The researchers remain attentive and open to new impressions. This process represents the first evaluation step in which impressions are processed into patterns. ³⁰ The extent to which this observation can be described as participatory is explained very well by Atteslander; ³¹ who differentiates between the various possibilities of participation: He does not distinguish between participatory and non-participatory, but between observations with a high degree of participation and observations with a low degree of participation. A minimum degree of participation is always given by the observation situation, e.g. by being present or being perceived with all sense organs.

Participant observation is the basis for the evaluation of the toolkit. In addition, there are subsequent conversations that serve to reflect on the field research. One of these conversations was recorded and is also used for the evaluation. The conversation with two service designers from the research team would be transcribed and finally analysed using also qualitative content analysis. ²⁶ Summary analysis was used and inductive categories about barriers in the inclusive design process were formed.

3.3.2 Data analysis: Mayring’s qualitative content analysis

All collected data were analyzed using qualitative content analysis according to Mayring and Fenzl. ²⁶ A categorization system was formed inductively from the data. This system is consistent across the first and second evaluation phases. Mayring’s qualitative content analysis was used as the method of evaluation for both the participant observations and the reflective discussion. There are three basic forms of this method. What all basic forms have in common, however, is the systematization through categories, so that it is a rule-guided, comprehensible procedure according to which the material is evaluated (ebd.).

The analyses resulted in the following system of inductive categories. The Table 2 shows the categories with examples and rules of the categorization system. There are four categories: orientation, mastering technology, materials and educational potentials:

4.1.1 Orientation and structure for safe handling

A central finding from the evaluation process is the need for improved orientation when utilizing the toolkit. Although the kit purports to be self-explanatory, this attribute is not fully realized in the initial prototype. While the color coding of the kit provides a rudimentary framework for orientation, there is an absence of predefined order. The intentional omission of module numbering, aimed at enabling users to freely engage with the modules, is regarded as less meaningful according to the experts. Clear guidelines are essential to prevent potential uncertainties from arising. Consequently, the inclusion of a step-by-step guide is deemed necessary. Numbering the individual modules was suggested to aid in establishing a cohesive narrative and fostering a stronger connection between the modules.

The instructions provided within the toolkit served as an additional framework for orientation. However, the evaluation reveals a disparity between the preferences of professionals and the vulnerable young people. While professionals observed in phase 1 expressed a desire for detailed instructions and supplementary videos, young people observed ion phase 2 tended to favor a more hands-on approach. This prompted the idea of developing facilitation instructions tailored for professionals while simplifying the instructions for vulnerable young people. Moreover, the instructions contain terminology that could benefit from being expressed in simpler terms. To minimize barriers, the instructions should be scrutinized for accessibility and written in plain language. Additionally, incorporating videos as an initial point of engagement with the kit, particularly for children and young people with dyslexia, is considered favorable. Moreover, the evaluation showed that an inclusion of subtitles in the videos can also help alleviate language barriers.

4.1.2 A playful approach to mastering technology

The professionals have displayed great enthusiasm for the toolkit, with a particular interest in the playful approach to mastering technology through an escape game. They view the riddles incorporated in the escape game as highly beneficial. It seems like the purpose and context of the kit only became clear through engagement with the escape game. Although professionals outline diverse objectives, these objects all share a central intention: identifying problems, formulating solutions and implementing them. The escape game served as a catalyst and source of motivation for this process. Riddles and their subsequent solutions fostered a sense of accomplishment and were perceived as rewards, for instance, the opening of the treasure chest.

The escape game component was perceived as the focal point of the toolkit, garnering significant attention and positive feedback. The riddles themselves are regarded as user-friendly and valuable. However, challenges arose within the game when hints were required and needed to solve a riddle. The existence of relevant hints on some papers was often overlooked, resulting in their underutilization by both professionals and vulnerable young people. Consequently, it was necessary to enhance the visibility and presence of them. Furthermore, including a comprehensive troubleshooting section within the moderation instructions could facilitate the efficient resolution of technical issues or obstacles that may arise during playing the game.

The toolkit, in conjunction with the escape game, employs smart home technology tools. From the perspective of professionals, this presents a potential risk of limiting vulnerable young people in terms of their creativity and design possibilities. The tools employed do not directly expose the underlying technology and are typically processed. Different viewpoints among professionals emerged in this regard, with some appreciating the user-friendly nature of smart home technology, while others express concerns about its mystifying effect due to processing. Consequently, it was recommended to critically examine the incorporation of smart home technology within the toolkit, including fostering critical reflections on data protection with the vulnerable young people. Additional suggestions in this regard included redesigning the treasure chest to eliminate its transparency, thereby creating a surprise element when it is opened.

Overall, the initial phase of evaluation revealed the need for stronger integration of the game within a narrative framework. Adopting a storytelling approach was seen to offer the potential to foster critical engagement with smart home technology. Furthermore, this strategy of redesign was expressed as an opportunity to establish a more cohesive connection between the game and the other modules. Such an approach was visioned as potentially impact positively on the motivation of children and young people by piquing their curiosity.

4.1.3 Educational potential

The preliminary test phase indicated that there was untapped potential for further education and learning opportunities in the domain of technology. While the use of technology modules was presented engagingly, there was a lack of comprehensive explanations regarding the functions of these modules. To demystify the technology and enhance understanding, it is essential to provide explanations of technical terms and functions to the vulnerable young people. One effective approach to achieving the potential was the development of a specialized booklet.

Furthermore, the evaluation revealed that the module of analog prototyping required improvement. The combination of coding cards was meant to picture the ideas of the target group. Professionals recommended aligning the cards more closely with an existing programming language. This approach enabled step-by-step comprehension of programming principles playfully. Additionally, employing assorted colors and shapes for the cards, based on the type of word, would not only facilitate learning effects but could also create novel incentives for proficient individuals to utilize the toolkits.

Additional suggestions have been proposed to enhance usability, mostly regarding the materials provided. For instance, the tests conducted revealed a lack of paper and pencils within the kit for notetaking. It was recommended to supplement the module for analog prototyping with blank coding cards to encourage the creative expression of the target group. Technical difficulties arose when participants used their devices; thus, the inclusion of a pre-programmed tablet could prove beneficial. Concerning additional materials, the potential implementation and utilization of a diary method (probe kits ³² ) was discouraged due to its extensive requirements, encompassing both data protection and material prerequisites. In conclusion, by addressing the need for comprehensive explanations and optimizing the module of analog prototyping the educational potential and learning processes associated with technology may be significantly enhanced. These improvements could be effectively implemented through the development of a specialized booklet and a closer alignment of coding cards design with an existing programming language.

4.2.1 Toolkit module 1: experiencing and understanding

A new module 1 has been designed. It aims at offering elements that allow for experiencing smart home technology and at understanding it. How can technology be experienced and understood? The escape game and the technology tools were connected and serve to allow for experience. In addition, the module has been expanded to include sort of a lexicon (booklet). This booklet contains background information on how the technology works and explanations of specific terms, such as IP address. It aims at promoting an understanding of technologies experienced. These explanations are written in simple language and designed for the target group. This triggers a learning process and exploits the educational potential of the toolkit. The focus is no longer only on experiencing the technology tools, but also on understanding the technology. For this purpose, smart home technology can also be tried out without playing the escape game.

In addition, the escape game has been optimized by adding additional riddles to include all technology tools in the game. To further enhance the learning effect, the game is now integrated into a narrative that illustrates the possible risks of smart home technology: A city has been taken over by hackers. Players have to take control of a smart home system to liberate the city. This creates space for a discourse on data protection and smart home technology in general. However, the storytelling provides a new incentive and motivates the juvenile kids to solve riddle after riddle. To get everyone involved, each riddle has one player who reads it out loud and takes a leading role. To further increase motivation, the treasure chest was redesigned, and a website was programmed as a control center. The aim of the game is to get to this website, from where all the technical tools can be controlled.

The instructions for the escape game have been revised and all relevant hints have been highlighted. There are also additional hints to help you solve the riddles. These hints are intended to prevent technical problems. A pre-programmed tablet is now available for this purpose.

4.2.2 Toolkit module 2: co-designing

A new module 2 has been designed. It aims at offering activities that let players experience their creative potential in co-designing solutions to problems in everyday life. What components were needed to reach this objective? Elements for design, methods and analog prototyping are combined. Methods for naming everyday problems and needs were hardly considered in the first phase of the evaluation. However, these methods are relevant to first describe a problem and second to find ideas and implement them. The redesign should emphasize this relevance and make the methods more accessible. To this end, the diary method was discarded because of its presuppositional nature, and a new playful method was introduced – the idea cube. The idea cube is designed to make the idea-finding process playful and low-threshold. It also provides a good transition between the game cards and the coding cards of the analog prototyping. This results in a stronger embedding in the overall context.

The coding cards have also been redesigned, based on the desire for word groups in assorted colors and shapes. The aim is to encourage logical thinking in vulnerable young people. Moreover, it gives them an idea of what programming is all about. In addition, the request for paper and pencil for notes and blank coding cards has been implemented. These additional tools are intended to make Module 2 more accessible.

4.3.1 Orientation and structure for safe handling

In the media education institution, two professionals assessed the toolkit. They recognized an improved structure of the kit with a clear starting point and a logical sequence to follow. In particular, the color codes were evaluated positively and provide a meaningful orientation framework. The reduction and consolidation of elements into two modules were also considered meaningful. Module 2 gained relevance through the new structure, so the kit then represented a comprehensive concept and was not solely limited to the escape game. The professionals described the goal and purpose as follows: the escape game provides experiences through action and introduction to digitized technology, followed by a module to expand these experiences creatively. The professionals referred to the second module as the computer science module.

The guidance provided was critically evaluated: although there was a request to provide more detailed instructions with background information, the professionals now perceived them as excessive. According to their statements, in a real life setting neither the professionals nor the vulnerable young people would review all the materials. They wished for the option of a quick start or video instructions. This would save time for professionals and reduce the amount of reading for the vulnerable young people. In general, color codes could also be helpful here to highlight particularly important passages. This resulted in an instruction manual with two options: first the comprehensive self-explanatory version and second the quick start version. In general, the professionals recommend making the instructions more explicit so that the essential aspects are immediately clear through the design. This also applied to the riddle manual of the escape game.

The groups of vulnerable young people initially approach the tests with some reservation. However, they recognize the sequence of steps. They do not have difficulties orienting themselves. Instead, they appear skeptical of the testing situation in general. In reflective conversations, it appeared that this skepticism may be based on underlying uncertainty. Once juveniles were encouraged to open the toolkit and when each step became clear to them, they did not question them. It seemed that in moments of disorientation, the uncertainties of young people often become an issue. As soon as the professional encouraged them to continue, the next step became clear to them. This uncertainty appeared also when reading or listening to the instructions. Texts, reading, and complex wording were hard to understand. The texts were often too long and contained incomprehensible terms. Obviously, language barriers and barriers related to reading and writing difficulties arose as a result. A solution could be to shorten the instructions and/or to highlight the essential parts. In addition to revising the texts, and audiovisuals with subtitles or translations could be a solution.

4.3.2 A playful approach to mastering technology

The professionals were pleased with the new narrative of the escape game and considered the new riddles and the incorporation of additional technological components to be meaningful for the toolkit. They perceived this as offering a new approach to motivation and education. A professional described how the toolkit could motivate and encourage people to get to know new technical components. In particular, she found the escape game very successful. All terminologies and technologies were now explained comprehensively. To make this information as accessible as possible, working with video explanations was once again suggested.

Initial skepticism dissipated in the testing process, especially when playing the escape game. The escape game progressively increased curiosity through riddles, creating an incentive for engagement. Playfully, the youth familiarized themselves with the functioning of the technological components, and exclamations like “Oh, I see!” and “Ah, now I understand!” were heard repeatedly. Although the booklet was used, it was only referred to when a solution was not found through mere experimentation. The escape game thus aimed at preparing the youth for the design module through experiential learning and understanding.

Although the level of activity varied among the youth, all of them participated. In particular, the narrative appeared to be a great motivator for the youth. Through the story, they were motivated to think through and engage in the riddles. It became evident that the youth understood the technological components and their functioning so well that they used the functions to reach their goals more quickly. For example, there was one riddle in which nobody was supposed to move. The youth in one institution found it particularly easy and simply covered the motion sensor, allowing them to continue moving freely.

4.3.3 Educational potential

However, they saw the greatest potential for learning through the redesign of Module 2. One professional alternately referred to the module as the computer science or logic module, as the methods may help the vulnerable young people learn logical thinking. The professionals described the game cards and dices as brainstorming tools to generate ideas. The kit, particularly with the method of analog prototyping, provides an entry point into IT language. The vulnerable young people may learn the logical sequence of programming and gain theoretical knowledge of coding. Thus, the toolkit could provide a starting point and an important creative approach to programming. It is conceivable for professionals to implement an extension for advanced young people to promote the learning process.

In addition, recommendations for an inclusive design process (specifically for the kit) were derived from the second evaluation. According to social workers evaluating the toolkit, the toolkit is not suitable for groups with similar physical disabilities; young people with physical disabilities can only participate in a heterogeneous group. This is because the Escape Game appeals to different senses, sometimes it needs good hearing, and sometimes it needs good motor skills and vision. However, according to the experts’ assessment, the kit may also be suitable for autistic children. They see the area of application partly in youth residential facilities or in schools and extracurricular leisure activities with media pedagogues.

However, in the real-world application, it is also apparent that the game requires more concentration and time than anticipated by the designers, resulting in decreasing motivation among the youth towards the toolkit. Not all participants remained until the end in both groups. This can be addressed by using the kit on multiple occasions. For example, one day could be dedicated to Module 1, and the next day to Module 2. Furthermore, Module 2 requires more support from the professional. The everyday life and needs cards (game cards) should be sorted in advance and tailored to the target group.

The toolkit was specifically evaluated with socially disadvantaged and therefore vulnerable youth. The toolkit was not self-explanatory. The evaluation showed that a technology toolkit aiming to make a new topic accessible to the target group of vulnerable young people in a social work residency required some guidance from a professional. In this regard, the support in media literacy skills or reading skills was not always as relevant as simply the mere presence of the professional. It is important that moderating professional’s role is only to encourage the youth and only intervene and provide support in specific cases. It became evident, particularly through the escape game, that the youth found an initial entry point into smart home technology and successfully applied its functionalities reaching the escape game’s final solution. Although the instructions posed a barrier, it was also observed that these are secondary, and much can be accomplished through experimentation. In Module 2, the instruction texts were already efficiently reduced, so there were no issues regarding reading or listening. Even if the youth misread or misspelled something, it became not necessarily an obstacle. They were still capable of using the game cards and dice and came up with ideas for their everyday challenges.