Exploring technical implications and design opportunities for interactive and engaging telepresence robots in rehabilitation – results from an ethnographic requirement analysis with patients and health-care professionals

2.1 Social robots in rehabilitation

Socio-robotic systems are interactive robots that are designed to interpret and respond to social signals. ²⁴ Social robots in rehabilitation and care can interact with humans using various sensors such as cameras, microphones and touch sensors. ²¹ They are able to recognise and respond to gestures, facial expressions and speech, making them valuable tools in therapy. ²⁵ Socio-robotic systems can for example be used in motor rehabilitation to help patients regain their movement skills. In recent years, the use of robotics in rehabilitation has increased significantly. Socio-robotic systems in particular, which combine social interactions and practical applications, open up new possibilities for supporting patients with different health challenges. ^{14 , 26} These systems combine technological innovation with social relevance to develop effective and user-friendly rehabilitation methods. A socio-informatic and practice-based approach is particularly relevant in the development of technology and robots, as it takes into account the social, cultural and individual needs of users. This promotes the acceptance and effectiveness of the technology, as the systems are directly tailored to the real-life conditions and requirements of patients. In rehabilitation, it is crucial that the technologies are not only functional, but also take into account the social dynamics and therapeutic needs of patients.

Compared to other devices, social robots can increase the engagement of patients. This is due to embodiment, mobile navigation and multimodal and interactive capabilities. ²⁷ The high engagement of the patients, along with positive feedback, was seen in Ref. ²⁸ where a long-term intervention with the robot Pepper was conducted. However, social robots assist the therapist, but cannot replace them. ²⁹ An important aspect of rehabilitation with social robots is persuasiveness, i.e. the robot must encourage the patient to perform certain activities. In this context, a goodwill or similar character is advantageous. ³⁰ To encourage patients to become more active, social interaction may be important, and here experiencing fun on the side of the patient may be especially beneficial. ³¹ In this context, also trust of the patient in the robot is relevant. ²⁹

Polak et al. developed a guideline based on experts evaluation for the use of social robots in rehabilitation. ²⁸ Main aspects of the guideline are that a variety of tasks with different levels of difficulty should be available and the feedback should be given in an appropriate manner. Moreover, the interaction should be personalised to the patient. This was also addressed by Winkle et al., where improving the ease of access is also mentioned. ²⁷ In addition, studies have shown that interactive robots such as NAO and PARO can be used effectively in post-stroke rehabilitation. These robots offer exercises, provide feedback and motivate patients through their social interaction. A study by Fasola et al. showed that socio-robotic systems can increase patient motivation and engagement, leading to better therapy outcomes. ³²

Social robots that provide emotional and physical support are playing an increasingly important role in the rehabilitation of older and disabled people. ³³ These robots not only contribute to physical recovery, but also to the emotional well-being of patients. The COVID-19 pandemic has further emphasised the importance and potential of social robots, as they are able to provide social interactions and support in times of isolation. A study by Bemelmans et al. showed that the use of robots such as PARO in care facilities led to a significant improvement in the emotional well-being of residents. ³⁴

2.2 Creative interventions and the role of music in rehabilitation

In the field of human-computer-interaction, design for emotional engagement and psychological well-being is becoming increasingly important. ^{35 , 36 , 37 , 38} This is especially crucial in contexts such as rehabilitation, where fostering motivation can significantly impact therapy success. ³⁹

Various technology-supported activities have already been implemented to facilitate interaction between people, including sports activities that enable running together over a distance. ^{40 , 41 , 42} Research has shown that such technology-enabled approaches can have a positive impact on participants. One therapy with significant potential in rehabilitation is music therapy. Studies have demonstrated that it can not only reduce perceived pain and effort but also alleviate social isolation while enhancing self-efficacy and mood. ^{43 , 44 , 45}

There is a wide range of literature on HCI in the context of music. Much of it covers aspects of music production (e.g. Ref. ^{46 , 47} ) which is clearly beyond the remit of this paper. A similarly divergent literature is interested in how computer systems might contribute to aspects of performance (e.g. Ref. ^{47 , 48 , 49} ), composition (e.g. Ref. ⁵⁰ ) and more musicological interests such as genre classification (e.g. Ref. ⁵¹ ). In this respect, and somewhat closer to our own interests, is a small body of research that looks at how to relate music to fitness-based activities ^{52 , 53} and rehabilitation (e.g. Ref. ⁵⁴ ). Some work has also taken a design perspective on the relationship between music, auditory cues and cognition. ⁵⁵

The interface between music and HCI, also known as music interaction, has made significant progress in recent decades. ⁵⁶ This design implication explores, how music can be used as a medium to enhance user interaction with digital systems. It analyses the cognitive, emotional and behavioural effects of music on the user experience and develops new technologies and methods to apply these findings in practice. Current fields of research include adaptive music systems, music-based gamification, therapeutic applications, the integration of AI into music systems and multisensory music interaction. ⁵⁷

It was shown by ⁵⁸ that music listening can improve recovery after stroke, which may include cognitive, emotional and anatomical aspects. Furthermore, a positive effect of music on mood and arousal has been demonstrated. It was shown that music can be a motivator to do activities ⁵⁹ and reduce the discomfort and difficulty perceived with activities. ⁶⁰ The potential of music in rehabilitation was shown in for example ⁶¹ where musical rhythm could be used to train movement pattern in patients with movement-related disorders.

3.1 Research approach, setting and objectives

Within our research framework we conduct a design case study following the methodology as Wulf et al. ⁶² originally articulated. This research approach is characterized by three ideal stages: (1) a pre-study to develop an understanding of people’s experiences, needs, and expectations, (2) a design-phase to iteratively design, evaluate and re-design the prototype, and (3) an appropriation study in which we investigate how residents and caregivers used our system. Beyond this, our approach was shaped by a particular praxeological understanding of design research – Ref. ⁶³ that is based upon the conjoint pursuit of design case studies and grounded design. ⁶⁴ Following the suggestions of Dewsbury et al., our aim is to place the needs of stakeholders at the centre of the system’s design. ²² Here, we focus on the results of the pre-study, for which we followed an ethnographic approach ⁶⁵ to better understand the context in which the prospective technology might be deployed. This qualitative approach consists of field notes from three observation days in a rehabilitation facility, where three researchers were each able to accompany a physiotherapist through their daily routine and ask questions if necessary, and semi-structured interviews with rehabilitation patients. The aim was to cover the views of both patients and professionals. Our research aim on a close collaboration between the targeted application area and the diverse stakeholders involved. ⁶⁶ The participatory inclusion of users and stakeholder takes a special focus in this context. The practical partner institution in <anonym> is an interdisciplinary rehabilitation and health facility that offers a wide range of services focused on prevention, therapy, and recovery. Established in 2005, it is closely connected to the <anonym> Hospital, ensuring seamless medical care and access to specialized expertise. The facility specializes in ambulatory rehabilitation, including orthopaedic and cardiological rehabilitation, and provides post-therapy programs to support recovery and reintegration into daily life. Their approach emphasizes personalized therapy plans tailored to individual needs, with a focus on integrating rehabilitation into patients’ everyday routines for sustainable health outcomes. The facility is certified for high medical standards and collaborates with various health insurers to ensure accessible care.

3.2 Data collection and analysis

In our pre-study we explored the existing individual perspectives of patients, as well as the multi-professional practices, organisational and individual work-flows and social routines in the multi-professional teams as well as challenges that various professionals in rehabilitation-facilities face in their everyday surroundings. In this way, participant observations could be methodologically flexible and open in order to capture emergent phenomena in the field. At the same time, the observation was guided by the clear interest of finding out what a helpful technical system could look like and how it could be implemented. Notes and memos were taken continuously and systematically to support reflection and data analysis. ⁶⁵ This process led us to 15 pages of field notes in total and included a broad interview study conducting 14 semi-structured interviews with patients of the cooperating rehabilitation facilities.

The interviews were conducted on several days in presence at the rehabilitation facility. All interviews were scheduled into the patients’ rehabilitation program. These interviews covered four areas: Firstly, personal rehabilitation training was addressed: (How) are specific exercises, for example, successfully taught to patients and is there sufficient supervision and feedback when performing them? What does the latter ideally look like for the patients? Secondly, the individual influence of music on motivation was analysed before the subjective affinity for technology was briefly surveyed. Finally, the project plan to temporarily introduce a music-induced, robotic movement system was presented, and attitudes and ideas about how this could be designed were asked.

The transcripts were coded and categorised using the MAXQDA program ⁶⁷ and then evaluated using qualitative content analysis. ⁶⁸ For this purpose, the transcripts were analysed in an iterative process, statements were categorised and then a hierarchical category system was created as described in the inductive categorisation method by Kuckartz & Rädiker. ⁶⁸ Based on that the subsequent summary and analysis was done in a category-oriented manner according to the qualitative content analysis.

The field notes and the interviews complement each other, as they create a sound basis for our design on two levels. On the one hand, the participant observation enabled us to understand on a general level how the robot can be used profitably in the daily routine and which exercises are both helpful and safe for all patients. For example, the research team quickly realised that the robot should be used during a so-called active break, and some exercises were identified that could be performed by knee, hip as well as shoulder patients, for example. On the other hand, in addition to these general social conditions, the interviews made it possible to determine the individual technical needs of the patients, as the technological design had to be adapted to the needs of a group that consists mostly of older individuals, which led, for example, to the use of a sans-serif typeface and a large font size on the tablet. In general, it can be said that the participatory observation served as a generator of context knowledge by personal experience. ⁶⁵

3.3 Sampling and participants

Both the treatments observed on the observation days and the people interviewed were selected by the rehabilitation employees. Care was taken to ensure that all rehabilitation treatment areas relevant to this study were covered. These included the introductory session, physiotherapy, group exercises and fitness training. The observed treatments within the treatment areas were selected randomly and according to the availability of staff and patients. The patients for the guideline interviews were also selected at random according to patient availability, but care was taken to cover all relevant clinical pictures of orthopaedic rehabilitation. As the interviewees were questioned at different times during their three-week rehabilitation, it is possible that a lack of experience due to the short duration of treatment may have influenced the rehabilitation-specific answers. Interviews were conducted with a total of 14 patients, of which only 13 could be analysed due to one person refusing to be considered and evaluated in the study. This person seemed generally suspicious of using IT in the field of healthcare sector and did not want to support the research team in “rationalising” people away. Based on discussion with the data protection officer, we were not allowed to collect any personalised health data according to the data protection declaration. We insisted that the sample should be as heterogeneous as possible in terms of age, gender, socio-economic background, degree of current movement restriction, rehabilitation needs and anticipatable technical attitude. Ten participants had undergone surgery prior to rehabilitation, while three participated without prior surgery. The reasons for rehabilitation varied: six participants were recovering from hip problems, three from slipped discs, two from shoulder issues, one from a knee operation, and one from a broken arm. One participant had already received physiotherapy before starting rehabilitation, and five participants had undergone multiple rehabilitation cycles due to previous joint replacements. Additionally, one participant reported a planned operation in the future. Rehabilitation was therefore a familiar environment for many participants, though their levels of physical activity varied significantly – from highly active individuals to those with sedentary lifestyles or severely restricted movement. Some patients are very sporty, while others are not active at all. Both the additional motivation and the different sporting levels of the patients could be covered by offering different levels of difficulty for the exercises. This diversity highlighted the need for adaptable and inclusive robot functionalities to effectively support the wide range of participant needs and abilities.

3.4 Ethical considerations

In this study, patients were informed in advance for ethical reasons and were only observed and interviewed with their consent. Treatments were partly observed passively, but we also interacted with patients and staff in the form of conversations in order to obtain more contextual information. We ensured the privacy and data security of the participants by conducting interviews with them in accordance with various documents that uphold the privacy of personal data and comply with legal regulations. These documents were only applicable upon obtaining consent, which participants could revoke at any time. It was very important for managers of the rehabilitation institution not to record and publish any personal data, which is why demographic data is not provided in tabular form here.

4.1 Individual patient needs, work-related routines and therapeutic practices

4.2 Socio-robotic requirements for motivation, monitoring and progress in rehabilitation

4.3 Daily challenges in therapeutical practice

4.4 Need for an adaptive and feedback-oriented system

5.1 User and system requirements – design, explanation, feedback, monitoring and control

By far the most important point that almost all participants mentioned as an expectation of the robot was that the operation should be simple and as self-explanatory as possible. Even though all participants have a basic knowledge of how to operate touchscreen-based devices, this is not always the case, especially for older patients. In addition, older patients may have limited motor skills, vision and hearing. The design and menu navigation of the application should therefore be chosen so that as few steps as possible are required to start an exercise. The controls should all be easy to reach. Orientation towards validated design guidelines for older adults, such as those outlined in ISO 9241-210:2019, ⁷⁰ would be useful. This standard emphasizes that the system should be designed to achieve the intended goal with minimal effort, and that the design should minimize both physical and psychological harm. For instance, it should avoid causing cognitive overload.

One of the patients’ main requirements is that the robot explains the exercises well. As all interviewees are also satisfied with the explanations used in rehabilitation to date, it is appropriate to use the previous methods of explanation as a model. The professional therapists employees argued that demonstrating the exercise and verbal explanations are most effective. One way of illustrating this on the robot would be explanatory videos in which the exercise is demonstrated and explained.

Almost all patients mentioned the control of the exercises by the robot during execution as a further requirement. This prevents incorrect training and, above all, prevents negative consequences/injuries. Patients who had knee surgery, for example, should not kneel too low. The application should therefore be able to recognize when exercises are performed incorrectly, notify the user and correct them. However, it was not clear from the interviews what this notification and correction should look like. One possibility would be to play a warning tone or an error description via an announcement to indicate incorrect performance. Optionally, a symbol or red signals could also be shown on the display to indicate this. The correct procedure should then be explained. This could be done verbally via an announcement or visually on the display.

Some patients also asked for a direct feedback – during and after the exercise. The feedback should show how well the exercise was performed, what was done wrong, but also what can be improved in the future. One person suggested that the exercise be graded. To ensure this, a score could be calculated after the exercise to determine how well the exercise was done objectively. At the same time, the number and type of errors should be shown and possibly also included in the calculation of the grade. As this is also intended to motivate, a form of praise could also be displayed for good performance.

The progress of the exercises in the previous rehabilitation is not always recorded. However, some digital devices in the fitness room already do this, which is positively received. Some participants also wanted the robot to have this function in order to motivate them and help them to improve. By assigning an individual ID and saving and displaying past results, it would be possible to fulfil this requirement.

5.2 Feature and innovation requirements – difficulty, implementation, exercises and music training

For some patients, improving their performance is a motivation that drives them to exercise. The possibility of different levels of difficulty in order to be able to improve was also desired by users in the context that progress is recorded. In addition, the potential patients are very diverse in terms of their performance. One reason for this is that the age range is very wide, as is the sporting background of the patients. Some patients are very sporty, while others are not active at all. Both the additional motivation and the different sporting levels of the patients could be covered by offering different levels of difficulty for the exercises.

Patients often have breaks in their program. In addition, staff are particularly busy when both rehabilitation groups are present. It therefore makes sense to offer the exercises with the robot during these times. Some patients would also like to do more exercises during these breaks. The offer should initially only be optional, as not all patients want to operate a robot with this type of introduction. Nonetheless, it would be questionable whether this would relieve the burden on staff or whether it would lead to an additional workload, as some of the staff fear.

It is also important that therapists are available to give patients an introduction and are available to answer questions. Older patients in particular saw this as a prerequisite for use. The handling of the robot would thus be similar to that of the equipment in the fitness room. This would also meet the desire of employees to use the mobile robot as a tool rather than independently. This would require staff training. However, as the employees are open to new technology and do not feel overburdened by it, this should not be seen as a problem.

Five exercises were identified together with the therapists that should be implemented for the robot. A ski jumping exercise, balance exercise, calf stretch, squats and lunges. These are exercises that are suitable for all groups of patients. Some of the exercises can be made more difficult or simplified using aids. This means that two levels of difficulty can be offered and consideration can be given to patients with mobility impairments. Almost all participants and staff agreed that music can have a motivating influence. The idea of adapting music to the rhythm of the patients was also received rather positively. The use of music in the application can therefore be rated as positive. However, it should be possible to select the music or at least the genre, as some patients are only motivated by music that they like. As the application is only intended for individual exercises, the resulting conflicts of interest between patients in group exercises do not play a role, but should be taken into account when considering functional enhancement. It should also be possible to carry out the exercises without music, as some patients feel disturbed by this. As some rooms are very quiet, this should also influence the choice of location so that other patients and staff are not disturbed.

6.1 General design objectives and technical infrastructure

Inspired by the related work and building upon the implications we conceptualized and designed a system that integrates four primary technical components: a mobile telepresence robot (developed by temi-robots ⁷¹ ), a pose detection module, an application program with different exercise training activities and adaptive music software, as illustrated in Figure 1. The system is designed to be operated through the Tablet of the telepresence robot temi via the internal Android system (Android version 6.0 onwards (SDK23)). The Android app acts as the control hub, managing pose detection and communication with the external music software as well as providing feedback and instructions to the users.

Figure 1:

Illustration of the technical setup. For pose detection, the Z-coordinate was ignored.

The mobile telepresence robot temi serves as the mobile physical interface for the user. It is equipped with an internal camera that we use to track the user’s movements and provide real-time feedback to the system and users. The robot’s mobility allows it to be remotely controlled, move and follow the user or position itself optimally to capture the user’s movements during exercise routines. The system was designed to be used in single-user mode, meaning that only one person can be detected. The UI is designed to be intuitive, providing users with real-time feedback on their movements and progress during exercise routines. The application displays a visual representation of the user’s body points and highlights any incorrect postures or movements, aiding in self-correction. Additionally, the user-interface was developed with older users in mind. To this end, the developer guidelines for the user-friendliness of apps for older individuals were followed (Ref. ⁷² ). This includes the use of simple language, intuitive navigation to achieve the aim, high contrast between background and text, large buttons, a simple design, avoiding technical terms, and providing a tutorial. To individually Log-In and keep track of the individual session we integrated a User-Login via IDs that were also the participant ID within the study design to ensure anonymity.

Combined with our design implications, it was clear in our own work that the following considerations needed to be borne in mind in designing the applications: (1) creating an emotive design that would make the participants feel comfortable; (2) ensuring minimum complexity so that participants would not be overwhelmed and harmed by the tasks; (3) making a good first impression to increase the likelihood of acceptance and use; (4) providing positive feedback from the system to put the participants at ease and make them feel that they were proceeding correctly; (5) keeping instructions and things to be learnt as simple and short as possible, with the option to repeat them as many times as necessary in case of doubt; (6) provide a musical atmosphere to support engagement and motivation; and (7) maintaining a simple, high-contrast design to ensure easy readability. Therefore, we overall designed and developed the following five digital exercises.

6.2 Digital exercise applications

Within our empirical studies, we also focused on the content and physical challenges of our applications. The exercises implemented, and also the underlying training and progression plan for the robot were co-designed with therapists to ensure they align with standard orthopaedic rehabilitation practices, are non-risky for patients, and cater to a broad range of abilities. Five exercises were selected: ski jumping, balance, calf stretches, squats, and lunges. These exercises, as shown in Figure 2, are fundamental in rehabilitation, suitable for all patient groups, and can be adjusted in difficulty using aids, allowing for personalization based on patient mobility. The empirical feedback highlighted that music could be a motivating factor, with participants suggesting the ability to choose or adapt the music to individual preferences and levels of activity during the session.

1. Ski Driver – To correctly recognize the skier’s movement, the user must stand to the side of temi in order to correctly recognize the arm movement. An important point in exercises in which the users stand to the side of the camera is the recognition of the direction of gaze. On the one hand, this is important because the landmarks on the side facing the camera are more accurate, as the body may prevent the other side of the body from being seen. On the other hand, it is necessary to determine what is behind and what is in front from the person’s point of view. A skier’s execution consists of taking his arms in front of him at shoulder height in a squat position and then pulling them behind him at the side of his body.

2. Squats – Squat detection works in a similar way to skier detection. Here, too, the person’s line of vision must first be recognized, as the exercise is also performed sideways to the camera. A repetition is counted when the person has reached the lower knee bend position and then stands upright again. The lower position is reached when the knee angle is less than 105°. The person stands upright as soon as the angle is greater than 165°. An activity value is then calculated on the basis of the repetition time.

3. Balance – In the balance exercise, the person stands facing the camera. The balance position is achieved when the person stands on one leg and raises the other leg at a right angle in front of them. To do this, check that the Y value of one knee is at least 40 pixels above that of the other. In addition to standing on one leg, it is also necessary to check how straight the person is standing. To do this, the height difference between the two shoulder landmarks is calculated. If the difference is less than 30 pixels and the person is standing on one leg, the balance position is reached and the balance score is counted up for each second. If the difference between the shoulders is greater, this is counted as an error. The activity value is also calculated on the basis of these two characteristics.

4. Calf Training – During calf training, the person also stands sideways to the camera and the direction of view must be determined accordingly. A repetition is counted when the person has pushed themselves up standing on their toes and then goes down again. It is counted as up if the difference between the Y value of the tip of the toe and that of the verse is greater than 35. It counts as down if the difference is less than 10. The time between repetitions is also measured here and an activity value is calculated from this.

5. Lunges – Even with lunge steps, the person only stands to the side of the camera, which is why the direction of gaze must first be recognized. A repetition is counted when the person has reached the lower lunge position and then stands up again.

Figure 2:

Visualization of the rehabilitation exercises.

While the exercise is being performed, the image from the front camera is displayed, allowing users to see themselves. The time remaining for the exercise is displayed in the top right corner. An exercise-specific progress value is displayed in the bottom right corner. For most exercises this is the number of repetitions, for the balance exercise it is a score depending on how long and how straight you stand on one leg. As the user stands to the side of the display for most exercises, the remaining time is also announced by voice output with 1 min remaining and the last 5 s. While the exercises are being performed, the user’s movement is evaluated in real time and the performance is assessed based on exercise-specific criteria and incorrect movement patterns are also recognized. If the exercise is performed incorrectly, the user is notified by voice message. As soon as the time is up, the results screen is called up.

6.3 Training schedule and tutorial

After logging in, the five exercises and the level of difficulty can be selected. Users can choose between the difficulty levels “Normal” and “Difficult”. “Normal” is selected by default. The selection is indicated by a change in the background colour and a green border. Pressing on an exercise starts it. The training schedule was designed with respect to needs, request and recommendations from the therapists and practitioners in the pre-study that were conducted and incorporated as final digital training activities. The exercises can be made more difficult or simplified with aids. This means that two levels of difficulty were offered and consideration can be given to patients with restricted mobility. The difficulty level can be adjusted according to the patient’s performance and progress. This approach allows to maintain, advance, or reset the difficulty level without sending any discouraging notifications to the player. Additionally, the system enables users to revisit previously completed levels, making it adaptable to their daily condition or desire to replay their favorite exercises.

Once you have selected an exercise, an self-created explanatory videos matching the selected exercise appears, as illustrated in Figure 3. In the video, the exercise is demonstrated from two perspectives. At the same time, the exercise is explained verbally. The instructions depend on the previously selected level of difficulty. The video is played in a loop until the “Start exercise” button is pressed. A countdown then starts, during which the person operating the device has time to get into the starting position. The countdown counts down from 10 and is shown on the display as well as counted down by voice announcement. In addition, the screen is set to a 5° angle so that the user’s entire body can be better recognized.

Figure 3:

Left: user interface displaying the exercises in German for users to select; Right: tutorial video.

6.4 Results and leaderboard

In the results section the user is shown a rating of between one and five stars for their exercise and is used to monitor their performance. The results are also saved permanently together with the ID and the time the exercise was performed. The calculation of the rating depends on the type of exercise and is usually based on the time of the exercise, the exercise-specific value (repetitions etc.) and the number of errors. The exercise-specific value and the number of errors are therefore displayed separately in addition to the score. By pressing the “Continue” button, the app starts again from the beginning (login). Alternatively, the ranking list can be called up beforehand. All saved executions of the specific exercise by all users are displayed in the ranking list. The “best” exercises are displayed at the top of the ranking list. All entries of the currently logged in user are highlighted in colour for better recognition.

6.5 Pose detection

The pose detection module is a crucial component that utilizes Google ML Kit’s pose detection API to recognize and track the user’s body movements. The ML Kit can identify 33 distinct body points, each assigned X and Y coordinates, which allows for precise tracking of body posture and motion. The Google ML kit pose detection was therefore used instead in this study to count repetitions and detect errors. ⁷³ Various predefined body points, so-called landmarks, are recognized and located on a two- or three-dimensional coordinate system. However, for this study, a two-dimensional approach was chosen, using only the height (Y) and width (X) coordinates, while the depth (Z) coordinates were ignored. This decision was made because a two-dimensional approach is computationally less intensive and sufficiently accurate for the purpose of counting repetitions and detecting errors in exercises.

6.6 Music-based control and feedback

The integration of adaptive music feature was identified and designed to enhance the user’s motivation and create a stimulating training environment. We integrated a software called JYMMiN ⁷⁴ into our system to combine the intensity and adapt to users movement of the exercises with music. JYMMiN is an innovative technology and training concept that integrates musically expressive performance with physical exercise. ⁷⁵ The music adaptation logic is based on the analysis of movement data. The music software is connected to the system via a REST API, allowing real-time communication and adaptation based on the user’s detected movements. For instance, if the user is performing high-intensity exercises, the system would increase the level of arousal of the music to match the pace. Conversely, for slower exercises or cooldown periods, the music may become softer and more relaxed. The user interface can be seen in Figure 4. Users can choose from nine different playlists, which are sorted according to themes such as “Around the world” or music genres such as “Classical” (1). Individual songs can be played in a continuous loop or otherwise the entire playlist (2). There is also a forward and back button to navigate through the playlist (3). The current song title is displayed at the top center (4) and the music can be muted (5). The current activity level and progression is displayed graphically using an arrow on a time scale (6). The level of difficulty can be set using the activity scale (7).

Figure 4:

High activity. (a) Low activity. (b) Tablet.

We combined the music-based software with the rehabilitation robot to detect and translate patients’ movements into dynamic music feedback. The system uses an activity value – a numerical score ranging from 0 to 100 – calculated based on the patient’s movement during exercises in front of the camera. This value determines the complexity and intensity of the music. As the activity value increases (when performing exercises correctly, which is detected by the pose detection), additional musical layers or instruments are added, creating a richer and more voluminous sound. At lower activity values (when the activities are performed incorrectly), an equalizer muffles the sound to provide less stimulating feedback. The system is adaptable: for example, an activity scale of 10 requires the maximum activity value of 100 to achieve the full musical experience, whereas an activity scale of 5 requires only a value of 50. The music-based software and exercise application operate on separate devices and communicate through an interface. For the current implementation, only the transmission of the activity value is needed. This value is continuously calculated during the exercise to provide real-time audio feedback reflecting the user’s performance. The system is designed to not only provide movement feedback but also support and initiate the positive effects of music in rehabilitation. However, the interface could be expanded in the future to integrate new functions. The programming interface was developed in C++ according to the REST principle. It therefore enables a C#-based Windows application such as our music-based software to communicate with our Java-based Android application. The system’s data flow is designed to be seamless, with low latency to ensure real-time feedback and interaction. The camera captures video frames, which are processed by the pose detection module. The movement data is then sent via the Android app to the music software using the REST API. The music software adapts the audio output and sends feedback to the user via the speakers of the robot, closing the loop.