Enhancing communication with elderly and stroke patients based on sign-gesture translation via audio-visual avatars

1 Introduction

The problem of communication between older people with stroke and people assigned to take care of them is a major challenge. When people suffer from speech difficulty due to stroke, they are not able to communicate effectively, and this can lead to their isolation and increase their feelings of isolation and depression. This situation requires a deep understanding of the situation and patience, as well as the use of alternative means of communication such as body language and gestures to facilitate communication and improve the quality of care and interaction between them and the people taking care of them.

Due to the lack of data on gesture signs for elderly people who have stroke in Iraq, it was necessary to create a dataset. Also, some signals are similar, such as “I want to eat” and “I want to change my clothes,” “Hello” and “Finished,” and “What” and “Why,” causing confusion in interpreting them. YOLOv8n has successfully addressed all these challenges except for the “Why” gesture. MobileNetV1 for classification, combined with U-Net for segmentation, have been used to overcome the difficulty in classifying the “Why” gesture that YOLOv8n faced.

Virtual reality offers interest in several technical areas [1]. Avatar is a digital representation of a user or their character [2], known as a digital human [3], or a virtual human [4]. There are a number of intelligent platforms that have been utilized to generate a life model like a virtual avatar, imitating a patient with which training doctors have the ability to communicate with it; Convai is an example of one of those platforms [5]. In addition, toolkit libraries and tools called XFace are used for animation, face recognition, speech generation, facial expression analysis, and other interactive functions. This technology is also utilized in building virtual avatars [6]. The challenges faced by the elderly who find it difficult to speak or hear are multifaceted, affecting their ability to communicate and fulfill their daily needs [7,8]. Avatars allow users to interact socially, work, play, and express themselves [9]. Common communication problems in seniors include hearing loss, stuttering, weakened facial muscles, and inability to write. These issues can be caused by various health problems, such as stroke [10]. Generating avatars using available platforms and tools has several challenges, such as avatar animation, realistic appearance, cost of creation, privacy and security, and system integration. On the other hand, deep learning models assist in solving some of these issues. “Generating Audio-Visual Avatar by Improved First-Order Motion Model” refers to using an enhanced version of the first-order motion model to create an animated representation that combines both auditory and visual elements. The first-order motion model is a framework used for transferring motion between video sequences, initially developed for applications like face and image reenactment. Improving this model typically involves modifications or enhancements to increase the motion accuracy or enhance the overall quality of the results. In this work, an audio-visual avatar is generated using the improved first-order motion model, which translates the sign gestures and converts them into written and audio text, depending on the results of YOLOv8 and MobileNetV1.

YOLOv8n is used for detection and classification; the traditional U-Net, U-Net with VGG16, and U-Net with MobileNetV2 are used for segmentation; and MobileNetV1 is used for classification. Based on the results of classification, the first-order motion model has been used to generate an interactive avatar that can speak English and Arabic and translate the meaning of the sign gesture motions into text and speech. The YOLOv8n model achieved values of 0.956, 0.939, and 0.971 for precision, recall, and mAP50, respectively, for detecting and classifying gestures. MobileNetV1 gained values of 0.94, 0.94, and 0.94 for accuracy, precision, and recall, respectively, for classification. Traditional U-Net for segmentation got 0.9845 as accuracy and 0.918 as the Dice coefficient. U-Net with VGG16 for segmentation achieved 0.9809 in accuracy and 0.889 in Dice coefficient. U-Net with MobileNetV2 for segmentation obtained 0.9478, 0.953, 0.9765, and 0.8034 as accuracy, Dice coefficient precision, and recall, respectively. The generated audio-visual avatar’s performance was 85.56%.

This study makes a unique contribution by the following ways:

• constructing a dataset of hand sign gestures specifically tailored for elderly individuals afflicted by stroke and having trouble in speaking for Iraqi Nursing Homes.

• propose mask annotation employed for segmenting the hand gestures using u-net model, thresholding segmentation, K-mean clustering, Image and Imglab tool.

• developing an audio-visual avatar capable of converting them into written text and speech for improved communication.

• integrating advanced models such as YOLOv8 and U-Net with MobileNetV1 for recognizing and classifying sign gestures, which is a robust methodological approach.

This contribution advances assistive communication technologies and provides a significant contribution to the field of elderly and stroke patient care.

The results show that the proposed avatar achieved excellent results, as illustrated in Section 4.2, Quantitative analysis. The remaining parts of this research work are structured as follows: The literature review is summarized in Section 2, the technique is discussed in more detail in Section 3, the results and discussion are presented in Section 4, and the conclusions are presented in Section 5 along with some suggestions for future study directions.

2 Literature review

An audio-visual avatar is a virtual person that blends visual and audible signs to improve human–computer interaction and the user experience [11,12]. Recent advancements in computer vision, artificial intelligence, and natural language processing have generated a lot of interest in the creation of visual and auditory avatars [13]. In addition to exploring the most recent developments and applications in this area, this section reviews the literature that has already been written on the production of audio-visual avatars.

In 2024, Zhang et al. [14] introduced Virbo, an intelligent talking avatar video generation system. It offers personalized functions, multilingual customization, voice cloning, face swapping, talking avatar dubbing, and visual special effect rendering. Virbo generates photo-realistic, lip-synchronized videos with better accuracy and authenticity. It can create videos comparable to professional creations. Future work aims to enhance speaker’s voice emotions and facial expressions. In 2024, López et al. [15] developed a novel hand gesture recognition (HGR) model using electromyography (EMG) signals and spectrograms. They compared the performance of convolutional neural network-long short-term memory (CNN-LSTM) and a post-processing algorithm. The results showed that memory cells improved the recognition accuracy by 3.29% compared to CNN models. However, post-processing had a more significant impact on recognition accuracy than memory cells via LSTM networks. This suggests that incorporating post-processing algorithms can enhance HGR models’ accuracy and robustness against EMG signal variability.

In 2021, García et al. [16] presented a description of an avatar production system, including the key technological specifications for the design of avatars that experience audio hallucinations, and an assessment of the system from the perspectives of both patients and therapists. Character Creator, Poser, Unity Multipurpose Avatar, and Adobe Fuse CC were all utilized for the avatar creation process. In 2023, Lu et al. [17] used a convenience sampling method to select 13 participants from a support group for senior citizens in the southern part of Ireland with hearing loss. Participants were interviewed in a semi-structured manner. NVivo 12 was used to audio-record interviews and transcribe the results. Themes related to the challenges faced in recent healthcare interactions and recommendations for improving comprehensive healthcare communication were identified using Clarke and Braun s thematic analysis technique. In 2022, Zhang et al. [18] introduced a 3D animated high-fidelity human model. However, it has some limitations: (1) it is based on skinned multi-person linear (SMPL) projections, and (2) it cannot capture extremely fine actions, such as facial emotion variations. In 2022, Athira et al. [19] suggested that signers use a cutting-edge vision-based movement recognition system that can recognize single-handed, dynamic, double-handed, and, using live video, finger-spell phrases in ISL (Indian sign language). For classification, they used support vector machine (SVM). The accuracy recognition for single-handed dynamic gesture was 89%, and for fingerspelling gestures it was 91%. In 2021, Li et al. [20] showed that a pipeline creates a 3D human form avatar from a single RGB photograph, producing a texture map for the entire body and three-dimensional human geometry. The technique separates the human body into its component components, fits them to a parametric model, and warps them into the desired shape. From the frontal photos, InferGAN infers the unseen back texture. Using MoCap data, it is simple to rig and animate their human avatars. A mobile application demonstrates the effectiveness of the solution for AR applications, showing its reliability and efficiency for both public and private datasets. In 2021, Sharma and Singh [21] proposed a CNN model for identifying gesture-based sign language that is based on deep learning. The model outperforms conventional CNN architectures in classification accuracy while using fewer parameters. In the ISL and American sign language (ASL) datasets, during training and testing, the VGG-11 and VGG-16 models achieved the highest accuracy of 99.96 and 100%, respectively. The model surpassed other strategies, according to experimental assessments, identifying the most gestures with the least amount of error. Rotation and scaling transformations had no effect on the model. In 2020, Thies et al. [22] introduced a pioneering technique for audio-based facial re-enactment that can be applied across diverse audio sources, enabling us to produce a speaking head video from an audio sequence of another individual as well as to generate realistic videos constructed with a synthetic voice. This implies that it is possible to create text-driven video synthesis with synchronized artificial voices. In 2020, Gupta and Rajan [23] employed inertial measurement data from categorization of continuously signed words from the ISL, which was carried out using an accelerometer and a gyroscope. A modified time-LeNet architecture was provided along with the use of time-LeNet and multichannel deep CNN (MC-DCNN) to address over-fitting. The models were compared for complexity, loss, and classification precision. Time-LeNet obtained 79.70% accuracy compared to MC-DCNN’s 83.94%. In 2019, Molano et al. [24] proposed the Candide parametric mask, a method for producing emotive avatars in runtime, speeding up 3D animation. It produced a variety of emotions, ranging from straightforward winks to complicated ones, and was inspired by the Ekaman emotional model. In 2019, García et al. [25] presented a scheme for creating treatments based on avatars. The strategy was based on a previously described tool to improve social cognition in people with cognitive impairment. First, the criteria for facial emotion identification in avatar-based treatments were developed. The supporting instrument for the therapy was then presented with a brief description. The administration of treatments has since been explained. For both the clinician and the patient, treatment execution was separated into pre-therapy and treatment. Table 1 presents a comparison between our generated avatar and those generated by earlier studies that generate an avatar that is required by a certain activity. Table 2 presents a comparison between our suggested approaches for classifying gestures for the elderly and patients with stroke who are unable to speak and earlier research that has classified sign language.

When comparing our work with previous studies, we found the following: the work of Zhang et al. [14] used a speaking avatar to build an intelligent system. The work of García et al. [16] used an avatar to represent auditory hallucinations. The work of Zhang et al. [18] featured an avatar of a human wearing three-dimensional clothing. The work of Li et al. [20] created an avatar for public use. In our work, we used avatars to facilitate communication between the patient and the doctor. Regarding the technology used to create the avatar in each study, the work of Zhang et al. [14] used Virbo. The work of García et al. [16] utilized Adobe Fuse CC, Mixamo, Autodesk 3D Studio Max, and Unity. The work of Zhang et al. [18] employed a 3D parametric human model (SMPL) and a deformation network. The work of Li et al. [20] used InferGAN. In that work, the authors improved the performance of the first-order motion model by enhancing the input image to create a more realistic avatar. The authors also proposed a multi-stage image segmentation method to enhance the accuracy of classification.

3 Method and implementation

Figure 1 depicts the schematic of the proposed sign gesture interpreter system to serve the elderly and stroke patients having special needs and provide assistance to those who take care of them:

Figure 1

Schematic of the suggested system.

The system operates bidirectionally (from the patient to doctor and vice versa), but in this research, we have emphasized on the forward phase.

3.1 Gesture hand dataset preprocessing

The performance of the proposed communication enhancement system for older people with special needs, specifically those with speech difficulties, was evaluated using constructed datasets. This was necessary because there is a lack of standard and publicly accessible datasets for individuals who are neither mute nor deaf. This constructed dataset consists of a gesture-hand description of specific things that elders need for daily life requirements. The dataset was created by visiting many nursing homes for the elderly and conducting interviews with both the residents and those in charge of their care. After settling on gestures and movements that meet the needs of the elderly, dataset images were acquired for these movements, and a group of people were used to represent them. The proposed dataset contained 26 classes; each class had 110 images of different samples. Image dimensions were 640 × 640. The proposed dataset included images from a diverse collection of people of various ages and genders, with various backgrounds and orientations based on the elderly and stroke life needs. The dataset was named “Sign Gestures for Elderly and Stroke Patients” (SGESP). The images were captured using web and mobile cameras, and the dataset included hand gestures for Iraqi elderly and stroke patients, as illustrated in Table 3.

Table 3

Signs for elderly people constructed dataset and their meanings

The dataset was prepared for both semantic segmentation and instance segmentation. The dataset was annotated in two ways to be appropriate, for instance, segmentation and semantic segmentation: bounding box annotation (manual labeling) and masking images (using manual labeling methods [JSON file], K-mean clustering, U-Net, and threshold segmentation). The dataset had augmented × 4 images [horizontal flipping, 5% rotation, 3% blurring, and 2% noising].

In this work, preprocessing included cropping, resizing, and converting images to JPG. Manual cropping was performed to remove areas that were uninteresting in the image, which helped focus the main content. Antialiasing technology was also used during the resizing process. These visual processes enhanced the effectiveness of pre-processing to improve the readiness of images for use in visual analysis or the training of artificial intelligence models. All the dataset images were resized to 640 × 640 pixels to ensure reliable model training. The total number of datasets for the original images was 2,860; it increased to 10.808 after augmentation. The proposed dataset was divided into 70% training, 20% validation, and 10% testing. The hold-out method was used for training. About 10.010 images were used for training, 512 images were used for validation, and 286 images were used for testing. The dataset contained 10.808 text files as annotation labels.

3.1.1 Proposed multistage image segmentation method

The first stage of the proposed method was segmenting the entire preprocessed dataset using segmentation by thresholding in order to produce an image dataset labeled for semantic annotations. About 10% of dataset images were accurately segmented. In the next stage, K-mean clustering was applied to 90% of remaining dataset images, and about 30% of the resultant images had accurate segmentation. The third stage was applying the ImgLab tool to the remaining images (60%) in order to get a JSON file, which was then converted into a segmented image. About 10% of the resultant images had accurate segmentation. Then, we had about 50% correctly segmented images, and they were used to train U-Net to obtain the best weight, while the remaining 50% which were not segmented correctly were used for testing. The result of U-Net was about 100% correctly segmented image, which can be used as a mask for semantic annotations.

Algorithm 3.3: Annotating sign gesture dataset for semantic annotations
Input: Proposed dataset images
Output: Images of dataset annotated for semantic annotations
Begin:
Step 1: For i = 1 to 26 Do//to login to every dataset folder
Step 2: Segmentation by thresholding:
	For j = 1 to 110 Do//to read every dataset image in each folder
		- Img = Read(image)
		- Segment image by thresholding segmentation using equation (2)
		- If the segmented region of hand gesture is accurate, then
			- save the selected image in the gesture-mask-folder
			- delete the selected image from the original dataset
	End for j loop.
Step 3: Segmentation by K-mean clustering:
	For j = 1 to 110 Do//to read every remaining image in each folder
		- Img = Read(image)
		- Segment image by K-mean clustering using algorithm (2)
		- If the segmented region of hand gesture is accurate, then
			- append the selected image to the gesture-mask-folder
			- delete the selected image from the original dataset
	End for j loop.
Step 4: Segmentation by ImgLab:
	For j = 1 to 110 Do//to read every remaining image in each folder
		- Img = Read(image)
		- Segment image by ImgLab tool segmentation to get the JSON file
		- convert the JSON file to segmented images
		- if the segmented region of hand gesture is accurate, then
			- append the selected image to the gesture-mask-folder
			- delete the selected image from the original dataset
	End for j loop.
Step 5: Semantic Segmentation using U-Net:
	For j = 1 to 110 Do//to read every remaining image in each folder
		- Img = Read(image)
		- Train U-Net using the resultant gesture-mask-folder
		- Obtain the best weight
		- Test the images remaining in the original dataset using the already trained U-Net//image which has not been selected
		- Append the segmented images to the gesture-mask-folder
		- Obtain the final segmented image using U-Net
	End for j loop.
Return the mask image for each image in the dataset
End

4 Results and discussion

The results are discussed based on the experimental setting and quantitative analysis.

4.2 Quantitative analysis

The results of YOLOv8 were evaluated using benchmark performance metrics. Recall, precision, F1 measure, and mean average precision from epoch 0 to epoch 195 are illustrated in Table 5 and Figure 2. Epoch 146 shows the best results.

Table 5

Comparison of the results of YOLOv8 from epoch 0 to epoch 195

Epoch	Train/box loss	Train/cls_loss	Train/dfl_loss	Metrics/precision(B)	Metrics/recall(B)	Metrics/mAP50(B)	Metrics/mAP50-95(B)	Val/box loss	Val/cls loss	Val/dfl loss
0	1.1778	4.0693	1.5151	0.25686	0.33499	0.23295	0.15871	1.1288	3.0318	1.4391
1	1.1586	2.9944	1.4688	0.4818	0.46913	0.46801	0.31006	1.2319	2.3013	1.5262
2	1.1802	2.6073	1.4557	0.47785	0.51797	0.52939	0.32774	1.3268	2.2235	1.6005
3	1.2184	2.384	1.4543	0.58504	0.6689	0.70701	0.43662	1.3896	1.5356	1.5991
:	:	:	:	:	:	:	:	:	:	:
:	:	:	:	:	:	:	:	:	:	:
146	0.74929	0.66212	1.1051	0.95499	0.9406	0.97009	0.77275	0.86865	0.4348	1.125
:	:	:	:	:	:	:	:	:	:	:
:	:	:	:	:	:	:	:	:	:	:
193	0.72875	0.63179	1.0947	0.95444	0.94036	0.96907	0.77275	0.86746	0.4248	1.1259
194	0.71511	0.61902	1.0884	0.95463	0.94033	0.96829	0.77179	0.8674	0.42481	1.1258
195	0.71785	0.6283	1.0931	0.95473	0.94035	0.96954	0.7725	0.86727	0.42433	1.1258

Bold values represent the best training and validation performance of the model, which likely corresponds to the optimal epoch (146) for selecting the best weight and final model.

Figure 2

Results of YOLOv8.

Figure 2 illustrates the values of these metrics for different training iterations. It appears that the training loss and classification loss decrease over time, which suggests that the model is learning to better predict bounding boxes and classify objects. The precision and recall metrics also increased, which suggests that the model is making better detections.

Moreover, the confusion matrix was calculated, as illustrated in Figure 3, where classes “Hello,” “I am good,” “Wait,” “I want to eat,” “Finished,” “What,” “I am married,” “I want to change my clothes,” “I cannot hear,” “Stop,” “Listen to me,” “Please,” “I love you,” “It is time,” and “Help me” achieved perfect results (1.00). While classes “Thank you,” “Correct,” “I want to drink water,” “Upset,” “Come,” and “Please” got excellent results with a range of 91 to 95. As for the classes “I want to take a shower,” “I am not sure,” and “Me” got a very good result in the range between 80 and 89. On the other hand, the class “You” obtained 0.73 and the class “Why” achieved 0.59.

Figure 3

Confusion matrix.

U-Nets’ segmentation assessment matrices for each class in Google Colab are presented in Table 6. MobileNetV1 and MobileNetV2 classification assessment matrices are presented in Table 7. MobileNetV1 classification evaluation matrices for each class in Google Colab are shown in Table 8.

Table 6

U-Net’s segmentation assessment matrices for each class in Google Colab

Model	Accuracy	Dice coefficient
U-Net for segmentation	0.9845	0.918
U-Net with VGG16 for segmentation	0.9809	0.889
U-Net with MobileNet2 for segmentation	0.9478	0.953

Table 7

MobileNetV1 and MobileNetV1 classification assessment matrices

Model	Accuracy	Precision	Recall
MobileNetV1 for classification	0.94	0.94	0.94
MobileNetV2 for classification	0.79	0.76	0.71

Table 8

MobileNetV1 classification assessment matrices for each class in Google Colab Pro

Class	Precision	Recall	F1-score	Support
Hello	1.00	0.97	0.99	40
I am good	0.97	0.85	0.91	40
Thank you	0.97	0.97	0.97	40
I want to take a shower	0.90	0.93	0.91	40
Wait	0.97	0.97	0.97	40
Correct	0.90	0.93	0.91	40
I want to eat	0.93	0.93	0.93	40
Finished	1.00	1.00	1.00	40
I cannot hear	0.87	0.97	0.92	40
I want to change my clothes	1.00	0.93	0.96	40
I want to drink water	0.97	0.88	0.92	40
Me	0.86	0.90	0.88	40
You	0.88	0.88	0.88	40
Upset	0.91	0.97	0.94	40
Come	0.95	0.95	0.95	40
I am not sure	0.88	0.93	0.90	40
I am married	0.95	0.95	0.95	40
What	1.00	0.97	0.99	40
Why	0.86	0.90	0.88	40
Excuse me	0.90	0.90	0.90	40
I love you	1.00	0.97	0.99	40
Please	0.93	1.00	0.96	40
Listen to me	0.97	0.95	0.96	40
Accuracy			0.94	1,040
Macro avg.	0.94	0.94	0.94	1,040
Weighted avg.	0.94	0.94	0.94	1,040

The video avatar was generated using a driving video and a Pacific image based on the first-order motion model, as shown in Figure 4(a), and then audio and text were embedded in the video avatar to generate the audio-visual avatar, as shown in Figure 4(b).

Figure 4

Generate audio-visual avatar: (a) generate Avatar and (b) generate audio-visual avatar after embedding audio and text to video avatar.

It can be concluded that when the proposed system used YOLOv8 and MobileNetV1 for the purpose of classification, YOLOv8 gave better results than MobileNetV1 in classifying hand gesture signs. Also, when the proposed system used traditional U-Net, U-Net with VGG16, and U-Net with MobileNetV2 for segmentation purposes, the traditional U-Net gave the best result in segmentation hand gesture signs. The results of detection and classification stages, as well as segmentation and classification stages, were evaluated using several images, video samples, and several DNN models.

For detecting and classifying hand sign gestures, the YOLOv8 model was used, in which 3,010,718 total parameters were utilized. The benchmark evaluation matrices gave precision, recall, mAP50, and mAP50-95 values of 0.956, 0.939, 0.971, and 0.775%, respectively, on Google Colaboratory (Google Colab Pro) with 25 GB RAM and 200 GB storage.

In order to segment the hand sign gestures, the proposed system used the traditional U-Net, U-Net with VGG16, and U-Net with MobileNet2 models for semantic segmentation. The system utilized (1.941.105), (16.195.617), and (416,209) parameters, respectively, on Google Colaboratory (Google Colab Pro) with 25GB RAM and 200GB storage. Accuracy and Dice coefficient values were used to evaluate the segmenting, which gave (0.9845 and 0.918) for the traditional U-Net model, (0.9809 and 0.889) for the U-Net with the VGG16 model, and (0.9478 and 0.953) for the U-Net with MobileNetV2 model.

In order to classify the segmented hand sign gestures, MobileNetV1 was used, in which 3,296,154 parameters were utilized. The evaluated metrics used were accuracy, precision, and recall, which gave 0.94, 0.94, and 0.94, respectively.

The improved audio-visual avatar, which was used to translate elderly hand sign gestures, achieved an effective performance of 85.56%, with 25 volunteers answering 15 pre-prepared questions, indicating its effectiveness in providing care to elderly individuals.

The evaluation metrics indicate that the work of Sharma and Singh [21] outperformed all studies due to their design using a deep neural network (CNN), which led to high accuracy in classifying ISL and ASL gestures. This model outperformed VGG-11 and VGG-16, demonstrating the ability to classify with high accuracy and a low error rate. Additionally, it was stable against transformations in rotation and scaling, as demonstrated in Table 9. However, our proposed system is superior to previous studies because it includes an avatar to enhance communication between the patient and the doctor. In our research, we constructed a dataset specifically designed for elderly individuals with stroke in Iraq. This dataset includes fundamental hand motions identified by visiting several nursing homes for the elderly to meet their daily needs, filling a gap where no such dataset previously existed in Iraq.

Table 9

Comparison of height performance results between previous research and the proposed method

Study	Accuracy (%)
López et al. [15]	90.55
Athira et al. [19]	89 and 91
Sharma and Singh [21]	100 and 99.96
Gupta and Rajan [23]	83.94, 79.70, and 81.62
Our proposed system	94

5 Conclusions and future scope

In conclusion, this research has successfully achieved its main objective of developing an effective deep learning-based system for translating elderly gestures into audio and text using avatars. The creation and utilization of the SGESP dataset highlight the importance of incorporating real-life scenarios and contexts into gesture recognition systems. Segmenting hand motions using multi-stage segmentation proved to be 100% effective. Some signals, such as “I want to eat” and “I want to change my clothes,” “Hello” and “Finished,” and “What” and “Why,” are similar and cause confusion for the system. YOLOv8n successfully addressed all these challenges except for the “Why” gesture. In order to solve this confusion, U-Net was used for the segmentation process. MobileNetV1 was then used for classification to prevent the confusion between “What” and “Why” that appeared when using YOLOv8 for classification purposes.