Read for me: developing a mobile based application for both visually impaired and illiterate users to tackle reading challenge

1 Introduction

Worldwide, there are still over 750 illiterate adults [1] and more than 300 million people who are blind or have moderate-to-severe visual impairment [2]. These people are facing daily challenges to do basic tasks, such as recognizing objects, navigating places, and reading texts that can be found in books, street signs, restaurant menus, and grocery products. However, the advancements in technology, especially the revolution of mobile devices and their applications, play a huge role in tackling some of these problems and can enhance the way these groups of people communicate with society and the world. For example, with mobile-based applications, visually impaired and illiterate people can make phone calls, receive and send messages, play games [3], and learn from mobile-based applications (apps) [4]. This confirms that people who have a visual handicap or are illiterate could use modern technology more efficiently by utilizing a variety of applications. They could also benefit from smartphones’ advanced functions rather than only making phone calls and sending text messages.

As a result, several researchers were encouraged to adapt different technologies and develop information and communication technology (ICT)-based applications that help visually impaired and illiterate people to overcome their reading limitations. For example [5], presented Raspberry Pi and a camera system to help Bangla-speaking people to read documents, whereas [11] proposed a system built from Raspberry Pi, the phone’s camera, a push switch, and earphones to help blind people to read text in their surroundings, and finally [8], developed a software application to help illiterate and visually impaired people to fill in banking forms without assistance using optical character recognition (OCR) technology and a speech interface.

Obviously, most of these studies avoid using smartphones when building a system for the blind in particular because of the difficulties that blind people face while interacting with touchscreens [12]. However, developing a mobile-based application makes it more accessible than other types of systems, as the statistics have reported that the usage of smartphones is about 43 % with a significant percentage from developing countries [13], which also have the major number of blind and illiterate people from around the world [1,2].

Therefore, the aim of this paper was to develop a mobile-based application that would be accessible to both visually impaired and illiterate people and could help them to read texts that can be found on different surfaces, such as fiber, paper, and plastic. The built app uses the smartphone camera with OCR technology to capture the texts. It also uses Firebase for real time translation to the local language in case the captured text is in English, and TTS (Text To Speech) for reading the text aloud. The app was designed according to users’ design preferences, which were collected through conducting a requirements elicitation study. Then, the reader app was developed and given to blind and illiterate users to evaluate it. The evaluation sessions showed that the users could efficiently interact with and were satisfied using smartphone applications, which are designed according to their requirements and developed to help them to take advantage of the available technology.

2 Previous works

This section provides a concise overview of a number of previous studies that were conducted to help visually impaired or illiterate people take advantage of using technology in their usual activities. Table 1 shows the summary of the prevoius works.

Firstly, as mentioned previously [5], proposed a system to help people who speak the Bangla language and are unable to read or have a significant loss of sight to read text documents. The system was built using Raspberry Pi, with a camera module for utilizing the OCR engine, computer vision techniques, and the Google Speech Application Program Interface. In this research, the accuracy for Bangla text images was 97.4 %. However, at night, the accuracy of the extraction of the text does not reach this level, as the quality of the captured image is not perfect in low light conditions. Another Raspberry Pi based system was developed by [6]. This system recognizes texts in images then generates vocal speech that the users could hear. In addition to reading, developers of this system utilized ultrasonic sensors to keep a safe distance between the objects and the visually impaired. Authors in [7] also used ultrasonic sensors and smart glasses with camera to recognize text and obstacles in the roads. Two-branch exposure-fusion network was also used to enhance low-light images and aided the system to recognize texts in the dark conditions.

Furthermore, the authors in [8] developed a software application that enables illiterate and visually impaired people to fill in bank forms efficiently and without outside assistance. The system was based on a speech interface for users’ inputs and OCR technology for data retrieval from the paper documents.

A more recent system was produced by [9] to make newspapers available on a daily basis to those who are illiterate or have a visual handicap. The system is a mobile-based system that converts newspaper images into audio using character recognition, artificial intelligence, and computer vision. Another OCR-based reader system was developed by [10]. This mobile-based application targeted visually impaired people and explained the possibility of interaction between them and smartphones. During the testing phase, the author addressed several issues inherent in the interaction between blind people and smartphones, such as the level of luminosity, the weather, and positioning the camera at the optimal distance. Similar interaction issues were addressed by [12] regarding blind people using smartphones with the additional issue of using a touchscreen rather than physical keys, which makes the interaction process more challenging for these people in particular. A solution for this problem was suggested by [11] using a push switch to act as a button to capture images on their system, which was developed to reduce the dependency of the visually impaired on other people to recognize texts in their surroundings. As mentioned above, the authors in this research built a system that consisted of Raspberry Pi, a camera, a push switch, and earphones. This means they did not utilize and benefit from the services offered by mobile devices, which are owned by a significant number of people who might use them only for basic functions, such as making or receiving calls [13]. As a result, the visually impaired might buy a new device to help them read text, while they could actually use their phone to do this task. Thus, in this study, we aimed to develop a mobile-based system that encourages illiterate and visually impaired people to use their smartphones for more advanced functions and help them to read texts without assistance. Moreover, this research mainly attempts to discover the possibility of building an application that could be used by both illiterates and blinds. Then find out whether they would use it with same efficiency. This was achieved through conducting two studies. The first one was carried out to identify the design requirements of visually impaired and illiterate users regarding the intended application. Then, the users’ requirements and design considerations identified by this study were applied to the new developed application. Next, an evaluation study was conducted with both groups of users (blind or illiterate) to measure their performance while using the app and gather their opinions about it.

Furthermore, none of the previously mentioned studies described the users’ requirements or system design features for the best interaction experience, although these systems were mainly devised for the blind or illiterate for reading purposes. Moreover, the earlier studies did not measure the users’ performance or collect their feedback on these systems. Therefore, this study attempts to fill this gap and give a deep understanding of both visually impaired and illiterate users’ needs and design requirements in a reader mobile-based application.

3 Methodological framework

This study was conducted following the five steps of Design Science Research (DSR) in an information system proposed by [14]. These steps are presented in Figure 1 and are explained as follows:

Figure 1:

The research methodology steps.

4 Design and development

4.1 Application design

Based on the design features suggested by the visually impaired and illiterate interviewees in the requirements elicitation study, the authors designed and developed a mobile application to serve as a text reader for illiterate or visually impaired people in Iraq. According to the participants’ preferences, the screens of the reader application should be primarily designed with simple and reduced content. The visually impaired participants in particular preferred few or no buttons to be included in the app and for them to be replaced with sounds. Similarly, illiterate participants expressed that they would rather have a simple interface based on sounds rather than texts, as this would make the app easy to use without others’ assistance. As a result, we developed a reader application that has a simple interface with only one big button placed at the bottom of the device screen. Moreover, the application provides a background audio that supports the users’ local language (Arabic) to ensure users receive instructions and guidance throughout their interaction with the application. Background audios were also used to provide feedback after every step the user takes. These features allow users to navigate the application effectively and reduce the number of errors that might occur while using the app. Figure 2 demonstrates the design framework of the application.

Figure 2:

The design framework for the reader application.

4.2 Application development

Several technologies and tools were utilized to realize the design’s conceptual framework and develop the proposed application. To begin with, Android Studio IDE was the main tool used to build the Android-based application. The backend of the app was coded in Java programming language, whereas the frontend used XML language. Firebase, a real time database, was also used in this app for translation purposes. If the captured text is in English, it is converted to Arabic then read aloud to the users in their native language. Another technology utilized in this application is OCR. This is a tool that extracts texts from images then transforms them into an editable text format. In addition, TTS technology was used to convert the extracted text into a spoken audio format. This technology supports over 100 languages including the Arabic language and different voice styles, such as male and female.

Once the user has launched the application, a vocal instruction is played to guide them on what to do next. The audio message is “Welcome to the ‘read for me’ app … To launch the phone camera, please press the button at the bottom of the screen”. When the user presses the button, the app launches the camera phone and simultaneously plays another vocal message which is “Please place your device facing the text within the distance of a hand span”. Then, the app captures the text and in the case that the extracted text is in English, translates it into Arabic. If the app does not capture the text, it gives an audio alarm saying that the process was not successful and that the user should try again. In the final step after capturing and extracting the text, the user hears the spoken text which is read aloud by the app. Figure 3 illustrates the flow of activities of the application.

Figure 3:

The flow chart for the application’s activities.

4.3 Implementation

The reader application was developed with one screen that contains only a single content, which is one big button at the bottom. The button takes up the entire bottom part of the screen, so wherever the user (especially blind people) presses, it will be on the required button. In addition, the entire application employs voice as user guidance and to give the user feedback after each step. Moreover, voice was used as an alarm to warn the user if the capturing process was not successful. All of these voices, which accompany every step of the application, are available in the local language (Arabic). The audio feature was added to the application to assist both illiterate and visually impaired users in dealing with their current situation and in understanding what necessary actions should be taken. When the user launches the app, he/she hears a 7-s audio that tells the user to go to the bottom of the screen and press the button. This launches the device camera. Along with launching the camera screen, another voice guide is played to inform the user that he/she should place the phone facing the required text. The voice also contains the required distance (hand span) between the text and the device for best capturing the text. If the text is captured and extracted, feedback is played to confirm the success of the process. There is also a sound that warns the user to try again in case the app has not captured the text. In addition, the app provides a real time translation for texts that are captured in English and transforms them into Arabic. At the end, the user hears the extracted text in their native language. Figure 4 shows the implementation of the reader application on different surfaces.

Figure 4:

Screens from the reader application after implementation.

5 Evaluation of the application

A usability evaluation study was conducted after building and testing the application. The aim of this step was to determine the effectiveness of the developed app and explore whether it has met both users’ requirements, which were identified in the requirements elicitation study. Moreover, during the evaluation sessions, the authors could observe the performance of both users (illiterate and visually impaired) and measure the differences between them. Lab-based sessions were carried out with a number of illiterate and visually impaired people to find out their impressions and gather their feedback while they were using the app.

5.3 Evaluation results

The data obtained from the evaluation sessions were analyzed to assess the usability of the developed application based on three distinct aspects: efficiency, effectiveness, and satisfaction [19]. Table 4 describes the findings from the evaluation sessions.

Table 4:

Summery of the evaluation study findings.

Evaluation metrics	Measuring indicator	Type of participants	Mean	SD
Efficiency	Task completion time	Visually impaired	46.28	±2.87
		Illiterate	33.21	±2.14
	Learning time: times assistance was required	Visually impaired	0.42	±0.53
		Illiterate	0.071	±0.26
	Learning time: time spent listening to the audio instructions	Visually impaired	0.42	±0.53
		Illiterate	0.107	±0.31
Effectiveness	Completion rate	Visually impaired	0.857	±0.37
		Illiterates	1	±0
	Error rate	Visually impaired	0.57	±0.53
		Illiterate	0.142	±0.35
Satisfaction	Overall users’ satisfaction with the application	Visually impaired	4.2	±0.95
		Illiterate	4.5	±0.69
	Simplicity and ease of use	Visually impaired	3.5	±0.53
		Illiterate	4.7	±0.46
	Willingness to use the app	Visually impaired	4.8	±0.37
		Illiterate	4.9	±0.26
	Recommendations to others	Visually impaired	4.5	±0.53
		Illiterate	4.8	±0.31

5.3.1 Efficiency

Efficiency refers to how quickly users can complete tasks once they have become familiar with the interface’s design. In other words, the system is considered efficient when no time is wasted [20]. Two metrics were utilized in this research to assess the efficiency of the reader application [19]. The first one was task completion time, which was measured by calculating the average time spent to complete the required task. The second metric was learning time, which includes two parts: the number of times the participant requested assistance, and the frequency with which the participant listened to the audio instructions.

5.3.1.1 Task completion time: the average time spent to complete the task

The system interface is considered efficient when it minimizes the time required to perform a task [21]. In this research, each participant was given the same task, which was reading a printed text using the developed app. The average time participants took to complete the task was 35.82 s. The standard deviation (SD) for the task was ±5.76. However, there was a distinctive difference between the time spent to accomplish the same task by visually impaired participants and illiterate participants, whereby the visually impaired spent much more time to complete the task than illiterate participants. The average time spent by blind participants was 46.28 s with an SD of ±2.87, while the average time spent by illiterate participants was 33.21 with an SD of ±2.14. Figure 6 shows the average time with SD values to complete the given task by visually impaired and illiterate participants. The minimum time spent to achieve the task by illiterate participants was 30 s while visually impaired participants took 43 s whereas the maximum time taken was 37 s and 51 s by illiterate and blind participants respectively. The number of participants who spent longer than the average time to finish the task was 8 (28.5 %) and 3 (42.85 %) for illiterate and visually impaired users respectively. These results confirmed that the majority of participants, whether they were illiterate or visually impaired, had used the app efficiently.

Figure 5:

Evaluating the app through lab study.

Figure 6:

The results of metrics used to measure the application’s efficiency. (a) Task completion time. (b) Number of times the participant requested assistance. (c) Number of times the participant listened to the audio instructions.

5.3.1.2 Learning time: the number of times the participant requested assistance

According to [22], efficiency is measured by the number of times the users ask for help while interacting with the system. As long as the users successfully perform the tasks without assistance, the interface is considered efficient. This study revealed that 3 out of 7 visually impaired participants required assistance to complete the task. In contrast, only 2 out of 28 illiterate participants needed help with how to perform the task. These results prove that the participants found the application’s user interface simple and easy to use. Additionally, the voice instructions, which accompany every step while using the app, made it easier for the participants to understand how to interact with the application and complete each task.

5.3.1.3 Learning time: the frequency of times the participant listened to the audio instructions

The developed application has an Arabic voice guide in every step to inform the user what to do next. An ideal user interface would be the one in which the participant listened to the vocal instructions just once, as having to listen repeatedly to the instructions would indicate a poorly designed user interface [22]. In this study and in order to do the reading task, 3 of the visually impaired participants (nearly 43 %) went back to listen to the voice guide provided by the app. Illiterate participants, in contrast, were more confident, as only 3 of 28 participants (about 11 %) repeated listening to the audio instructions. This means most of the illiterate and less than a half of visually impaired participants could successfully perform the reading task after listening to the voice instructions only once. These findings demonstrate that the voice guide was clear and simple enough so the participants could follow the instructions easily. Moreover, these audio instructions contributed significantly to the application’s efficiency by guiding the user on how to complete the task.

5.3.2 Effectiveness

Effectiveness was described by [19] as the degree of accuracy and completeness with which users accomplish specific goals. In this study, two fundamental indicators were considered to measure the effectiveness of the reader application: the completion rate and the error rate. Figure 7 illustrates the effectiveness mesurement of the application.

5.3.2.1 Completion rate

Also known as the success rate, this refers to the number of participants who managed to complete the given task. In order to calculate the completion rate for the application, we used the following equation:

Completion rate = N u m b e r o f t a s k s c o m p l e t e d s u c c e s s f u l l y Number of attempted tasks × 100 %

The results reported a high success rate between both visually impaired and illiterate participants, as 6 of the visually impaired participants (about 85.7 %), and all of the illiterate participants (100 %) were able to complete the reading task. This means that the application is completely effective for both groups of participants.

5.3.2.2 Error rate

Another primary method to measure the effectiveness of the system is by counting the mistakes or errors made by participants while trying to accomplish a specific task [23]. The below formula was used to calculate the error rate for this study:

Error rate = T h e n u m b e r o f e r r o r s o c c u r r e n c e T h e t o t a l n u m b e r o f a t t e m p t s × 100 %

In this experiment, 4 of the visually impaired participants (57.14 %) made mistakes during the reading session. The errors mostly occurred when the participant was attempting to capture the text. They miscalculated the distance that should be between the text and the device. Therefore, they repeated the process of capture after they had heard the warning feedback from the app, which states, “Capturing text was not successful”. Regarding the illiterate participants, they also recorded errors while using the app; 4 out of 28 participants (14.2 %) did not follow the voice instruction related to the hand span distance between the text and the device. Consequently, the device did not capture the text directly. However, the warning voice message provided by the app helped them to recover from this error and then complete the task properly. These results showed that, although the participants made errors while interacting with the app, the accompanying feedback with every step in the app certainly helped them to recover from these errors and accomplish their task.

5.3.3 User satisfaction

Satisfaction is defined as the level of comfort and positive attitudes of users toward the system [19]. User satisfaction can be measured through interviews, questionnaires, or attitude rating scales. In this study, we rated the users’ satisfaction with the developed application using a Likert scale that ranged from 1 (extremely disagree) to 5 (extremely agree). At the end of each session, the participants were asked four questions in order to express their opinions about the app. The first question was, “Overall, are you satisfied with the application?” The responses were distributed in a range from 1 (not satisfied at all) to 5 (very satisfied). The second question was, “Was the app simple and easy to use?” The responses ranged from 1 (not at all easy) to 5 (very easy). The third question was, “Would you use the reader app in the future?” The responses ranged from 1 (not at all willing) to 5 (very willing). The final question was, “Would you recommend this app to others?” The responses ranged from 1 (not recommend at all) to 5 (would strongly recommend). According to the hypothesis, satisfied users rated their average satisfaction as equal to or higher than 2.5, which represents the mean value for satisfaction on the Likert scale.

Overall, the study findings reported that the application rating given by the illiterate participants was higher than the rating given by the visually impaired over the four satisfaction questions. Furthermore, the most difference in the application rating between the two groups was noticed in question 2 “Was the app simple and easy to use?” Figure 8 illustrates the participants’ satisfaction rating on the application.

Figure 7:

The results of two metrics used to measure the application’s effectiveness. (a) Completion rate. (b) Error rate.

Figure 8:

Users’ satisfaction rating on the application.

Regarding question 1, both groups of participants were pleased and showed extreme satisfaction with the reader application; the average satisfaction rating was 4.2 and 4.5 for visually impaired and illiterate users respectively. They also expressed intensive interest in the application, and they were keen to use it to help them read texts (average 4.8 for visually impaired and 4.9 for illiterate users). Moreover, illiterate participants confirmed that they would advise others to use the reader app (average 4.8), and most of the visually impaired participants (average 4.5) concurred with this opinion. Finally, the feedback on question 2 had a gap rating between the two groups of participants (visually impaired and illiterate users). While illiterate users found the application very easy to use and rated this aspect at 4.7, some of the visually impaired users faced difficulties when using the application. This made them rate the average ease of use of the application at 3.5, which is still above the 2.5 average. That means the application was easy to use for both visually impaired and illiterate users.

6 Discussion

In this research, a mobile-based application was designed and implemented to help two groups of people in Iraq (visually impaired and illiterate people) to read texts. The developed app would assist both groups to explore what is written on different surfaces, such as paper, fabric, plastic, and wood. Moreover, the technologies, for example, OCR and real time translation, used to build the app enable it to capture different forms of words like names, numbers, and special symbols and allow it to support the local native language.

Firstly, a requirements elicitation study was conducted with both visually impaired and illiterate participants to determine the requirements they would like to add to or remove from the application. Similar to the results of earlier studies, including [4,21], and [24], our study found that the significant preferences for both visually impaired and illiterate users was to first build the app based on audio content and eliminate texts. The later evaluation study proved that building an application based on audio content helped both visually impaired and illiterate users in several aspects; for example, audio instructions were utilized to guide the users on how to use the app without needing to ask for help. This led to an increase in their feeling of user independency and in their willingness to use more digital tools. Moreover, audio feedback after every step the user undertaken helped user to recover if they made mistakes during using the app. Such that increased the level of application effectiveness and assist them to successfully complete the tasks.

Secondly, the analyzed data from the requirements elicitation sessions showed that the illiterate users did not mind having graphical content, especially simple and well-known symbols. Visually impaired users, in contrast, expressed a firm refusal to include any visual content in the app. Furthermore, during the requirements elicitation study, the visually impaired users were concerned about interacting with a smart touchscreen, and they wondered how they could find the buttons on such screens. This issue was also addressed by [12] as interacting with touchscreens compared to physical keys is not efficient or easy for blind users. As a result, visually impaired users would either need assistance or would need to find a better method of interacting with touchscreens [12]. Some earlier studies, such as [5,11], proposed solutions for this problem through building physical keys systems to read text and so avoid touchscreens. That means building new systems and not taking advantage of already available smartphones. Therefore, in this research, we utilized the smartphones that are widely available and owned by many people, even blind and illiterate people [13], to develop the proposed reader system. However, to tackle the problem of interaction with touchscreens, we reduced the number of buttons and developed an application that has only one big button, which is located at the bottom of the screen. This approach can drastically enhance the experience of the visually impaired interacting with touchscreens, which was proved in the results reported after the app evaluation.

Additionally, because most of the visually impaired and illiterate users had little or no experience with using smart devices, they thought they would face difficulties using and interacting with the app. However, the majority of participants from both groups, that is, visually impaired and illiterate users, demonstrated an extremely competent performance when using the app. This confirmed that by following the previously suggested design features (voice guide, auditory feedback, no texts or visual content, few or no buttons, local native language), the developed text reading application was efficient and effective and gained a high level of satisfaction for both visually impaired and illiterate users.

Finally, it is worth mentioning that illiterate users’ performance was higher than that of the visually impaired in all three evaluation metrics, namely, efficiency, effectiveness, and user satisfaction. This could be an indicator that some blind users, who could not complete the reading task, might need training before using the app. Training was suggested by the study participants and presented by [12] as a solution to help blind users be aware of the gestures available and to be more familiar with the app interface.

7 Research limitation

The small sample size of participants who were recruited for the evaluation study is the research only limitation. In total, 35 individuals (7 visually impaired and 28 illiterate users) were chosen to provide feedback on the application. This might be considered a modest number, especially with regard to the visually impaired participants. However, it has been noted that 12 participants in qualitative studies and 25–30 participants in quantitative studies are sufficient to approach data saturation [25,26]. As a result, the authors believed that 35 participants would be sufficient to address the research objectives and produce insightful findings.

8 Conclusions

This research presents a mobile-based application for visually impaired and illiterate users to read texts using various technologies, such as OCR and real time translation. Two studies were carried out; the first study was conducted with both groups of users to gather their requirements regarding their need for a reader application and the design of it. The study resulted in several design recommendations that supported both visually impaired and illiterate users, for example, voice guidance throughout the application, auditory feedback after each step, few or no buttons.

The second study was carried out after the reader application had been developed to assess its usability and gather users’ feedback. The evaluation findings showed that the reader application was effective and efficient for both groups of participants (visually impaired and illiterate users), since they managed to complete the given task within or under the average time and with no or only a minor number of errors. Another finding of the evaluation study was that the visually impaired users found interaction with the application more difficult compared to the illiterate users, as their performance was less competent over the two evaluation metrics of efficiency and effectiveness. However, these results could be improved by training the users on how to use the app.

In addition, the application gained a high level of user satisfaction, as users expressed a strong desire to continue using it and would recommend it to others. These findings demonstrate that creating applications that are usable will encourage both visually impaired and illiterate users to employ modern technology more, and this will help them overcome their concerns about smart devices. As a result, this will help to reduce the digital divide and increase the connectivity between communities.