Using social robots for language learning: are we there yet?

2.1 From industrial robots to social robots

Robots are machinery agents which can carry out a series of actions automatically, based on pre-set programmes. They are equipped with hardware and software to collect information, process signals, and convert electrical signals into physical movement (Bartneck et al., 2020, pp. 18–37). In comparison with robots used in the industrial domain, which mainly focus on completing physical tasks, social robots are expected to communicate and interact with people on an emotional level (Darling, 2016). With the rapid development of speech and language technologies in recent decades, social robots have gained the ability to perceive, process and produce speech, such as automatic speech recognition (ASR) or text-to-speech (TTS), natural language processing (NLP) and text-to-speech (TTS). In other words, robots can interact with human users via speech.

This speech-enabled capability has fostered robots’ transition from the industrial domain to social domains, such as service industries, healthcare, entertainment and education (Bartneck et al., 2020, p. 163). In this last regard, robots, as pedagogical tools, are not only popular for science, technology, engineering and maths (STEM) education but also show great promise in social interaction with increasing cognitive and affective outcomes (Belpaeme et al., 2018). The popularity of using social robots in educational environments has been increasing over the years. Analysts expect the robotics education market to reach a market value of $2.6 billion by 2026 (MarketsandMarkets, 2021). Language learning is one of the three major application areas for social robots (Mubin et al., 2013).

2.2 Educational social robots

2.2.1 Features, advantages and myths

In comparison with traditional digital learning tools used for language learning and teaching, a distinct advantage that social robots have is their physical embodiment. Social robots’ embodiment tends to be multimodal. It combines multiple sensors, actuation and locomotion. Thus, it can offer a wider range of interactive possibilities in language classrooms than other forms of technology (e.g., tablets, computers and smartphones). For example, it can interact with the learning environment and learners physically. As shown in (de Wit et al., 2018), the robot’s use of gestures positively affected students’ long-term memorisation of words in the second language (L2).

As early as 1986, Harwin, Ginige and Jackson proposed using robots for physical interaction in early education (Harwin et al., 1986). In addition, studies have shown that the presence of social robots can (1) help students achieve better task performance compared to virtual agents or robots displayed on screens (Leyzberg et al., 2012; Li, 2015) and (2) increase people’s evaluation of robots and their interactions with robots by making robots appear more appealing, perceptive and enjoyable (Jung & Lee, 2004; Wainer et al., 2007). This advantage is likely attributable to many factors. One major reason could be robots’ positive effects on learning motivation, which could be very rewarding for second language acquisition, according to Krashen and Terrel’s Affective Filter Hypothesis (Krashen, 2009). Such a positive connection between robots’ embodiment and motivation has been found in many types of robot-assisted learning, yet generally not found in other types of technology (Van den Berghe et al., 2019).

However, social robots’ ability to motivate students is not completely clear. Students’ enthusiasm could be sparked by the new technology, which would not have been sustained over long periods, as discussed in Van den Berghe et al.’s review in 2019 (Van den Berghe et al., 2019). Besides this novelty effect, social robots’ physical presence could also cause unexpectedly worse performance. In a study investigating children’s grammar learning, children performed worse when the robot looked at them (Herberg et al., 2015). It is unclear whether such counterproductive effects were caused by increased pressure (a possible explanation provided by the experimenters in Herberg et al. (2015)) or increased distraction or even fear. Apart from that, robots’ presence may also cause negative effects if they do not have the touch-input capability, as shown in a study in 2004 (Jung & Lee, 2004). The study has flagged the importance of (1) bridging users’ expectations of an embodied robot and its capability and (2) using tactile communication (not necessarily via a touch screen) to enhance robots’ social presence.

Added to motivating students, social robots also have many advantages, such as their capability when handling repeated tasks without fatigue. Also, they have the flexibility to be programmed to take up various roles in the classroom, such as teachers or learning companions (Aidinlou et al., 2014; Mubin et al., 2013). These potential advantages of social robots also come with many issues that need to be addressed, including technology limits, robots’ credibility and explainability, usability and social impacts on language teaching and learning. These will be addressed in the implementation section later. Based on the extensive review by Randall in 2019 (Randall, 2019), which includes 79 papers about RALL, it is clear that interest and enthusiasm about RALL have been on the rise over the past decade (shown in Figure 1). Along with the rapid growth of communication technology and robotics, it is anticipated that RALL will keep growing in the coming decades.

Figure 1:

Numbers of published RALL papers from 2004 to 2017.

2.2.2 Types by autonomous function and appearance

According to Randall’s review in 2019 (Randall, 2019), at least 26 different social robots were used in 79 RALL studies from 2004 to 2017. The review provides a detailed description of the characteristics of robots used in RALL, such as autonomous functions, forms, voices, social roles, verbal and non-verbal immediacy and personalization. Among these features, robots are often categorised by their autonomous functionality and appearances.

According to the degree of automation, robots used in RALL can be: (1) fully autonomous: acting upon predefined programmes and generating contingent responses when the participants behave as expected; (2) fully teleoperated (or telepresent): being operated remotely by the people to generate more flexible real-time (re)action; (3) transformed: being between the above two levels of operations (Han, 2012). Although it is natural to test the usability of a fully autonomous robot, all three types of robots are often used in RALL because they can serve different purposes. For example, due to technical limitations of handling automatic speech recognition (ASR) of non-native speakers, as well as limited incremental dialogue systems, robots may have difficulties understanding non-native language learners’ speech, not to mention adapting their speech behaviours accordingly (e.g., speech rate, rephrase, emphasis specific parts). Thus, teleoperated or semi-teleoperated robots are more suitable for studying advanced interactions in order to identify appropriate robot properties or behaviours for further development. Some examples of these robots are provided in Figure 2 (Furhat Robotics, 2023; Ishiguro et al., 2001; Lee et al., 2006; Yun et al., 2011).

Figure 2:

Robots used in RALL depending on their autonomous functionalities. (a) Autonomous Robot: Robovie, (b) Teleoperated Robot: iRobi, (c) Transformable Robot: Furhat.

Robots used in RALL can also be categorised by their forms. Robots’ forms and functions are interconnected (Bartneck et al., 2020, p. 43). In other words, the form of a robot represents how it can interact with people and also sets physical constraints. For language learning purposes, robots have facial expressions, body gestures and speech-enabled capabilities that are preferable. Hence, differentiating from companion robots that could be beneficial to have an animal-like form with limited functions (e.g., a seal-like robot companion “Paro”^[1]), robots used in RALL are often equipped with more human-like characteristics or even human-like forms. As shown in Figure 3, some RALL robots are still zoomorphic but built with more expressive functions, like iCat (van Breemen et al., 2005); some are cartoon-like, like DragonBot (MIT Media Lab, n.d.); some are more human-like, which is known as being anthropomorphic, like NAO (Gouaillier et al., 2009),^[2] which is currently the most popular research platform in social robotics (Bartneck et al., 2020, p. 14) (Randall, 2019) (as shown in Figure 4). Robots like “NAO” are called humanoid robots, which resemble the human body in shape, partially or fully. A minimal social robot, “Keepon”, has also been popular in RALL studies. It is shown that the simple form is sufficient to achieve the expected interaction outcomes (Kozima et al., 2009).

Figure 3:

Robots used in RALL depending on their forms. (a) Zoomorphic Robot: iCat robot, (b) Carton-like Robot: DragonBot, (c) Human-like Robot: NAO, (d) Minimal Social Robot: Keepon.

Figure 4:

Robots used in RALL research by countries and regions (2004–2017).

This wide range of options has provided many possibilities in RALL studies, especially the research hotspot of children-robot interaction in RALL. For the robots used in 79 relevant RALL studies (2004–2017), Randall has provided a detailed table in the review (Randall, 2019, p. 7: 5). On the basis of Randall’s table, it can be seen that most areas involved in RALL have used only one or two types of robot. South Korea has used the largest variety of robots in RALL (as shown in Figure 4). One possible reason could be that the choice of a robot to use in RALL studies is often related to the availability and affordability of the robot. Nevertheless, where possible, the choice of the robot should also consider the perception of the robot by the teaching audience, including its size, appearance and cultural suitability. Perception matters because, according to affordance theory (Gibson, 1977; Matei, 2020), perception affects expectations and interaction with the robot. For those interested in robots’ characteristics, Van de Bergh et al. have summarised them in a clear table (Van den Berghe et al., 2019, p. 287). It is also worth bearing in mind that robots’ human likeness is not directly linked with their expressiveness, as illustrated in Engwall and Lopes’s review in 2022 (Engwall & Lopes, 2022). The current RALL is generally applied to younger learners. As future RALL research pays more attention to adult second language acquisition, it is hoped to see more adult-friendly language teaching robots with appropriate forms.

3.1 RALL in East Asia

Early research and commercialisation of RALL began in countries and regions where English has been popular as a second language to acquire. In East Asia, the market for teaching English as a second language is huge, and the development of robotics research is at the forefront of the world. Thus, it is easy to imagine why Asian countries have explored and experimented with language-teaching robots earlier and to a great extent. To address the dilemma of the lack of native English teachers in their home countries, South Korea and Japan have successively experimented with robot teachers with the support of their governments.

3.2 RALL in Europe

Compared to RALL in Asia, the use of robotic language teachers in Europe has a shorter history and is less widespread. One of the larger projects in the early years was Second Language Tutoring using Social Robots (L2TOR^[3]). The project was funded by the European Commission’s Horizon 2020 programme and ran from 1 January 2016 to 31 December 2018. Using NAO robots, the project studied how children aged 4–6 learn a second language, with the help of social robots, through interaction with NAO. Examples include how native speakers of Dutch, German and Turkish learn English vocabulary, how native English-speaking children learn French vocabulary and grammar, and how children who have immigrated to the Netherlands from Turkey learn Dutch vocabulary.

In the classroom, the interactive human-computer platform is usually a tablet with learning games that create learning situations for students. Students sat next to the NAO, listened to the NAO interpret the game, repeated the words with the NAO’s voice and movements, or touched the tablet to interact with the NAO. The project’s main focus was a large-scale field study of 200 Dutch children learning 34 English words in seven lessons with the help of the NAO robot. The findings show that children can acquire words through interaction with NAO but that the same results can be achieved if NAO was removed and a regular tablet was used (Vogt et al., 2019). The L2TOR project has produced a rich body of research, including five focused articles, six PhD theses and several other journal articles from 2015 to 2022.^[4] These have contributed to a greater understanding of the role, impact, challenges and opportunities of social robots for children’s second language acquisition.

In the Nordic region, a study called “Collaborative Robot-Assisted Language Learning” (CORALL^[5]) is being funded by the Swedish Research Council for 2017–2020. Its social relevance is to contribute to more effective Swedish-language immigrant education by combining collaborative learning pedagogy with computer-assisted language learning and social robotics.

Another research-driven project is Early Language Development in the Digital Age, or “e-LADDA”.^[6] The project, which runs from 2019 to 2024, is a collaboration between academics, the non-academic public sectors and technology companies in the industry. It aims to investigate how digital tools affect language development and performance in young children, as well as to improve understanding of the technology itself and how it is used. The project’s primary goal is to provide a unified research methodology for studying how digital technologies affect early childhood language learning and to provide guidelines for policymakers, educators, practitioners and families on navigating, regulating and adapting to emerging digital environments.

4.1 For language learners

Among the available RALL studies, vocabulary learning has taken up the largest proportion, followed by reading and speaking skills, with minimal research done on grammar learning (Van den Berghe et al., 2019). From the learners’ perspective, studies have shown that RALL has generated positive results in helping L2 learners to acquire reading (Hong et al., 2016) and grammar skills (Herberg et al., 2015; Kennedy et al., 2016). According to (Van den Berghe et al., 2019), there is a mixed picture in terms of vocabulary learning and speaking. As for affective gains, as discussed earlier, L2 learners tend to be highly motivated by the physical presence of and interaction with robots, especially in the short term. However, it is not clear how much a feeling of novelty has impacted the L2 learners’ performance. According to the extensive review (Randall, 2019), students’ classroom participation and self-confidence were improved under RALL. Additionally, contradictory results about RALL’s influence on L2 learners’ social behaviours are given. For example, L2 learners did not show any lexical and syntactic alignment when they spoke to an embodied robot or the virtual agent (Rosenthal-von der Pütten et al., 2016). However, L2 learners have shown different responses to the robot’s feedback: sadly, punishment feedback is shown to be more effective than reward feedback (de Haas & Conijn, 2020).

4.2 For language teachers

From the teachers’ perspective, language teaching and learning is a continuously interactive activity which requires collaboration as much as other spoken interactions (Holtgraves, 2013). Thus, it is worth finding out how to work with robots in the language classroom effectively. Olov’s review has shed some light on this aspect by reviewing teaching strategies and how they are combined with robots used in RALL (Engwall & Lopes, 2022). It has inspired creativity in language classrooms. Robots can take on different roles, either as teaching assistants, learning companions, or even as ‘little villains’ who deliberately make mistakes, depending on the complexity of tasks and freedom in human-robot interactions. A more specific review of RALL’s oral interaction, including 22 empirical studies from 2010 to 2020, shows that communicative language teaching (CLT) is most widely used in RALL, followed by teaching proficiency through reading and storytelling (TPRS) (Lin et al., 2022). It also details what actions teachers took in those experiments to collaborate with robots.

4.3 For technologies used in RALL

From the robot developers’ perspective, studies of RALL have flagged many aspects to consider, including robots’ form, voice, social roles and behaviours (Randall, 2019). For example, with the advantage of embodiment, the robot can also use encouraging gestures to enhance learning (Harinandansingh, 2022). In theory and practice, building a language-learning-assisted robot is achievable. Qilin, a Chinese vocabulary-learning robot, is an example (Nomoto et al., 2022). It utilised mature Application Programming Interfaces (API) like Google Cloud on a low-cost computer like Raspberry Pi. It was tailored for specific language-learning purposes in the experiment. Whilst using multimodal cues like lights and movement enhances the interactive effects, the choice of Qilin’s voice seems not liked by students. Plus, Qilin’s camera does not provide any practical use other than making the robot look more like a human.

Notably, while developers may well want human-robot interaction to be as natural as human interaction, this does not mean that every aspect of the robot has to be as human-like as possible. The design of robots’ behaviours and affordances is not only a technical but also an ethical issue (Hildt, 2021; Huang & Moore, 2022). For example, people could react negatively to a robot’s deceptive praise (Ham & Midden, 2014).

4.4 Session summary

The above assessment of RALL is only a glimpse into the whole picture. Most of the RALL at this stage has focused on children and adolescent second language learners, with less research on adult second language acquisition. The different tasks were used in different studies, with different experimental designs, sizes of participants and lengths of human-computer interaction, or even lack of control conditions. All of these add to the difficulty of cross-sectional comparisons and come to cogent conclusions.

5.1 Technological challenges

In theory, RALL has essential advantages that other technology-based language teaching tools do not have. It is important to note that these advantages are currently underdeveloped in reality (Van den Berghe et al., 2019). One of the reasons is that, compared to CALL, RALL has to face a more dynamic interactive environment. This set higher demands for coherence and robustness of language-learning-assisted robots. From the designer’s perspective, misalignment would cause conflicting perceptions, which could lead to decreased motivation to interact with a robot (Meah & Moore, 2014). As for the state-of-art artificial intelligence, visual recognition system, automatic speech recognition system and dialogue system are not yet sufficient to allow the robot to automatically and fluently talk to any second language learner, partially because L2 speakers’ speeches are more unpredictable and more difficult to recognise. Also, a significant advantage of the robot is the ability to interact with the teaching environment and students using an embodiment. This advantage is not fully utilised. It is hoped that developers will soon break down the limitations of robotics (e.g. stability in movement, overheating issues) and allow the advantages of the robot to be better utilised.

In addition to these matters, the technical challenges faced by RALL require consideration of the specificities of language teaching and learning. Evaluation criteria in academia or industry may not be the most appropriate standard for language learning. Take the Automatic Assessment of Pronunciation (AAP) in L2 for example. The principle of robot judgement is to compare the sound produced by L2 learners with the sound that the system considers correct. This judgement started from the phoneme-based comparison in the early days (e.g., distinguishing between voiced stops and voiced fricatives (Weigelt et al., 1989)), with a shifted focus on fluency at the sentence level (Bernstein et al., 2011; Kim et al., 1997), and then has developed further by taking into count the influence of social factors (e.g., age, gender) (Strand, 1999). In recent years, the importance of measuring genuine listener intelligibility has attracted more attention. Instead of native-like accuracy, clear speech has become a more realistic goal for intelligibility assessment (O’Brien et al., 2018) but has yet to further develop or commercialise.

Thus, adopting automatic judgement in language learning requires careful consideration. It might be worth considering if teaching aims to get students to pronounce exactly like L1 speakers. But the question is, what are these supposedly correct L1 speakers’ data that are used to train the model? Why should L2 speakers’ pronunciation be identical to L1 speakers’? How good is L2 users’ pronunciation good enough? How important is the pronunciation of words and sentences in oral interactions? What is the impact of emphasising correct pronunciation on the development of students’ oral or general language skills? If these questions cannot be answered, it may be unwise to promote technology-based pronunciation assessment, including RALL. Thus, on the one hand, it is essential to adopt the human-centred approach to develop more adaptive assessment tools to suit the diverse needs of second language learners. On the other hand, product developers are expected to enhance the transparency and explainability of the technology. This will make it easier for users to (1) choose the appropriate context of use and (2) judge how to interpret the results obtained.

Another aspect to consider is the relationship between robots and humans. Given a robot is not a one-off disposable product in language classrooms, can the robot maintain a long-term, healthy relationship with the student? Also, the language learning effect could be variable. Can a robot adapt its behaviour (e.g., feedback) to the student’s individual characteristics and difficulties? Additionally, the interaction between RALL and students and teachers is bound to generate a lot of data. Data collection, storage and sharing is also an ethical matter worth considering.

5.2 Opportunities for L2 teachers

As mentioned earlier, the novelty that the robot brings to the students is likely to wear off after a few interactions. Therefore, if L2 teachers are interested in trying RALL, they must consider introducing the robot as a new teaching tool to their students to reduce the novelty effect. A more pragmatic approach for language teachers is to be grounded in reality, identify the real-world problems that need to be addressed, understand the shortcomings of existing products, and then see what can be done and how far it can go within the confines of the syllabus based on the current level of technology.

Would robots replace language teachers on various fronts? Teaching is not a simple process of telling students what to do but a multi-linked activity. The teacher’s function involves selecting, organising, and presenting materials, monitoring the effectiveness of student learning, giving feedback, and adjusting one’s teaching to changes in the external learning environment and the needs of students. An experienced teacher can take into account all aspects and tailor the teaching to the needs of the students and the local context. Should or could a language-learning-assisted robot be as skilful and all-encompassing as that? Furthermore, interaction is a complex two-way collaborative process, and oral interaction is the challenging part of human-computer interaction (Moore, 2016). For language teaching, the interaction between teachers and students contains many other elements besides the transmission of knowledge and information, and further research on its complexity is needed. After all, robots, like other technology-based teaching tools (e.g., PowerPoint), can serve certain aspects of teaching and learning. The key is not to use it or not, but to use it in what situations and how.

While waiting for the technology to mature, teachers can increase their knowledge of RALL and familiarise themselves with this burgeoning technology. This is not only in preparation for future classroom use but also to facilitate participation in the development of RALL. This requires AI’s cognitive mechanics and interactive capabilities. That makes artificial social interaction ‘one of the most formidable challenges in artificial intelligence and robotics’ (Belpaeme et al., 2018). Teachers have an important role to play in the development of RALL. Here are some potential directions to explore.

1. RALL Teacher Training: One example is to consider teacher training related to RALL, from theory (Mishra & Koehler, 2006) to practice (Li & Tseng, 2022).

2. Non-language Aspects in L2 Acquisition: Another example is to expand the application of RALL from language teaching to cultural teaching to help students understand the social etiquette and customs of the target language country (Wallace, 2020).

3. Work Around the Limits: Consider innovative ways to work with the limits of robots, like having 2-to-1 group interaction instead of 1-2-1 interaction between robots and students (Engwall et al., 2021; Khalifa et al., 2018).