A meta-analysis examining AI-assisted L2 learning
-
Guanyao Xu
,
Aiqing Yu
und
Lin Liu
Abstract
Numerous quantitative studies have investigated how artificial intelligence (AI) impacts the development of second language (L2). While individual studies delve into the effects of AI interventions on L2 learning, a meta-analysis provides a comprehensive evaluation of AI’s effectiveness in second language acquisition (SLA). Despite the growing body of meta-analytical research in AI-assisted language learning, several potential moderators have not been thoroughly investigated in previous meta-analyses. This meta-analysis examines the effectiveness of AI-assisted L2 learning and analyzes factors that can influence the effectiveness. The analysis included 15 studies that involved a total of 2,156 participants and generated 53 effect sizes. After correcting for measurement and sampling error, AI-assisted L2 learning demonstrated a positive large effect with d = 1.167. The Q statistic suggested that the true effect sizes varied significantly across studies, which warranted conducting a theory-based moderator analysis. The results of the moderator analysis revealed that type of AI-assisted interactions was not a significant moderator affecting the effectiveness of AI-assisted L2 learning; AI-assisted L2 learning was more beneficial in developing receptive skills than productive skills; AI technologies excelled at building learners’ vocabulary skills compared to other language skills; the effectiveness of AI-assisted L2 learning was higher in an in-class context than in an out-of-class context; IMALL was more impactful for L2 learning than ICALL; and there was no significant difference in the effectiveness of AI technology intervention for L2 learning between K-12 and college learners.
Funding source: Sichuan Office of Philosophy and Social Science
Award Identifier / Grant number: SC24BS043
Funding source: Sichuan Higher Education Society
Award Identifier / Grant number: GJXH2024YB-005
Acknowledgments
The authors would like to express their gratitude to the researchers who implemented studies in the related areas.
-
Research ethics: We confirm that this research has complied with all relevant national regulations and institutional policies.
-
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
-
Conflict of interest: The authors state no conflict of interest.
-
Research funding: This work was supported by Sichuan Office of Philosophy and Social Science (SC24BS043) and Sichuan Higher Education Society (GJXH2024YB-005).
-
Data availability: Data are available on request.
Appendix A: Studies included in the meta-analysis
Chen, Chih-Hung, Chorng-Shiuh Koong & Chien Liao. 2022. Influences of integrating dynamic assessment into a speech recognition learning design to support students’ English speaking skills, learning anxiety and cognitive load. Educational Technology & Society 25(1). 1–14.
Chen, Hsiu-Ling, Gracia Vicki Widarso & Hendri Sutrisno. 2020. A chatbot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research 58(6). 1161–1189.
Chen, Mei-Hua, Chung-Chi Huang, Shih-Ting Huang, Jason Chang & Hsien-Chin Liou. 2014. An automatic reference aid for improving EFL learners’ formulaic expressions in productive language use. IEEE Transactions on Learning Technologies 7(1). 57–68.
Chew, Chiou Sheng, Wen-Chi Vivian Wu, Norisma Idris, Er Fu Loh & Yan Piaw Chua. 2019. Enhancing summary writing of ESL learners via a theory-based online tool: System development and evaluation. Journal of Educational Computing Research 58(2). 398–432.
Choi, Inn-Chull. 2014. Efficacy of an ICALL tutoring system and process-oriented corrective feedback. Computer Assisted Language Learning 29(2). 334–364.
Dizon, Gilbert & John Gayed. 2021. Examining the Impact of Grammarly on the Quality of Mobile L2 Writing. jalt call Journal 17(2). 74–92.
Dodigovic, Marina. 2013. Intelligent sentence writing tutor: A system development cycle. International Journal of Artificial Intelligence in Education 22(3). 141–160.
Hu, Bo. 2021. English listening teaching model in flipped classroom based on artificial intelligence fusion control algorithm. Mathematical Problems in Engineering 2021. 1–14.
Jeon, Jaeho. 2023. Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis. Computer Assisted Language Learning 36(7). 1338–1364.
Kennedy, James, Paul Baxter, Emmanuel Senft & Tony Belpaeme. 2016. Social robot tutoring for child second language learning. Paper presented at the 11th ACM/IEEE international conference on human-robot interaction (HRI), IEEE, 7 March.
Li, Bing & Miaomiao Peng. 2022. Integration of an AI-based platform and flipped classroom instructional model. Scientific Programming 2022. 2536382.
Liu, Ming, Yi Li, Weiwei Xu & Li Liu. 2016. Automated essay feedback generation and its impact on revision. IEEE Transactions on Learning Technologies 10(4). 502–513.
Troussas, Christos, Konstantina Chrysafiadi & Maria Virvou. 2019. An intelligent adaptive fuzzy-based inference system for computer-assisted language learning. Expert Systems with Applications 127. 85–96.
Wu, Wen-Chi Vivian, Rong-Jyue Wang & Nian-Shing Chen. 2015. Instructional design using an in-house built teaching assistant robot to enhance elementary school English-as-a-foreign-language learning. Interactive Learning Environments 23(6). 696–714.
Ziegler, Nicole, Detmar Meurers, Patrick Rebuschat, Simón Ruiz, José Moreno‐Vega, Maria Chinkina, Wenjing Li & Sarah Grey. 2017. Interdisciplinary research at the intersection of CALL, NLP, and SLA: Methodological implications from an input enhancement project. Language Learning 67(S1). 209–231.
References
Ai, Haiyang. 2017. Providing graduated corrective feedback in an intelligent computer-assisted language learning environment. ReCALL 29(3). 313–334. https://doi.org/10.1017/s095834401700012x.Suche in Google Scholar
Anderson, Richard & Peter Freebody. 1981. Vocabulary knowledge. In John Guthrie (ed.), Comprehension and teaching: Research reviews, 77–117. Newark: International Reading Association.Suche in Google Scholar
Bahari, Akbar. 2022. Affordances and challenges of technology-assisted language learning for motivation: A systematic review. Interactive Learning Environments 31(9). 5853–5873. https://doi.org/10.1080/10494820.2021.2021246.Suche in Google Scholar
Benson, Phil. 2011. Language learning and teaching beyond the classroom: An introduction to the field. In Phil Benson & Hayo Reinders (eds.), Beyond the language classroom, 7–16. London: Palgrave Macmillan.10.1057/9780230306790_2Suche in Google Scholar
Bin-Hady, Wagdi Rashad Ali, Abdu Al-Kadi, Abduljalil Hazaea & Jamal Kaid Mohammed Ali. 2023. Exploring the dimensions of ChatGPT in English language learning: A global perspective. Library Hi Tech. https://doi.org/10.1108/lht-05-2023-0200 (Epub ahead of print).10.1108/LHT-05-2023-0200Suche in Google Scholar
Chen, Lijia, Pingping Chen & Zhijian Lin. 2020. Artificial intelligence in education: A review. IEEE Access 8. 75264–75278. https://doi.org/10.1109/access.2020.2988510.Suche in Google Scholar
Chen, Xieling, Di Zou, Haoran Xie & Gary Cheng. 2021. Twenty years of personalized language learning: Topic modeling and knowledge mapping. Educational Technology & Society 24(1). 205–222.Suche in Google Scholar
Chen, Chih-Hung, Chorng-Shiuh Koong & Chien Liao. 2022. Influences of integrating dynamic assessment into a speech recognition learning design to support students’ English speaking skills, learning anxiety and cognitive load. Educational Technology & Society 25(1). 1–14.Suche in Google Scholar
Choi, Inn-Chull. 2016. Efficacy of an ICALL tutoring system and process-oriented corrective feedback. Computer Assisted Language Learning 29(2). 334–364. https://doi.org/10.1080/09588221.2014.960941.Suche in Google Scholar
Cowan, Nelson.. 2011. The focus of attention as observed in visual working memory tasks: Making sense of competing claims. Neuropsychologia 49(6). 1401–1406. https://doi.org/10.1016/j.neuropsychologia.2011.01.035.Suche in Google Scholar
Davies, Norman. 1976. Receptive versus productive skills in foreign language learning. The Modern Language Journal 60(8). 440–443. https://doi.org/10.1111/j.1540-4781.1976.tb03667.x.Suche in Google Scholar
De Wilde, Vanessa, Marc Brysbaert & June Eyckmans. 2020. Learning English through out-of-school exposure. Which levels of language proficiency are attained and which types of input are important? Bilingualism: Language and Cognition 23(1). 171–185. https://doi.org/10.1017/s1366728918001062.Suche in Google Scholar
Derakhshan, Ali & Farhad Ghiasvand. 2024. Is ChatGPT an evil or an angel for second language education and research? A phenomenographic study of research‐active EFL teachers’ perceptions. International Journal of Applied Linguistics 34(4). 1246–1264. https://doi.org/10.1111/ijal.12561.Suche in Google Scholar
Derakhshan, Ali, Timothy Teo, Esmaeel Saeedy Robat, Janebi Enayat Mostafa & Akbar A. Jahanbakhsh. 2024a. Robot-assisted Language Learning: A meta-analysis. Review of Educational Research. 00346543241247227. https://doi.org/10.3102/00346543241247227.Suche in Google Scholar
Derakhshan, Ali, Timothy Teo & Saeed Khazaie. 2024b. Investigating the usefulness of artificial intelligence-driven robots in developing empathy for English for medical purposes communication: The role-play of Asian and African students. Computers in Human Behavior 162. 108416. https://doi.org/10.1016/j.chb.2024.108416.Suche in Google Scholar
Gibson, James. 1977. The theory of affordances. In Robert Shaw & John Bransford (eds.), Perceiving, acting, and knowing: Toward an ecological psychology, 67–82. Erlbaum.Suche in Google Scholar
Grgurović, Maja, Carol Chapelle & Mack Shelley. 2013. A meta-analysis of effectiveness studies on computer technology-supported language learning. ReCALL 25(2). 165–198. https://doi.org/10.1017/s0958344013000013.Suche in Google Scholar
Hasan, Muhammad Asif, Nurul Fazmidar Mohd Noor, Siti Soraya Binti Abdul Rahman & Mohammad Mustaneer Rahman. 2020. The transition from intelligent to affective tutoring system: A review and open issues. IEEE Access 8. 204612–204638. https://doi.org/10.1109/access.2020.3036990.Suche in Google Scholar
Heilman, Michael, Kevyn Collins-Thompson, Jamie Callan, Maxine Eskenazi, Alan Juffs & Lois Wilson. 2010. Personalization of reading passages improves vocabulary acquisition. International Journal of Artificial Intelligence in Education 20(1). 73–98.Suche in Google Scholar
Hitch, Graham & Michael Halliday. 1983. Working memory in children. Philosophical Transactions of the Royal Society of London. B, Biological Sciences 302(1110). 325–340.10.1098/rstb.1983.0058Suche in Google Scholar
Hunter, John & Frank Schmidt. 2004. Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks: SAGE.Suche in Google Scholar
Kalyuga, Slava. 2012. Instructional benefits of spoken words: A review of cognitive load factors. Educational Research Review 7(2). 145–159. https://doi.org/10.1016/j.edurev.2011.12.002.Suche in Google Scholar
Kerins, John & Allan Ramsay. 2012. Developing a visual temporal modeller: Applying an extensible nlp system to support learners’ understanding of tense and aspect in English. ReCALL 24(1). 40–65. https://doi.org/10.1017/s0958344011000280.Suche in Google Scholar
Krashen, Stephen. 1982. Principles and practice in second language acquisition. Oxford: Pergamon Press.Suche in Google Scholar
Kreijns, Karel, A. Kirschner Paul & Wim Jochems. 2002. The sociability of computer-supported collaborative learning environment. Journal of Educational Technology Society 5. 8–22.Suche in Google Scholar
Lai, Chun. 2014. Perceiving and traversing in-class and out-of-class learning: Accounts from foreign language learners in Hong Kong. Innovation in Language Learning and Teaching 9(3). 265–284. https://doi.org/10.1080/17501229.2014.918982.Suche in Google Scholar
Lai, Chun. 2019. Learning beliefs and autonomous language learning with technology beyond the classroom. Language Awareness 28(4). 291–309. https://doi.org/10.1080/09658416.2019.1675679.Suche in Google Scholar
Lai, Chun, Yang Liu, Jingjing Hu, Phil Benson & Boning Lyu. 2022. Association between the characteristics of out-of-class technology-mediated language experience and L2 vocabulary knowledge. Language, Learning and Technology 26(1). 1–24.Suche in Google Scholar
Lai, Chun, Qianshan Chen, Yabing Wang & Xuedan Qi. 2024. Individual interest, self‐regulation, and self‐directed language learning with technology beyond the classroom. British Journal of Educational Technology 55(1). 379–397. https://doi.org/10.1111/bjet.13366.Suche in Google Scholar
Lee, Je-Young & Yohan Hwang. 2022. A meta-analysis of the effects of using AI chatbot in Korean EFL education. Studies in English Language and Literature 48(1). 213–243.Suche in Google Scholar
Lee, Hansol & Jang Ho Lee. 2024. The effects of AI-guided individualized language learning: A meta-analysis. Language, Learning and Technology 28. 134–162.Suche in Google Scholar
Liu, Guangxiang Leon, Ron Darvin & Chaojun Ma. 2024. Unpacking the role of motivation and enjoyment in AI-mediated informal digital learning of English (AI-IDLE): A mixed-method investigation in the Chinese context. Computers in Human Behavior 160. 108362. https://doi.org/10.1016/j.chb.2024.108362.Suche in Google Scholar
Ma, Qing. 2017. Technologies for teaching and learning L2 vocabulary. In Carol Chapelle & Shannon Sauro (eds.), The handbook of technology and second language teaching and learning, 45–61. Hoboken NJ: Wiley.10.1002/9781118914069.ch4Suche in Google Scholar
Moinzadeh, Ahmad & Roghaieh Moslehpour. 2012. Depth and breadth of vocabulary knowledge: Which really matters in reading comprehension of Iranian EFL learners? Journal of Language Teaching and Research 3(5). 1015. https://doi.org/10.4304/jltr.3.5.1015-1026.Suche in Google Scholar
Mosca, Michela & Kees De Bot. 2017. Bilingual language switching: Production vs. Recognition. Frontiers in Psychology 8. 934. https://doi.org/10.3389/fpsyg.2017.00934.Suche in Google Scholar
Paas, Fred & Jeroen van Merriënboer. 1994. Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review 6(4). 351–371. https://doi.org/10.1007/bf02213420.Suche in Google Scholar
Pederson, Kathleen Marshall. 1986. An experiment in computer-assisted second-language reading. The Modern Language Journal 70(1). 36–40. https://doi.org/10.1111/j.1540-4781.1986.tb05242.x.Suche in Google Scholar
Qureshi, Muhammad Asif, Asadullah Khaskheli, Jawaid Ahmed Qureshi, Syed Ali Raza & Sara Qamar Yousufi. 2023. Factors affecting students’ learning performance through collaborative learning and engagement. Interactive Learning Environments 31(4). 2371–2391. https://doi.org/10.1080/10494820.2021.1884886.Suche in Google Scholar
Rosenthal, Robert. 1991. Meta-analytic procedures for social research. Thousand Oaks: SAGE.10.4135/9781412984997Suche in Google Scholar
Schmidt, Frank, In-Sue Oh & Theodore Hayes. 2009. Fixed‐versus random‐effects models in meta‐analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology 62(1). 97–128. https://doi.org/10.1348/000711007x255327.Suche in Google Scholar
Schulze, Mathias. 2008. AI in CALL – artificially inflated or almost imminent? CALICO Journal 25(3). 510–527. https://doi.org/10.1558/cj.v25i3.510-527.Suche in Google Scholar
Shardlow, Matthew, Sam Sellar & David Rousell. 2022. Collaborative augmentation and simplification of text (CoAST): Pedagogical applications of natural language processing in digital learning environments. Learning Environments Research 25(2). 399–421. https://doi.org/10.1007/s10984-021-09368-9.Suche in Google Scholar
Shi, Zhan, Fengkai Liu, Chun Lai & Tan Jin. 2022. Enhancing the use of evidence in argumentative writing through collaborative processing of content-based automated writing evaluation feedback. Language, Learning and Technology. 106–128.Suche in Google Scholar
Sweller, John. 2017. Cognitive load theory and teaching English as a second language to adult learners. Contact Magazine 43(1). 10–14.Suche in Google Scholar
Sweller, John, Jeroen van Merriënboer & Fred Paas. 1998. Cognitive architecture and instructional design. Educational Psychology Review 10(3). 251–296. https://doi.org/10.1023/a:1022193728205.10.1023/A:1022193728205Suche in Google Scholar
Wang, Yongliang & Lina Xue. 2024. Using AI-driven chatbots to foster Chinese EFL students’ academic engagement: An intervention study. Computers in Human Behavior. 108353. https://doi.org/10.1016/j.chb.2024.108353.Suche in Google Scholar
Weng, Xiaojing & Thomas Chiu. 2023. Instructional design and learning outcomes of intelligent computer assisted language learning: Systematic review in the field. Computers and Education: Artificial Intelligence. 100117. https://doi.org/10.1016/j.caeai.2022.100117.Suche in Google Scholar
Wible, David, Anne Li-E Liu & Nai-Lung Tsao. 2011. A browser‐based approach to incidental individualization of vocabulary learning. Journal of Computer Assisted Learning 27(6). 530–543. https://doi.org/10.1111/j.1365-2729.2011.00413.x.Suche in Google Scholar
Wijekumar, Kausalai Kay, Bonnie Meyer & Puiwa Lei. 2013. High-fidelity implementation of web-based intelligent tutoring system improves fourth and fifth graders content area reading comprehension. Computers & Education 68. 366–379. https://doi.org/10.1016/j.compedu.2013.05.021.Suche in Google Scholar
Wu, Xueqing & Rui Li. 2024. Unraveling effects of AI chatbots on EFL learners’ language skill development: A meta-analysis. The Asia-Pacific Education Researcher. 1–12. https://doi.org/10.1007/s40299-024-00853-2.Suche in Google Scholar
Wu, Hanwei, Yunsong Wang & Yongliang Wang. 2024a. “To Use or Not to Use?” A mixed-methods study on the determinants of EFL college learners’ behavioral intention to use AI in the distributed learning context. International Review of Research in Open and Distributed Learning 25(3). 158–178. https://doi.org/10.19173/irrodl.v25i3.7708.Suche in Google Scholar
Wu, Hanwei, Wentao Liu & Yonghong Zeng. 2024b. Validating the AI-assisted second language (L2) learning attitude scale for Chinese college students and its correlation with L2 proficiency. Acta Psychologica 248. 104376. https://doi.org/10.1016/j.actpsy.2024.104376.Suche in Google Scholar
Xia, Qi, Thomas Chiu, Min Lee, Ismaila Temitayo Sanusi, Yun Dai & Ching Sing Chai. 2022. A self-determination theory (SDT) design approach for inclusive and diverse artificial intelligence (AI) education. Computers & Education 189. 104582. https://doi.org/10.1016/j.compedu.2022.104582.Suche in Google Scholar
Xu, Guanyao, Aiqing Yu, Cong Xu, Xianquan Liu & Guy Trainin. 2024. Investigating pre-service TCSL teachers’ technology integration competency through a content-based AI-inclusive framework. Education and Information Technologies. 1–32. https://doi.org/10.1007/s10639-024-12982-8.Suche in Google Scholar
Yu, Aiqing & Guy Trainin. 2022. A meta-analysis examining technology-assisted L2 vocabulary learning. ReCALL 34(2). 235–252. https://doi.org/10.1017/s0958344021000239.Suche in Google Scholar
Zhai, Chunpeng & Santoso Wibowo. 2023. A systematic review on artificial intelligence dialogue systems for enhancing English as foreign language students’ interactional competence in the university. Computers and Education: Artificial Intelligence 4. 100134. https://doi.org/10.1016/j.caeai.2023.100134.Suche in Google Scholar
Zhang, Shunan, Cheng Shan, John Sie Yuen Lee, ShaoPeng Che & Jang Hyun Kim. 2023. Effect of chatbot-assisted language learning: A meta-analysis. Education and Information Technologies 28(11). 15223–15243. https://doi.org/10.1007/s10639-023-11805-6.Suche in Google Scholar
Zhao, Yong. 2005. The future of research in technology and second language education: Challenges and possibilities. In Yong Zhao (ed.), Research in technology and second language education: Developments and directions, 445–457. Greenwich: Information Age Publishing.Suche in Google Scholar
© 2025 Walter de Gruyter GmbH, Berlin/Boston
