The linguistic accuracy of chatbots: usability from an ESL perspective
-
David Coniam
David Coniam is Chair Professor in the Faculty of Education Studies at the Hong Kong Institute of Education, where he is a teacher educator, working with ESL teachers in Hong Kong secondary schools. His main publication and research interests are in language teaching methodology and language assessment.
Abstract
This paper reports on the linguistic accuracy of five renowned “chatbots,” with an evaluator (an ESL teacher) chatting with each chatbot for about three hours. The chatting consisted of a series of set questions/statements (determined as being in the domain of an ESL learner) – aimed at assessing the accuracy and felicity of the chatbots' answers at the grammatical level. Results indicate that chatbots are generally able to provide grammatically acceptable answers, with three chatbots returning acceptability figures in the 90% range. When meaning is factored in, however, a different picture emerges, with the chatbots often providing meaningless, nonsensical answers, and the accuracy rate for the joint categories of grammar and meaning falling below 60%. The paper concludes on the note that although chatbots as “conversation practice machines” do not yet make robust chatting partners, improvements in chatbot performance bode well for future developments.
About the author
David Coniam is Chair Professor in the Faculty of Education Studies at the Hong Kong Institute of Education, where he is a teacher educator, working with ESL teachers in Hong Kong secondary schools. His main publication and research interests are in language teaching methodology and language assessment.
©2014 by Walter de Gruyter Berlin/Boston
Articles in the same Issue
- Frontmatter
- The creation of an “imagined community” in readers' letters to the Daily Sun: an APPRAISAL investigation
- The linguistic accuracy of chatbots: usability from an ESL perspective
- A drift toward direct structures in Dutch direct mail sales letters
- “You are children but you can always say ...”: hypothetical direct reported speech and child–parent relationships in a Heritage Language classroom
- “Christians” and “bad Christians”: categorization in atheist user talk on YouTube
- Rapport management in strong disagreement: an investigation of a community of Chinese speakers of English
Articles in the same Issue
- Frontmatter
- The creation of an “imagined community” in readers' letters to the Daily Sun: an APPRAISAL investigation
- The linguistic accuracy of chatbots: usability from an ESL perspective
- A drift toward direct structures in Dutch direct mail sales letters
- “You are children but you can always say ...”: hypothetical direct reported speech and child–parent relationships in a Heritage Language classroom
- “Christians” and “bad Christians”: categorization in atheist user talk on YouTube
- Rapport management in strong disagreement: an investigation of a community of Chinese speakers of English
