Developing writing skills and feedback in foreign language education with chatGPT: a multilingual perspective

4.1.1 Students’ methodology for completing tasks

Across the various languages represented in the experiment, looking at the students’ chat history data, we saw that they used ChatGPT to generate an essay plan, correct their grammar and to provide content and ideas. Some participants engaged in a collaborative dialogue with ChatGPT, adopting an enquiry-based approach. For instance, a participant in Chinese asked about translations of specific sentences and words, sought definitions of certain terms, and requested that ChatGPT analyse sentences. The student also questioned ChatGPT’s word choices, asking why it had opted for certain words over others. Furthermore, they sought clarification on the distinctions between various words and requested example sentences demonstrating particular grammatical constructions. At the other end of the spectrum, one of the participants in French used ChatGPT as a vocabulary tool exclusively asking questions about: essential vocabulary related to university; how to use “encore plus”; connecting words; French words for “as well”. Similarly, one learner of Spanish, after asking for an essay plan, prompted ChatGPT to provide “some connectives or phrases” in the target language to be used at the beginning of each paragraph. The other participant in Spanish initiated the dialogue with ChatGPT by asking “Can you write essays in Spanish?” Subsequently, the student entered the writing task prompt in the chat and copied and pasted the entire essay generated by the machine, without making any modifications. Arabic students did not use ChatGPT to produce texts for them at all. One student used it to answer grammar and vocabulary questions, and the other one used it to translate phrases and to answer questions. A minority of students (3 out of 18) reported that using ChatGPT to complete this writing task required less time compared to their usual writing tasks. These various experiences suggest that not all students understood how to use ChatGPT efficiently for learning. Indeed, many participants did not have any prior experience of ChatGPT, and the data challenged some of the expectations we had about students’ digital literacy, considering their “digital nativeness” (Ng 2012). It revealed that students had varying levels of digital skills and several of them struggled with prompt writing, as will be discussed below. This skill is essential for the optimal use of ChatGPT and the difficulty in forming effective prompts had an impact on the participants’ perception of ChatGPT (Tables 1–6).

Participants were asked to compare the feedback provided by their teachers and by ChatGPT. While several participants noted that ChatGPT’s feedback was helpful, some students thought it was not helpful at all. Participants enjoyed the immediacy of ChatGPT’s feedback, with one participant in Chinese noting that they appreciated ChatGPT’s “unlimited patience” in providing detailed, sentence-by-sentence analysis, including grammatical mistakes and improvement suggestions (see also Xiao and Zhi 2023). However, overall participants preferred the feedback they received from teachers. They felt that the feedback provided by ChatGPT was often too vague and generalising as well as potentially inaccurate. Several participants observed that ChatGPT cannot tell whether an expression is idiomatic and were disappointed to find that the revised version of their essays still contained mistakes, as pointed out by the teacher in their feedback. Similarly, ChatGPT appeared to have a “people pleasing” tendency and its feedback tended to be overly generous. For example, one of the participating students in French asked ChatGPT to mark their essay. The original essay received a mark of 90 %. The participant then submitted a revised version of the essay and asked ChatGPT to assess it, based on the prompt: “How accurate is the French in this essay?” ChatGPT provided detailed feedback with a slightly lower mark of 85 %. While the student was very pleased to obtain such high marks, they observed that they would not be able to achieve such marks at university and that it would be helpful to supply ChatGPT with a marking scheme, to receive more accurate feedback.

4.1.2 Students’ reflection on their own task competence

When reflecting on their ability to complete tasks with or without the use of GenAI, some respondents expressed that they tended to be less meticulous when using GenAI because it addressed grammatical issues and offered linguistic variation. However, several participants also observed that ChatGPT occasionally made errors and tended to satisfy users when being questioned, so they emphasised the importance of being critical and analytical when integrating GenAI into writing tasks.

In terms of motivation, most students reported feeling more motivated when using ChatGPT because it improved their performance and increased the speed at which they could complete the task. This echoes previous studies conducted about students’ motivation and engagement (Nghi et al. 2019; Klímova and Ibna Seraj 2023). For example, one student who was studying Japanese mentioned that ChatGPT helped alleviate their anxiety, because they lacked confidence in learning foreign languages, especially in writing. However, some respondents expressed feeling less motivated when using ChatGPT. One, who was studying German, stated that,

“[…] by the time I’d finished writing the second one with ChatGPT I just thought, I don’t really want to do this anymore. I felt a bit demotivated and like my work wouldn’t be good enough without ChatGPT. […] I feel more proud of the work that I did by myself.”

Some participants had a more neutral position on GenAI. For example, a student studying Italian said:

“ChatGPT maybe impact my confidence in writing in English, […] but it does not substitute my effort to improve my skills. […] it can strengthen my confidence, but not determine it.”

Overall, many respondents acknowledged that ChatGPT has supported their language learning process by promoting self-critique via immediate and tailored feedback, which encouraged active engagement and reflection in their language studies. Such engagement may, in turn, help learners develop their autonomy, defined by Palfreyman (2014:182) as:

the capacity for intentional use in context of a range of interacting resources toward learning goals. (…) The autonomous learner will identify in her environment resources relevant to her purposes, make effective use of these, be open to new affordances in her environment and be able to adapt to changing circumstances by seeking out new resources or adopting new ways of using them for learning.

However, similar to other studies on the topic (e.g., Szabó and Szoke 2024), our findings affirm that care must be taken when guiding students in the use of GenAI, as some participants were not necessarily aware of their overreliance on ChatGPT to construct their sentences and texts, at the risk of bypassing learning and distancing themselves from their own voice in the target language. Learners must be equipped with the skills to be critical and analytical about the work generated by GenAI, and to understand GenAI systems’ tendency to hallucinate (Román Mendoza 2023; Szabó and Szoke 2024). Furthermore, students should be provided with foundational knowledge about how large language models work, so they can recognise that making mistakes-including errors that a GenAI system might not make-is a natural and essential part of language acquisition, and one that has to do with authenticity and finding one’s own voice in a new language (van Lier 1996). Indeed, mistakes should be embraced as part of the learning process. GenAI, therefore, should be regarded as a supportive learning and guidance tool to assist learners, rather than as a tool with which to compete.

4.1.3 Students’ reflection on their experiences of GenAI

In reflecting upon their experiences using ChatGPT for writing tasks, the majority of students expressed a willingness to continue using it for work and study. However, some identified specific scenarios in which they would refrain from using ChatGPT. For example, a student who was studying Russian said that they would avoid using ChatGPT for translation because completing translation tasks independently was more beneficial for their language learning. They also indicated that they would neither rely on ChatGPT for literary analysis, citing concerns about the trustworthiness of its information sources, nor generate full sentences or essays, saying, “You could potentially just stop thinking, […] and […] not make any progress”.

Indeed, 11 out of 18 students reported challenges with prompt formulation while using ChatGPT for writing tasks. For example, some who were studying Japanese mentioned that they “could only come up with limited prompts” and encountered difficulties in crafting the right prompt, providing detailed instructions to improve their essays, or adjusting the prompt when the ChatGPT output was unsatisfactory. A respondent studying French expressed a similar concern, stating: “It was really annoying, because I couldn’t figure out what to say to get the answer I wanted. But when I did, the answer was good”.

Most of the participants stated that their experience using ChatGPT had provided them with valuable insights into GenAI’s capabilities, particularly to support the development of their grammar and writing skills. One student learning Russian shared that “it was a mind-blowing discovery that everything’s better with ChatGPT”, referring to the end product. However, several students also voiced concerns and negative views about the use of GenAI in language learning. Half of the students felt that the work produced through ChatGPT was not their own and believed that it was more harmful than beneficial in terms of learning retention.

4.2.1 GenAI capability

As noted, several participants were impressed by ChatGPT’s ability in the target language. One of the participants in French was surprised to find out that ChatGPT could write a better essay in French than she could, but across languages, most participants believed that ChatGPT had better proficiency than them. For instance, one of the participants in Chinese, while acknowledging their own proficiency limitations, found the ChatGPT’s Chinese output to be fluent. The student suggested that this was likely due to the abundance of Chinese training data available. On the contrary, participants in Arabic did not use ChatGPT as a writing tool, as they had doubts about its linguistic ability. At the same time, students noted that while ChatGPT was often better than they were in terms of grammatical accuracy, it was not always helpful in terms of creating interesting content. Participants observed that ChatGPT often repeated structures and ideas and was not capable of generating in-depth content, only formulating vague answers. One of the participants in French suggested that ChatGPT was helpful for writing short essays but was too generic for literary analysis and more substantial pieces of writing. Students additionally mentioned that ChatGPT lacked cultural awareness and personality. For example, a participant in Spanish noted that “[i]t lacks nuances in terms of […] how normal Spanish people speak”. Similarly, one of the participating students in Chinese expressed uncertainty about ChatGPT’s cultural sensitivity, particularly regarding nuanced word choices and context-specific language use.

When comparing ChatGPT to other AI tools, participants overall thought that ChatGPT was better than Google Translate, as well as other translation tools. Participants enjoyed how comprehensive ChatGPT is, combining several tools (used for paraphrasing, translating, looking for synonyms, etc.) in one. Several participants noted how fast and convenient it is to refine their own language use. However, participants expressed their concerns, including in the initial benchmark survey, about over-reliance on GenAI. As mentioned earlier, the researchers had access to participants’ saved chat histories with ChatGPT, which revealed how they had utilised the tool to complete the writing task and obtain feedback. These were valuable empirical data that complemented the participants’ reflections and comments gathered through the other data collection methods. For instance, there were notable differences in how students acted on the afore-mentioned shared concerns in their actual usage of ChatGPT, as their chat history revealed that some students relied heavily on ChatGPT to generate ideas and structure as well as well as correct and even translate in the target language.

In addition, several participants noted that ChatGPT could not replace human teachers. A participant in Chinese believed that ChatGPT cannot completely replace language teachers, but that it can handle some basic tasks. They thought ChatGPT was capable of delivering grammar lessons, explaining new constructions, and providing examples. However, for more complex tasks like improving language use in context, understanding subject nuances, or exploring deeper linguistic concepts, the student felt that a teacher or trained linguist would be superior. To our surprise, participants did not mention the risk of obtaining hallucinations when prompting ChatGPT for grammar explanations (see Román Mendoza 2023). Similarly, a participating student in Spanish noted that the human element is essential in language learning: “it does take out the human and cultural element of it. I think one of the things I’ve loved learning about my target language is also […] the cultural accumulation, the learning of the cultural history of the language with it.”

4.2.2 Ethics of using GenAI

Participants across languages shared concerns about plagiarism and the need for clear university guidelines on the ethical use of GenAI. In addition, participants expressed their concerns around ethical questions of authorship, ownership and authenticity. A participant in French expressed her opposition to using ChatGPT for marked assessment, while at the same time feeling confused about ethical boundaries:

“I definitely wouldn’t use it for […] an essay for class that we’re gonna mark in class, […] I don’t think it’s appropriate. It is cheating, because it’s not your own ideas, but also that being said […] you’re allowed to quote research papers and stuff in your exams. So, is that not the same thing? Oh, I don’t know, actually, because I guess I copy other people’s ideas, anyway by using like journal articles so sorry. I don’t know what my answer is.”

At the same time, ChatGPT was also perceived as a source of confidence by some participants. The knowledge that students can write without making mistakes was perceived as a positive part of the experience. Indeed, the question of confidence and trust appeared repeatedly in students’ responses. Students’ trust in GenAI’s ability was mixed. Some participants highly trusted ChatGPT. For instance, a participant in Chinese rated their trust in ChatGPT at 7 out of 10, expressing higher confidence in its content knowledge and cultural explanations than in its ability to generate or translate full texts. The participant viewed GenAI as an educational tool comparable to dictionaries or textbooks, maintaining that work produced with GenAI assistance was still fundamentally their own. Other participating students, notably in Russian studies, had no trust at all in ChatGPT. Conversely, some participants did not have much trust in their own language skills and felt that ChatGPT could do better than what they can achieve.

Hence, while ChatGPT can be seen as a valuable tool for enhancing confidence and language accuracy, participants displayed divided perceptions of trust in its use for language learning. The lack of clear guidelines on its ethical use can generate confusion amongst students, particularly regarding its role in assessments and the boundaries of authorship and authenticity. In this regard, although his work predates the digital age, the three conditions identified by van Lier (1996) for developing one’s own voice in a foreign language may still be relevant today: awareness of language and learning; autonomy and self-determination in using the language and in the learning processes, and authenticity in communicative events. The dilemma for many is how to determine the level of authenticity of a text written with the help of GenAI. While some of the participants viewed ChatGPT as a supportive tool, others were skeptical of its cultural and linguistic reliability or worried about its impact on their independence and skills. As noted above, these mixed reactions highlight the need for institutions to establish clear guidelines and promote a balanced approach, as also suggested by Xiao and Zhi in their study of students’ perception of ChatGPT (2023), emphasising ChatGPT as a complementary tool rather than a replacement for personal effort and critical thinking (Leal and Torres 2023).

4.2.3 Future of GenAI in language learning

Regarding the hypothetical use of GenAI in language learning, one-third of students highlighted its potential to support their language studies as a tutor. For example, one student who was studying Spanish suggested that submitting a selection of recently written essays to GenAI, which could summarise the recurring errors that would typically be identified by a teacher, and then generate tailored exercises for the student, could be advantageous. This approach was described as highly beneficial for fostering independent learning. A different student, who was studying Italian, emphasised the use of GenAI for improving their grammar and identifying resources for practising specific grammatical structures. Similarly, one-third of the students expressed that utilising GenAI for speaking practice would be useful, particularly for improving pronunciation (see Klímova and Ibna Seraj 2023). Other students noted the potential of using GenAI to generate tailored exercises, such as vocabulary tests.

Students evidently understand their language learning needs and that GenAI could be a tool to support their studies. However, caution must be exercised to avoid over-reliance on GenAI in the language learning process (Darwin et al. 2023). For example, one student, who was learning Chinese, mentioned that they hoped to use GenAI to summarise texts when pressed for time during reading exercises. While GenAI could potentially improve reading efficiency for learners, it may also deprive students of opportunities to develop and strengthen their reading skills. Some students suggested that teaching how to use GenAI by offering introductory classes on its applications would be helpful. We also believe that providing training for teachers on how to guide students in appropriately and effectively using GenAI to support their learning is essential.

As technology continues to develop at an unprecedented pace, the integration of GenAI in language teaching and learning has become inevitable. It is encouraging that some students have already recognised both the advantages and the limitations of GenAI. To guide students in using GenAI effectively, responsibly and ethically for independent learning, it is crucial for teachers to emphasise the importance of actively engaging with the learning material and environment to ensure that GenAI is utilised as a tool to support their learning, rather than a quick solution that replaces their cognitive processes and effort. Furthermore, it is also essential to equip students with the necessary skills for prompt engineering so that they can craft precise prompts, increase their learning efficiency and gain more meaningful insights while interacting with AI. Following this research project, several members of the team have been involved in the development of guidelines in MLAC and CFLS to support students on the ways in which they can and cannot use GenAI in completing their tasks and how to acknowledge its use when permitted. The guidelines clearly establish forbidden uses of AI (such as plagiarism and direct submission of AI content or as a redrafting tool of the students’ own writing) and encourage students to use AI as a tool to support their own independent thinking and language learning process, through idea generation, dialogue and translation assistance. They also state that any use of AI should be explicitly referenced. The guidelines seek to address ethical concerns as well as students’ need for guidance and, similarly to our research project, they engage proactively with AI as a language learning tool.