Enhancing advanced vocabulary in EFL writing: an AI-assisted intervention for English studies students in Poland

2.1 Advanced vocabulary: conceptual considerations

The term “advanced vocabulary” does not have a precise definition (Juanggo 2018; Leńko-Szymańska 2019). Advanced – or sophisticated – words are generally conceptualised as low-frequency, rarely used words (Crossley 2020; Kim et al. 2018). These words go beyond the 2,000 most commonly used words; they are determined based on word frequency data from corpora (Leńko-Szymańska 2019) and are typically found in specialised fields, e.g., academic texts (Crossley 2020). Apart from frequency, there has been limited consensus on the definition of advanced (sophisticated) words. Kim et al. (2018), for example, extend that definition and argue that advanced words are “words that are less frequent, less concrete, less imageable, less orthographically and phonologically dense, more specific, exposed and acquired at a later age, and processed more slowly” (p. 137). In addition to this, the conceptualisation of advanced lexical items has been expanded to include advanced bigrams and trigrams (Crossley 2020; Crossley and Kristopher 2018; Kim et al. 2018; Kyle and Crossley 2016). Yet, it is important to note that while relying solely on word frequency in defining advanced vocabulary has been criticised, corpus-based frequency information often remains a key classification criterion (Leńko-Szymańska 2019).

The CEFR (Council of Europe 2001, 2018), which establishes standards for L2 teaching and testing not only in Europe, does not explicitly define the term “advanced vocabulary” either. Rather, it provides criteria that describe vocabulary range and control at various proficiency levels. Specifically, the vocabulary range scale describes language users’ or learners’ ability in terms of “the breadth and variety of expressions used” (Council of Europe 2018, p. 132), while the vocabulary control scale denotes users’ or learners’ ability “to choose an appropriate expression from their repertoire” (Council of Europe 2018, p. 134), both of which, as Leńko-Szymańska (2019) points out, can be related to language proficiency complexity and accuracy. The C1 level, known as “Effective Operational Proficiency,” is considered advanced as it “represents an advanced level of competence suitable for more complex work and study tasks” (CEFR 2001, p. 23; present author’s emphasis). With this in mind, Tables 1 and 2 display the vocabulary scales for the C1 proficiency level, alongside the B2 and C2 levels for comparison.

As demonstrated above, the C1 level user exhibits a command of a wide range of vocabulary, including idioms and colloquialisms. Furthermore, while B2 level users can utilise the specialised vocabulary, C1 level users are able to employ a range of technical vocabulary. Notably, the C1 level user displays the command of less common vocabulary in a manner that is idiomatic and suitable, which is not mentioned for the B2 level user. This shows the differences between proficiency levels concerning the quality of vocabulary. However, the analysis also reveals that there exists ambiguity regarding the exact definition of “less common vocabulary” expected from C1 level learners and the differentiation between technical and/or specialist words deemed suitable for C1 as opposed to B2 levels. These uncertainties give rise to practical challenges, making it difficult for instructors and examiners to effectively teach and assess vocabulary in accordance with CEFR guidelines.

2.2 CEFR-based vocabulary-oriented tools

The use of existing CEFR-based tools can offer substantial relief to instructors and examiners involved in English language teaching. Tools such as the English Vocabulary Profile (accessible at https://www.englishprofile.org/) or the Oxford Advanced Learner’s Dictionary (accessible at https://www.oxfordlearnersdictionaries.com/) serve as valuable resources, providing precise information about word proficiency levels. Importantly, these tools enable instructors and learners to ascertain the words and phrases that are known and utilised by language users at various, including advanced, CEFR levels. However, despite the usefulness of these tools, their integration into EFL writing courses presents its own set of challenges. The process of individually checking word proficiency levels using these tools can be very time-consuming for both teachers and learners. Moreover, the need to assess numerous texts written by multiple students throughout the course makes frequent word proficiency level checking impractical.

The CEFR Checker developed by Cathoven A.I. (2023) offers a solution addressing the aforementioned problems. The CEFR Checker is a text analyser that utilises artificial intelligence, i.e. natural language processing, techniques. This tool examines the structure of sentences, verb tenses, and vocabulary employed in a given text, making it possible to assess and estimate the corresponding CEFR level (Cathoven n.d.). What is important, however, information about its accuracy or reliability is not provided by developers at the time of writing this paper.

Moving forward, a question arises: how can the CEFR Checker be incorporated in the L2 classroom as a teaching tool to enhance students’ awareness of advanced words and their ability to apply them in their writing? In order to address this issue, the next sections overview existing research on awareness-raising vocabulary instruction for improving the quality of texts written by L2 learners as well as research on AI-based tools for vocabulary instruction.

2.3 Research on awareness-raising vocabulary instruction for quality writing

Given the lack of research on the efficacy of technology-assisted instruction for raising the awareness of word proficiency levels and advanced vocabulary use in writing among higher proficiency L2 learners, this study builds on insights from previous research on explicit instruction. Specifically, it draws from studies focused on enhancing learners’ awareness of collocations (Oskuee et al. 2012), formulaic sequences (AlHassan and Wood 2015; Liou and Wen-Feng 2018), and academic lexical phrases (Cai 2016). This strand of research provides a foundation for exploring how similar explicit instructional strategies can be applied using technology to support advanced vocabulary development.

In one study, Oskuee et al. (2012) aimed to understand the effect of pre-teaching vocabulary and collocations on writing development. The researchers made the assumption that learners can enhance their writing skills gradually over time, as a result of receiving instruction, feedback, and engagement in regular practice. Forty advanced EFL learners in Iran who completed intermediate and upper-intermediate courses were randomly assigned to either a control or experimental group, and pre-test and post-test compositions were administered to both groups. The learners took part in twenty sessions of instruction and wrote five texts. In the experimental group, the learners were engaged in a treatment involving the selection of a topic of their interest, receiving instruction on relevant vocabulary related to various topics, and being provided with the phrases that were regarded as useful. They were also given exercises that allowed for vocabulary and collocations practice. After that, the learners were assigned a writing task that required them to write about a specific topic. The results indicate that pre-teaching vocabulary and collocations can be an effective strategy for enhancing students’ writing quality, as there was noticeable difference between the two groups.

In the second study, AlHassan and Wood (2015) sought to investigate the effect of focused instruction of formulaic sequences on academic paragraph writing skills. Twelve participants – learners of English in Canada with different proficiency levels, ranging from lower-intermediate to upper-intermediate and advanced – were involved in the study. For a duration of ten weeks, the participants were taught formulaic sequences for 90 min per week. The researchers created lists of thematically arranged phrases using, among others, the Corpus of Contemporary American English. Worksheets were prepared to introduce the participants to the desired formulaic sequences and collocations. Explicit instruction was centered on awareness-raising, which involved the presentation stage, practice including activities such as writing introductory/concluding sentences using sentence frames or completing matching tasks, and the production stage, during which the participants were required to apply the target formulaic sequences in sentences. Three paragraphs – summaries written in response to the prompt – were produced by each participant, which were next rated by three judges. According to the first judge, there was a rise in scores across the three writing sessions. However, in two other judges’ assessment, there was a moderate increase in mean scores between the pre-test and post-test, followed by a slight decrease in mean scores for the delayed post-test, although the latter still remained higher than the pretest mean. Based on their results, the researchers conclude that when an explicit instructional approach involving the presentation, practice and production stages is implemented for formulaic sequences, it can lead to an improved acquisition of the sequences.

Similarly, Liou and Wen-Feng (2018) conducted a study to examine the effects of teaching academic formulaic sequences on writing skills. The participants were fifteen EFL third-year English-major students taking a writing course at a private university, who had been learning English for a minimum of eight years. In this study, the explicit instruction was developed based on the principles of noticing, retrieving, and generating. Using these principles, the researchers created five-week instruction that focused on fifty specific formulaic sequences selected from five academic formula lists. The explicit instruction consisted of presentation, as well as practice and production activities. During the presentation stage, the participants were taught the meaning, usage, and functions of the targeted formulaic sequences using example sentences and online corpus sources. Following the presentation stage, the participants were assigned in-class tasks such as constructing sentences or completing gap-filling and multiple-choice exercises using the formulaic sequences they had learned. In order to help the participants remember the formulaic sequences, the researchers gave them extra materials and assigned homework in which the students wrote a paragraph using at least three formulaic sequences they had learned. Additionally, they were required to read texts, write summaries, book reports and research reports. Research data were drawn from tests, writing tasks, and a questionnaire. It was found that explicit instruction led to an increase in the number of formulaic sequences used in the participants’ writing, with 60 % of the taught sequences being present in their written work.

Finally, Cai (2016) incorporated genre-based pedagogy and corpus-informed explicit instruction in the teaching of academic lexical phrases. The course was structured into four units, each focusing on a different part of research articles, i.e. Introduction and Review, Methods, Results, and Discussion and Conclusion. This study was conducted in China with Masters students over 15 weeks, each lesson lasted 70 min. The academic formulas list and language learner corpora were the sources from which the lexical phrases were selected. Data were gathered via receptive knowledge test, genre and lexical phrases awareness test, knowledge scale, as well as the rewriting task. The findings showed that the participants made considerable progress in the understanding of lexical phrases, and learners with varying proficiency levels achieved comparable levels of proficiency after receiving instruction.

This body of research offers valuable insight into interventions aimed at improving the writing skills of advanced language learners through explicit instruction and the enhancement of their understanding of specific lexical items. The findings generally suggest that exposing learners to such vocabulary and explicitly teaching these phrases can increase their awareness and ultimately led to the incorporation of those into their own texts. However, there is a lack of empirical investigation into instruction focused on developing learners’ knowledge of word proficiency levels. Consequently, there is limited understanding about the pedagogical practices that can enhance advanced vocabulary recognition and use in EFL writing classrooms.

2.4 Research on AI-based tools for vocabulary instruction

Advances in AI technology have created new opportunities for L2 vocabulary instruction. Consequently, the CALL field has seen growing interest in integrating AI-based tools to support vocabulary learning, particularly among EFL learners. While research in this area is still emerging, initial studies have yielded promising results in terms of facilitating student vocabulary learning and teacher-led vocabulary instruction.

To support student learning, a range of AI technologies, including conversational agents, AI-driven language learning mobile apps, image recognition, object detection, and immersive environments, have been implemented. Conversational agents (chatbots) facilitate vocabulary learning by engaging learners in conversations during which learners can practise word meaning and use (Yildiz 2023). For instance, ChatGPT and Poe, by affording real-time dialogue interactions, enable students to learn new expressions, refine vocabulary to communicate with more precision, and build confidence in language use (Karataş et al. 2024; Pham et al. 2024; Yeh 2024). In addition, mobile apps such as Duolingo, Babbel, or NovoLearning adjust content based on learner performance (Arini et al. 2022; Liang and Zhang 2024; Zhao et al. 2024). Next, AI-supported image recognition and object detection technologies are used to contextualise vocabulary learning by connecting words with visual prompts, which helps learners associate vocabulary with real-world contexts (Hsu et al. 2023; Liu and Chen 2023). Lastly, immersive AI-extended reality (AI-XR) environments create conditions where learners engage in authentic conversation practice (Divekar et al. 2022).

AI also holds benefits for teachers as this technology allows for more efficient vocabulary instruction through the use of AI-powered platforms, chatbots, adaptive teaching materials, and computational thinking principles. Platforms, such as UNIPUS AIGC and iTEST, provide customised learning support and assessment (Wang et al. 2024). As to chatbots integrated into classroom activities, such as those built using Google’s Dialogflow, those assist teachers in dynamic vocabulary assessment by providing graduated assistance (Jeon 2023). AI applications also support teachers in designing adaptive materials (Yeh 2024). Finally, interventions based on computational thinking principles have been employed to improve vocabulary use in essay writing (Tang and Ma 2024).

Taken together, although the reviewed studies demonstrate the potential of AI-based tools to enhance vocabulary learning, the benefits of applying these tools to support advanced vocabulary development in writing remains underexplored. Current studies have not addressed the issue of how AI can support learners’ word recognition and use in writing in alignment with CEFR standards. Moreover, there is a need to investigate the role of AI technology in raising L2 learners’ awareness of word proficiency levels. Accordingly, this study bridges these gaps by investigating the effectiveness of an AI-assisted intervention using the CEFR Checker to enhance B2 level students’ awareness and use of C1+ and B2 level vocabulary in their writing. By employing the CEFR Checker, an AI-based text analyser, this study aims to provide empirical evidence on how AI-assisted instruction can improve students’ ability to recognise and incorporate advanced vocabulary into their writing. The investigation is guided by the following research questions (RQs):

3.1 Participants

Participants were selected through convenience sampling from first-year BA full-time and extramural English Studies students enrolled in a Writing course, an obligatory component of the Practical English curriculum at a university in Poland. Initially, 103 students were enrolled in the course across four groups. However, only data from students who provided informed consent for the use of their coursework at the end of the course were analysed. Consequently, the final sample consisted of 71 students with the number of participants in each group varying across the sessions due to irregular student attendance (see Table 3 for detailed participant information).

The students’ English language proficiency was approximately at the B2 level, as defined by the Common European Framework of Reference for Languages (CEFR) (Council of Europe 2001). This proficiency level was demonstrated by their performance on the extended B2 level secondary school leaving examination, which is a prerequisite for admission to the university programme. The CEFR, widely adopted in Europe and beyond to guide language curricula, examinations, and coursebooks, serves as a reference framework for this study due to its integration into the Polish educational system. Poland, as a member of the European Union, has aligned its foreign language education policies with the standards set in the CEFR, which defines language proficiency goals for each stage of education.

3.2 Instruments

Two tasks were used in this study: a summary writing task and a word recognition task. In the summary task, students were asked to write summaries of articles they had read. This task allowed the researcher to analyse the students’ use of vocabulary, specifically focusing on the occurrence of B2 and C1+ level words. Each instance of a B2 or C1+ word used in the summary was awarded one point. The word recognition task required students to identify three B2 and three C1+ level words in their own writing. This task provided data on students’ ability to recognise these words in their texts. Students could earn up to three points for correctly identifying B2 words and up to three points for C1+ level words.

The CEFR Checker (Cathoven A.I. 2023) was used during the intervention to identify B2 and C1+ level words in the students’ texts. The tool was available without usage limitations or a subscription fee, presenting advantages in terms of time efficiency when determining word proficiency levels according to CEFR standards.

3.3 Intervention

Following the introductory session, the intervention took the form of five sessions integrated into the regular coursework. Each session consisted of three phases: 1) exposure to written texts containing vocabulary that corresponds to the target level (C1+), 2) the use of C1+ words in writing, and 3) explicit instruction on word proficiency levels with regard to C1+ versus B2 levels (see Tables 1 and 2 for the vocabulary scales corresponding to the B2, C1, and C2 proficiency levels). Those activities were considered optimal, following research results concerning the challenges in advanced writing (Hou 2017; Maamuujav et al. 2021; Malmström et al. 2016; Rokoszewska 2021; Sajjad and Esmat 2023) and explicit instruction of collocations (Oskuee et al. 2012), formulaic sequences (AlHassan and Wood (2015); Liou and Wen-Feng 2018), and academic lexical phrases (Cai 2016). Each stage is described in more detail below.

Introductory session. The intervention was initiated by the researcher-instructor discussing the quality of vocabulary the students are expected to produce, what vocabulary is considered advanced (C1+), and how it differs from B1 and B2 levels. The students were then displayed an online text (accessible at https://thinkingallowedjournal.weebly.com/university-life1/staying-motivated) written by a peer taking the course a year earlier, and were asked to find three B1 words, three B2 words, and three C1+ words. Afterwards, the instructor showed the text with words annotated according to their proficiency level using the CEFR Checker. B1 words were displayed in light blue, B2 words were in dark blue, C1 words were in yellow, C2 words were in orange, and words beyond the C2 level were in red (Figure 1). The students were instructed to check the accuracy of their answers.

Figure 1:

Introductory session. The text annotated with the CEFR checker displaying word proficiency levels.

Exposure to written text. Prior to class, the students were asked to read a free article from the Nature, New Scientist, or Scientific American websites. The websites were selected based on the assumption that these contain vocabulary that exemplifies the quality expected in the writing of advanced EFL learners (readability checkers were not employed for the measurement of text difficulty in this study). To ensure task completion, the students were asked to paste links to the articles of their choice in a specific space provided on Moodle – the Virtual Learning Environment platform provided by the researcher’s university for online/blended learning and teaching.

Using C1+ words in writing. In class, the students did a summary writing task, i.e., the students were instructed to write a paragraph summarising the article read prior to class for 10 min. The writing was performed on paper and the minimum word count was not required for the summaries.

Explicit instruction–noticing. Having finished writing, the students did a word proficiency level recognition task. Specifically, they were asked to list three B2 words and three C1+ words they used in their summaries below their text. The instructor collected the texts.

After class, the instructor typed the texts in MS Word, correcting spelling mistakes in the B2 and C1+ level words to enable their identification by the CEFR Checker. Next, each text was analysed using the CEFR Checker; B2 words were circled in blue and C1+ words were circled in orange by the instructor. The accuracy of the words provided by the students in the recognition task was also confirmed (Figure 2).

Figure 2:

A sample of student text with words labelled for word proficiency levels using the CEFR checker.

In the next class, the instructor returned the texts to the students, allowing them time to become familiar with the results. The students could seek clarification on any questions or concerns they may have had regarding their work. After that, the students worked in small groups to complete two practice tasks. First, each student individually called out the words they used in their text for which other students were to supply word proficiency levels (B2 or C1+). The student who called out the word would then confirm the accuracy of the provided level. Next, the students would take turns in their group to recall the B2 (round 1) and C1+ words (round 2) used by themselves and their peers.

3.4 Data collection and analysis

Data were collected over five sessions within one winter semester (October-January). The full-time group (n = 1) completed all the sessions by December, whereas the extramural groups (n = 3) took a break over the Christmas period and had a final session in January. The summaries were analysed for the use and recognition of B2 and C1+ level words at the end of the intervention.

The analysis included both descriptive and inferential statistics. Descriptive statistics (frequency, means, standard deviations) were calculated for the number of B2 and C1+ words used and recognised in each session. As to inferential statistics, paired t-tests were conducted to compare performance between the first and last sessions. Pearson correlations were calculated to examine the relationship between word use and word recognition for both B2 and C1+ levels. Effect sizes were also calculated to determine the magnitude of observed changes; the Pearson correlation coefficient was used to provide effect size values, interpreted as small (r = 0.25), medium (r = 0.4), and large (r = 0.6) (Plonsky and Oswald 2014). The analyses were conducted using GNU PSPP (PSPP – GNU Project 2013).

4.1 Use of C1+ and B2 level words in the summaries

Table 4 shows that the mean number of C1+ level words applied in summaries increased from M = 4.74 (SD = 3.26) in Session 1 to M = 5.98 (SD = 3.68) in Session 4. Although there was a slight decline in the mean number of words used in Session 5 to M = 5.26 (SD = 2.98), this was still higher than the mean number of words used in Session 1. A paired-samples t-test was conducted to determine whether a statistical difference existed in the number of C1+ words used by the students between the first and last sessions. The results did not reveal a statistically significant difference (t (63) = −0.93, p = 0.358) between the number of C1+ words used in Session 1 and Session 5, and the effect size was small (r = 0.2).

Table 5 presents the distribution of the number of students using C1+ words in their summaries across the sessions. In Sessions 1 and 2, many students applied up to 7 words in their texts, but rarely used more than 8 words (though there were 5 students who used 9 words in session 2), and none of them used 15 words or more in their texts. In addition, in Session 1, there were three students who did not apply any Cl+ level words. In Sessions 3 and 4, there were a few students who applied 8 or more words, with one student applying 20 words in their text in Session 3. In the last session, there were no students who displayed the use of 0 words.

As presented in Table 6, the mean number of B2 level words used in summaries increased from M = 4.87 (SD = 2.33) in Session 1 to M = 6.27 (SD = 2.67) in Session 4 (Table 5). While there was a decrease in Session 5 (M = 5.5, SD = 2.63), the mean number of words was higher than the mean number of words used in Session 1. The results of the paired-samples t-test did not reveal a statistically significant difference (t (63) = −2.06, p = 0.44) between the number of B2 words used in the first and last sessions, but the effect size was medium (r = 0.5).

Table 7 presents the distribution of the number of students applying B2 words in their summaries across the sessions. In Session 1, while many students applied up to 7 words, they rarely used more than 8 words in their texts, albeit one student used 13 words in their text. In Sessions 2–5, there were slightly more students who used 8 or more words, with one student applying 14 words in their texts in Session 2 and 4.

4.2 Recognition of C1+ and B2 level words in the summaries

As displayed in Table 8, the mean number of C1+ level words recognised in summaries increased from M = 1.04 (SD = 0.99) in Session 1 to M = 1.73 (SD = 1.00) in Session 4, with a slight decline in Session 5 (M = 1.72, SD = 1.02) (Table 7). The results of the paired-samples t-test revealed a statistically significant difference (t (62) = −3.97, p = 0.00) between the number of C+ words found in Session 1 and Session 5. The effect size was small (r = 0.3).

Figure 3 displays the distribution of the number of students who recognised C1+ words in summaries across the sessions (Figure 2). The number of students not recognising any words decreased, from 24 students in Session 1 to 8 students in Sessions 4 and 5. In addition, the number of students recognising 3 words increased, from 8 students in Sessions 1 and 2 to 18 students in Session 5.

Figure 3:

Identified C1+ level words – distribution of the number of students by number of words identified per session.

Table 9 indicates that the mean number of B2 level words identified in summaries increased from M = 0.91 (SD = 0.90) in Session 1 to M = 1.09 (SD = 0.85) in Session 5. However, there was a small decrease in Session 4 with M = 0.94 (SD = 0.77). The results of the paired-samples t-test did not reveal a statistically significant difference (t (62) = −1.24, p = 219) between the number of B2 words found in Session 1 and Session 5, and the effect size was small (r = 0.033).

Figure 4 presents the distribution of the number of students who recognised B2 words in their summaries across the sessions. Sessions 1–3 presented a similar distribution of words recognised, many students identifying 0 words or 1 word, a few of them identifying 2 words, but very few identifying 3 words. In Session 4, the majority of students found 1 word, and, at Session 5, there were more students who identified 2 words, compared with previous sessions. Moreover, the number of students recognising 0 words slightly decreased across the sessions, from 26 students in Session 1, to 17 students in Session 5.

Figure 4:

Identified B2 level words – distribution of the number students by number of words identified per session.

4.3 Changes in the word use-recognition relationship over time

Word use and recognition were positively correlated, as presented in Tables 10 and 11. For the C1+ level words, all the correlations were strong and statistically significant, the strongest being in Session 1 (r = 0.677) and second strong in the final session. With regard to the B2 level, the correlation was the strongest and statistically significant in Session 1 (r = 0.485). The correlations in the next two sessions were far weaker and not statistically significant, though the value of the coefficient increased in the last two secessions, from r = 0.154 (p > 0.05) to r = 0.382 (p < 0.05) and was statistically significant.