Machine translation as a form of feedback on L2 writing

2.1 Evolution of TCF in L2 writing: effects and drawbacks

Effects of TCF have been controversial in the field of L2 writing since Truscott (1996) claimed its uselessness. However, subsequent empirical studies and recent meta-analyses and comprehensive reviews (e.g., Li and Vuono 2019) have shown that TCF is generally useful. For example, Kang and Han (2015) conducted a meta-analysis of 21 empirical studies and concluded that TCF does improve grammatical accuracy in L2 writing. These researchers also compared the effectiveness of various types of TCF, including direct versus indirect and focused versus comprehensive. Although the results were statistically inconclusive, they found that direct TCF is more effective than indirect TCF and that focused TCF is more effective than comprehensive TCF in terms of “substantial” (Li and Vuono 2019: 102) effect sizes. Despite these findings, we opted to use direct comprehensive TCF as the counterpart to MT because it is more similar to MT feedback than other types of TCF (e.g., indirect or focused) as it covers each learner’s entire L2 text.

Although TCF has been regarded as one of L2 “writing teachers’ most important tasks” (Hyland and Hyland 2019: xiii), it has its own drawbacks. Of these, as mentioned earlier, two of the most relevant to our study are its inefficiency and the possibility of appropriation. First, comprehensive feedback “can be exhausting for teachers to give and overwhelming for students to receive” (Ferris and Hedgcock 2023: 341), which explains why focused feedback is often recommended as a more realistic approach (e.g., Lee 2019). Yet, comprehensive TCF continues to be widely used as a result of students’ preferences, teachers’ beliefs, or institutional requirements (e.g., Lee 2008), especially in EFL contexts.

A second concern is the risk of appropriation, which has evolved over time since it was first problematized by Sommers (1982) as the teacher taking ownership of the students’ voice by rewriting their texts. In contrast, in addition to introducing five recent studies of dialogic appropriation (i.e., between teacher and student writer), Tardy (2019) demonstrated in a case study how appropriation can emerge from multiple dialogic interactions between mentor and writer, the latter being not only an L2 student but a writer with sociocultural obligations (e.g., having to fulfill a senior co-author’s intention outside the writing class). Though such appropriation processes may enable students to reflect their own voice in their writing, such interactions take up a great deal of time for both teachers and students.

2.2 Machine translation (MT) as a learning and teaching tool in L2 writing

Before 2016, when the field entered the era of Neural MT, MT output contained so many lexico-grammatical errors that most of it was categorized as “ill-formed examples of L2” (García and Pena 2011: 473) to be corrected in post-editing when used as teaching or learning aids in L2 writing. However, after 2016, many researchers started to view MT outputs as good models. In fact, Yulianto and Supriatnaningsih (2021) reported that free and fast MT platforms such as Google Translate and DeepL were arguably comparable with human translation if the two languages belonged to the same broad family (e.g., French and English). Among those studies examining MT as a tool for helping L2 writers after 2016, most relevant to our study are those that require students to write an L1 text, translate it manually into the L2 (Own L2 Text), translate their initial L1 text using MT (MT L2 Text), and finally edit their Own L2 Text by looking at the MT L2 Text. We operationally call this approach “comparative analysis” because in the final stage, students are asked to edit their Own L2 Text by comparing it with their MT L2 Text (Table 1).

One of these earlier and exemplary studies using comparative analysis was conducted by Lee (2020), who had 34 Korean university students with 75–90 TOEFL iBT scores describe a video clip in their L1, translate their text into English, compare their output by MT of their text into English, and finally edit the English text by comparing it with the MT version. Lee’s results showed a positive effect on the edited version of the English texts, which contained fewer lexico-grammatical errors. However, Lee’s qualitative data (interviews and reflection papers) indicated that the students were divided by their L2 proficiency in their appreciation of the value of MT as a learning tool. While lower-proficiency students found MT useful for improving their writing, higher-proficiency students only found it helpful for learning vocabulary but also found MT output inaccurate in longer texts and therefore inappropriate for academic purposes. However, this may be due to the still imperfect quality of MT when Lee conducted her study.

In a later study, Lee (2022) found that higher-level (university) students benefited more from such comparative analysis than lower-level students despite receiving help from (still imperfect though probably better) MT translation. In this mixed-methods study, Lee also reported that students at all proficiency levels found MT helpful to their revisions of various linguistic elements (e.g., vocabulary, grammar, expression) as well as to their efficient and individualized provision of translated texts.

Lee’s (2020) study was replicated in another mixed-methods study by Yang et al. (2023) with 31 Chinese EFL university students who were comparable to those in Lee (2020). Yang et al.’s results were similar to Lee’s in that the MT L2 texts improved in quality, especially in terms of lexical complexity. In addition, 90 % of the students found MT helpful in improving their writing ability even though they thought that MT translation was not perfect (see Klimova et al. 2023 for more studies using a comparative design).

2.3 MT as a feedback tool: rationale for the study

As explained in the previous section, studies using a comparative analysis design not only used MT like traditional TCF but also addressed the two drawbacks of TCF mentioned earlier. First, unlike traditional reformulation feedback, MT L2 texts more closely reflect L2 writers’ intentions as expressed in their original L1 writing. Further, the provision of MT is time-efficient as well as customized to the student’ needs because it is based on their Own L2 Text, thus better reflecting each student’s voice. Nevertheless, one critical difference between the comparative analysis design and past TCF studies is that effects of TCF are measured without reference to feedback itself during the revision session, thus making it impossible to ascertain whether any knowledge or skills gained from the TCF has become internalized, especially when framed by a cognitive perspective, as were most past TCF studies (see, for example, Peng et al. 2023). In other words, though past MT studies using the comparative analysis design showed that MT was useful as a learning and teaching tool, its effects as L2 writing feedback have not been adequately assessed. Further, though most studies using comparative analysis collect qualitative data from participating students regarding their perceptions of MT’s usefulness along with their affective reaction, none of them considered these questions in relation to other types of TCF. However, As Kahneman and Tversky (1979) suggest, people’s decision-making may vary when an object is considered in isolation versus when compared with alternatives. Thus, it is crucial to systematically compare MT with other types of feedback when we consider L2 writing classrooms where multiple feedback options may be offered.

2.4 Complexity, accuracy, and fluency (CAF) as quantitative measures of the effects of feedback

Recently, an increasing number of studies have treated TCF not simply as a grammar correction tool but as a way of facilitating broader language development through “learners’ engagement with feedback” (Manchón 2020: 14). This position is supported by studies that used not only accuracy as their measure (e.g., Benson and DeKeyser 2019) but also fluency (e.g., Criado et al. 2022), complexity (e.g., Valizadeh 2022), or all three CAF measures (e.g., Zhang and Cheng 2021). Among these measures, TCF was shown to have significant effects on accuracy, whereas the results of studies that investigated its effects on fluency and complexity have been mixed (e.g., Li and Vuono 2019).

2.5 Participant engagement as qualitative measures of the effects of feedback

As many scholars (e.g., Peng et al. 2023) have pointed out, engagement is one of the crucial determinants of TCF effectiveness. Following Ellis (2010: 342), we operationally define engagement as “how learners respond to the feedback they receive.” Compared to the effects of various types of TCF, how L2 writers engage with TCF has not been adequately investigated (e.g., Crosthwaite et al. 2022). Given Han and Hyland’s (2015) systematic and detailed classification of Ellis’s (2010) multi-dimensional framework for investigating engagement with TCF, the results of many subsequent studies support Ellis’s perspective but also reveal complex interactions between cognitive, behavioral, and affective engagement processing. However, the methods they employ (e.g., retrospective interviews) have their own demerits (e.g., memory loss, difficulties with quantification), which need to be overcome (e.g., Mackey and Gass 2022).

2.6 The study

On the basis of previous literature, we asked the following questions:

• RQ1: Do direct TCF and MT have different effects on revised L2 texts in terms of complexity, accuracy, and fluency?

• RQ2: Do the two feedback types have different effects on L2 writers’ engagement when examined from a multi-dimensional perspective?

• RQ3: Which feedback type do participants find more useful or motivating and why?

These three questions will be first answered quantitatively, and the quantitative results will be supplemented by qualitative data to help us interpret the quantitative data.

3.1 Participants

A sample of 23 participants were recruited from a second-year university English L2 writing class in Japan. Written consent was obtained from participants after they were informed that they were free to withdraw from the study at any time and that this would not affect their grades in any way. To confirm that sample size was adequate, we performed an a priori power analysis with G*Power version 3.1.9.6 (Faul et al. 2007) to estimate the number of participants needed to reach the recommended threshold of 0.80 with an alpha level of 0.05 (Cohen 1988). Using effect size (f) converted from Hedge’s g (0.68) as reported in Kang and Han’s (2015) meta-analysis of TCF, the analysis confirmed that our sample size reached the desired statistical power threshold because it was over 14, the necessary number of participants in the repeated measure ANOVAs (within-by-within design) employed here. The participants’ mean TOEFL ITP score was 511.3 (SD = 27.6), indicating that their English proficiency was at about CEFR B1 level.

3.2 Procedure

We used a within-participant crossover design (Lui 2016), in which participants were randomly divided into two conditions: MT first-TCF second (MT-TCF; 11 students: 7 females), and TCF first-MT second (MT-TCF; 12 students: 2 females). We chose this design because it would help minimize individual variation caused by two feedback types while addressing the ethical issue of providing treatment to only one group, which can be a problem in a simple experiment-control design.

Data collection took place over four weeks in the same classroom. The treatment was divided into four phases (Table 2):

3.3 Types of feedback

3.3.1 Machine translation (MT)

MT feedback consisted of the Japanese text the participants had written the previous week in addition to its translation into L2 English using DeepL (Figure 1). We chose the DeepL platform (https://www.deepl.com/translator) because several studies (e.g., Yulianto and Supriatnaningsih 2021) have reported that it was one of the best MT applications available at the time. We also added the comparison between the students’ original English text from the previous week and the MT version, with modified segments marked in light blue to make them readily comparable to the TCF feedback.

Figure 1:

Example of MT feedback.

3.3.2 Direct comprehensive teacher corrective feedback (TCF)

For English texts in TCF condition, errors were jointly coded following the 40-category classification in Sachs and Polio (2007). The two coders were the first author and an ESL instructor with a doctorate in applied linguistics and 20 years of EFL teaching experience. Inter-coder agreement for Sessions 1 and 2 combined was 0.92, and each disagreement was resolved through discussion. Once agreed, the number of errors was also used to measure accuracy in the L2 texts. As in Sachs and Polio, direct and comprehensive feedback with insertions marked in blue and deletions marked in red (Figure 2) was returned in that format to each TCF participant.

Figure 2:

Example of teacher corrective feedback (TCF).

3.4 Quantitative measures

To quantitatively compare the effects of TCF and MT (RQ1), we used the three above-mentioned CAF measures (complexity, accuracy, and fluency). For complexity, we adopted an aggregated component score that included clause complexity, phrase complexity, and syntactic sophistication (Kyle 2016). This was chosen over conventional measures such as mean length of T-units, which have been criticized as being unsuitable for measuring various stages of writing development (Norris and Ortega 2009). To calculate this aggregated component score, we used the Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC) developed by Kyle (2016). For the accuracy measure, we used the number of errors per 100 words, following Zhang and Cheng (2021), who evaluated the effects of comprehensive direct TCF on their participants’ academic writing development. We selected this measure because it better reflects accuracy in the revised texts than measures such as percentage of error-free clauses (Zhang and Cheng 2021) when many items may require correction in a single clause. Lastly, to gauge fluency, we measured the total word count produced within the allotted time.

3.5 Qualitative data

3.6 Data analysis

Before comparing the effects of the two feedback types in terms of CAF (RQ1), we examined data normality using a Shapiro–Wilk test as well as histograms (Figure 3). Due to non-normal variables, we used a robust 20 % trimmed-mean alternative to a two-way repeated measures ANOVA using Wilcox’s (2021) function. This analysis was accompanied by bootstrapping with 599 replicates for fair evaluation (Plonsky et al. 2015). The number of bootstrap samples (B) was 599, the default value appropriate for a 0.05 alpha level (Wilcox 2021). We selected these methods because no non-parametric alternatives to two-way within-by-within ANOVA are available for non-normal data (see Larson-Hall and Herrington 2010 for robust statistics in applied linguistics).

Figure 3:

Histograms of all variables.

R version 4.2.1 was used for all analyses. For reproducibility, all data and the R code used in this study are available in OSF (https://osf.io/emny7).

4.1 Research question 1: effects of TCF versus MT on CAF

Table 3 shows the descriptive statistics for the CAF measures before and after each feedback session. Due to non-normal variables (p < 0.05, Shapiro-Wilk), the table also reports the median and 20 % trimmed mean. Figure 3 shows histograms for each variable before and after the TCF and MT sessions.

Table 4 summarizes the results of the 20 % robust trimmed-mean test and its bootstrap-t method corresponding to a two-way repeated measures ANOVA for main effects and interactions (i.e., TCF or MT and time). Figure 4 visually represents these interactions, revealing no significant main effects for TCF or MT feedback.

Figure 4:

Interaction plots of two main effects. Note. Means and standard errors were calculated based on 20 % trimmed means. Error bars show 95 % confidence intervals.

Additionally, interaction between condition and time was not significant except for the bootstrap-trimmed means of accuracy (p = 0.049), which was close to the alpha level of 0.05. This suggests that neither condition significantly outperformed the other in either the pre- or post-CAF measures. However, significant main effects of time were observed in all CAF measures and were corroborated by a parametric two-way ANOVA.

To better understand these time effects, we conducted post-hoc tests using paired-sample robust t-tests with 20 % trimmed means. Figure 5 summarizes the trimmed-mean differences for each CAF measure in TCF and MT conditions.

Figure 5:

Post-hoc trimmed-means difference for each CAF measure in TCF and MT conditions. Note. Error bars show 95 % confidence intervals.

As regards complexity, a significant mean difference was observed in TCF condition, Mdiff = 0.075 [0.004, 0.148], Yt (14) = 2.260, p = 0.004, with a medium effect size (ξ) of 0.347 [0.000, 0.688]. In contrast, MT condition showed no such difference, Mdiff = 0.078 [−0.190, 0.033], Yt (14) = 1.504, p = 0.154, ξ = 0.284 [0.000, 0.732]. Effect sizes ranged from 0 to 1 (see raw trimmed-mean differences for easier interpretation in Figure 5).

For accuracy, defined as number of errors per 100 words with lower values indicating better performance, MT condition showed significant medium effect size change post-feedback, Mdiff = −3.99 [−7.043, −0.941], Yt (14) = −2.807, p = 0.014, ξ = 0.453 [0.090, 0.807]. In contrast, TCF condition showed no significant change, Mdiff = −1.58 [−4.534, 1.378], Yt (11) = −1.175, p = 0.265, ξ = 0.231 [0.000, 0.672].

In terms of fluency, MT condition exhibited a significant change, Mdiff = 18.133 [4.501, 31.765], Yt (14) = 2.853, p = 0.013, ξ = 0.303 [0.000, 0.633]. However, TCF condition showed no significant difference, Mdiff = 5.267 [−3.248, 13.781], Yt (14) = 1.327, p = 0.206, ξ = 0.082 [0.000, 0.460].

In summary, while neither TCF nor MT feedback proved more effective than the other, post-hoc analysis revealed significant gains in complexity for TCF and in accuracy and fluency for MT when comparing pre- and post-feedback texts.

4.2 High-achieving participants’ responses to the CAF retrospective improvement questionnaire

As explained above, we administered a CAF retrospective improvement questionnaire to find out why one type of feedback (TCF or MT) facilitated improvement in certain CAF aspects of L2 texts. Even though we collected responses to this questionnaire from all participants, due to space limitations, we focus here on the responses of the top six participants (Top 2 for each aspect of CAF; Table 5). When a participant ranked high on more than two measures, the higher-ranking participant of the two candidates was chosen as the respondent for that variable.

Table 5 shows that these high-achieving students all actively engaged with the feedback. Furthermore, more than half of all responses (4 out of 6) show that their behavioral engagement sometimes went beyond the text they were reviewing (i.e., the underlined segments).

4.3 Research question 2: engagement with TCF versus MT

Table 6a presents percentages of participants who chose different types of cognitive engagement in answer to the Post-TCF/MT Engagement Questionnaire. First, more participants (91.3 %; Item No. 1 in green) noticed the TCF, while 45.5 and 72.7 % compared the MT with, respectively, the Japanese and English texts they wrote the previous week (Items No. 1 and 5 in green). Second, for both TCF and MT, approximately half to two-thirds of participants (TCF: Items No. 1–3, M_% = 69.6 %; MT: Items No. 1–7, M_% = 52.6 %) engaged with immediate problem-solving (IPS) operations, such as “I wonder why the item was corrected” for TCF and “I wondered how my Japanese text was translated” for MT. In contrast, a greater number of participants (Items No. 4–6 in blue; M_% = 27.5 %) engaged with future-oriented (FO) cognitive operations; e.g., “I wonder how I would write the part that did not exist in the original text”) when given TCF compared to MT (Items No. 8 and 9 in blue: M_% = 15.9 %).

Table 6b presents percentages of participants who chose items indicating affective engagement in response to the Post-TCF/MT Questionnaire administered after they were given either TCF or MT. When independently asked, more than half the participants reported feeling positive about both TCF and MT. However, many more felt much more positive and motivated about TCF because they felt they could learn more from it than from MT.

4.4 Research question 3: comparison between MT and TCF

Table 7 presents the results of the Feedback Comparison Questionnaire, which addressed Research Question 3. Although Items 2 and 3 in Table 6b and Items 6 and 7 in Table 7, respectively, are similar, their percentages are quite different, probably because Items 2 and 3 in Table 6b were asked independently for each type of feedback (TCF and MT) whereas Items 6 and 7 in Table 7 comparatively. This is especially evident in Item 2, Table 6b, where both MT (50 %) and TCF (69.6 %) received relatively positive evaluations from participants, which contrasts sharply with their comparative choices in Items 6 and 7 in Table 7, where, when asked which of the two feedback types they preferred, about 80 % or more chose TCF as more useful and motivating. In fact, their responses to the open-ended questionnaire almost unanimously (17 out of 18) showed that they found TCF to be more useful because it allowed them to see their specific grammatical problems.