The assistance role of LLMs and NMT in student translators’ Chinese–English post-editing: differences in workload, translation quality and user perception

2.1 LLMs in translation practice

LLMs, a type of language models that utilizes neural networks containing billions of parameters, are generative mathematical models of the statistical distribution of tokens in the vast public corpus of human-generated text. Their abilities have been proved in various language-related tasks, including text synthesis, translation, summarization, question-answering, and sentiment analysis, by leveraging deep learning techniques and large datasets (Raiaan et al. 2024; Shanahan 2024).

In the context of translation, LLMs offer assistance through two primary affordances: access to expansive knowledge bases and real-time interactive feedback. Compared with traditional NMT systems, LLMs are pretrained on diverse and extensive textual corpora, which allows them to capture various forms of knowledge – including event, relational, factual, and commonsense knowledge (Da et al. 2021; Han and Chai 2024; Heinzerling and Inui 2021; Kauf et al. 2023; Safavi and Koutra 2021). This breadth of information enables LLMs to support translators in generating target texts that are more contextually relevant and factually accurate, thereby improving both translation quality and efficiency.

A further advantage of LLMs lies in their capacity for real-time, dialogic interaction. Through multi-turn conversation, these models can tailor feedback to users’ individual needs, cognitive styles, and translation goals. Techniques such as chain-of-thought prompting simulate human cognitive processing, thereby enhancing the depth and personalization of the interaction (Dai et al. 2023; Lu and Chen 2024). Within translation contexts, feedback serves not merely a corrective function but also a pedagogical one – enabling translators to read their translation work from the perspectives of readers or users, and fostering the capacity for self-directedness and self-assessment (Washbourne 2014).

ChatGPT, as a prominent example of LLMs, can provide real-time feedback and offer targeted suggestions for revisions (Dai et al. 2023). It is characterized by short response time, high efficiency, numerous outputs, diverse options, and personalized, context-aware supports. Previous empirical studies demonstrate that both postgraduate and graduate students of translation major significantly improve their translation quality through revision assisted by ChatGPT’s interactive feedback (Zhu and Shao 2024, 2025).

Although ChatGPT and other models have demonstrated strong translation competence and are increasingly adopted in both translation practice and academic research, relatively few studies have focused on LLMs developed in China. Their potential to support student translators during PE – especially in comparison with established NMT systems – remains under-investigated.

2.2 AI-assisted PE

PE refers to the systematic evaluation, refinement, and modification of MT outputs to meet specific quality standards. This process involves correcting errors, improving accuracy, and enhancing overall readability (Bowker and Ciro 2019; Feng and Cui 2016; ISO 2014). PE has become an integral part of modern translation workflows, especially in the context of increasingly advanced AI systems.

Operationally, PE is typically categorized into two levels: light post-editing (LPE) and full post-editing (FPE). LPE emphasizes semantic correctness and basic comprehensibility, tolerating some grammatical or syntactic inaccuracies as long as the core information remains intact. The output may retain traces of machine-generated style, with imperfect grammar or unnatural phrasing, but is expected to be accurate in content. In contrast, FPE aims for grammatically, syntactically, and semantically correct translations that are also stylistically appropriate. While the final product may not match the stylistic nuance of a native-speaker human translator, it is expected to meet professional translation standards in terms of fluency, coherence, and linguistic correctness (Nitzke and Gros 2020).

As Pym (2012) observed, “Statistical-based MT, along with its many hybrids, is destined to turn most translators into post-editors one day, perhaps soon.” This prediction has materialized in part with the rise of LLMs, which have shifted translation from a linear, one-directional process to a dynamic, interactive, and increasingly intelligent collaboration between humans and AI systems. Translators now routinely engage with AI-driven tools to improve productivity and output quality (Wang et al. 2023).

Trained on multilingual corpora, LLMs, such as ChatGPT, are capable of managing complex linguistic patterns and generating fluent, contextually appropriate translations in PE tasks. Their strength lies in handling a wide range of general-domain content with coherence and high speed. However, limitations persist, particularly in relation to domain-specific texts or distant language pairs, where LLMs’ outputs often fail to match human-level precision or stylistic appropriateness (Bhattacharyya et al. 2023; Li and Li 2025). In these cases, human intervention through PE remains indispensable. The integration of human expertise and AI assistance continues to be a promising and complementary approach in PE workflows.

Recent studies have examined PE from multiple perspectives, including strategies for improving translation quality and reducing cognitive load (Fan and Yang 2024; Geng 2024; Shin and Chon 2023), comparisons between human translation and MT output (Jia and Sun 2022; Wang et al. 2024), the relationship between PE and translator effort (Lu and Sun 2018; Wang and Wang 2024; Zhong et al. 2024), and the application of PE in translation education (Feng and Liu 2018; Wang and Wang 2023; Zhong and Shu 2020). While the emergence of LLMs has opened a new avenue for human–AI collaborative PE (Geng and Hu 2023), empirical studies in this area remain limited (Khasawneh and Khasawneh 2023). Understanding the assistance roles of different tools in PE – particularly their effectiveness in assisting students during practices – is of significant importance for both the improvement of translation competence and the advancement of translation technologies.

To address this gap, the present study adopts a mixed-methods approach, incorporating keystroke logging, error annotation, screen recording, and questionnaire surveys, and recruits MTI students from a comprehensive university in Central China to participate in the study. They conducted full post-editing tasks on C–E MT output, using both NMT and LLM tools – specifically, DeepL, ChatGPT-4o, and DeepSeek-V3. The investigation focuses on three dimensions: workload, translation quality, and user perception. By evaluating student performance and perceptions across these dimensions, the study aims to shed light on the assistance role of different tools in supporting student translators’ PE. Ultimately, the findings are expected to contribute to the optimization of PE strategies, the development of student translators’ PE competence, and the innovation of AI-integrated translation pedagogy.

3.1 Participants

This study recruited 30 MTI students as participants. Among them, 5 were male and 25 were female, with an average age of 23.6 years (SD = 0.93). A total of 100 % had passed the Test for English Majors Band 8 (TEM-8), indicating that their language and translation proficiency met the general standard for professional translators. All participants were native Chinese speakers with English as their second language. The pre-experiment questionnaire confirmed that all participants were skilled at typing and had not previously read the experimental texts. Informed consent was obtained from all participants before the experiment, and they were compensated for their participation.

3.2 Materials

3.3 Experimental procedures

The 30 participants were randomly assigned to three groups.

• Group 1 (G1) used the traditional NMT tool DeepL, with the option to consult online resources such as Baidu.com for reference when needed.

• Group 2 (G2) was assisted by ChatGPT-4o developed by OpenAI.

• Group 3 (G3) worked with DeepSeek-V3 developed by Hangzhou DeepSeek Artificial Intelligence Co., Ltd.

Prior to the formal experiment, each group received targeted technical training sessions, aiming to reduce disparities in participants’ familiarity with technological tools and PE-related knowledge, thus minimizing their potential impact on experimental outcomes.

As Youdao Translate has relatively stable performance across various kinds of texts (Dai and Liu 2024), it was employed to generate the raw machine translations of the source texts and some examples were shown in Appendix A. In Translog-II, the source texts and their MTs were displayed as Figure 1. Participants needed to complete the PE tasks of all four texts on Translog-II User as quickly as possible.

Figure 1:

An example of the student translator’s post-editing interface.

After completing the tasks, participants’ PE files were collected from the three groups, yielding 120 translation products along with their editing process records, and then, participants were asked to complete a brief questionnaire designed to collect their subjective evaluations of the error correcting capabilities of each tool, as well as of the tools’ advantages and disadvantages in supporting their PE tasks.

4.1 Workload

Krings (2001) classified the workload of PE into temporal, technical, and cognitive efforts. Temporal effort refers to the time needed for PE, technical effort relates to deletions, insertions, and other mechanical operations, while cognitive effort involves the mental processing activated during PE. In this study, to measure these three types of efforts, we adopted task duration, keyboard events, and pause length, which were recorded and processed using Translog-II and CRITT TPR-DB (Carl et al. 2016).

First, the normality of the three types of effort data was examined using the Shapiro–Wilk test in SPSS 27.0 for Windows. A p-value higher than 0.05 indicated that the data followed a normal distribution, while a p-value lower than or equal to 0.05 suggested a non-normal distribution. Subsequently, a one-way ANOVA was conducted for normally distributed data, and the Kruskal–Wallis test was employed for non-normally distributed data. When a significant difference was found, post hoc comparisons were conducted using the Bonferroni test for parametric data and the Mann–Whitney U test for non-parametric data, in order to determine whether the use of NMT or LLMs had a significant impact on student translators’ performance.

Task duration refers to the time needed to complete PE tasks, and a longer task duration indicates more temporal effort. Data analysis revealed that there was no significant difference (p = 0.228) in total task duration among the three groups when text type was not taken into account. Similarly, when text type was considered, there were also no differences in task duration across different groups for popular science (p = 0.147), prose (p = 0.543), novel (p = 0.256), or academic work text (p = 0.738) tasks.

Keyboard events refer to translators’ keyboard operations, and a higher number of events indicates greater technical efforts. Based on comparative statistics, there was no significant difference in the total number of keyboard events among the three groups (p = 0.079).

However, when analyzed by text type, a significant difference (p = 0.008) was observed in the number of events for the Text 1 task. Based on Table 2, the post hoc comparison result indicated that, students of G2 and G3 required fewer keyboard events than that of G1 in this task. Nevertheless, for Text 2 (p = 0.225), Text 3 (p = 0.098), and Text 4 (p = 0.338), no significant differences in the number of keyboard events were observed among the groups.

In terms of cognitive effort, a threshold of 1,000 ms was adopted to filter out shorter pause lengths, and only pause lengths equal to or longer than this threshold were included in the analysis. The longer the pause length, the greater the cognitive effort required during PE tasks (Kumpulainen 2015).

Our results showed that, when text type was not taken into consideration, there was no significant difference (p = 0.965) in total pause lengths among the three groups. Similarly, when examining student translators’ cognitive effort across different text types, no significant differences in pause lengths were found among the groups for Text 1 (p = 0.995), Text 2 (p = 0.804), Text 3 (p = 0.924), or Text 4 (p = 0.886).

In summary, from the perspective of PE workload, the findings suggest that when student translators engage in PE tasks involving texts of similar complexity, the use of either LLMs or NMT does not lead to significant differences in total temporal, technical, or cognitive efforts. The only notable exception was found in the PE task involving the popular science text, where students assisted by ChatGPT-4o and DeepSeek-V3 completed the task with fewer keyboard events compared to those using DeepL. This may be attributed to the fact that popular science texts – unlike prose, novels, or academic work texts – are generally easier for LLMs to process.

4.2 Translation quality

Although translation quality assessment involves both automatic and human evaluation (Yang 2012), our prior experimental experience and the results of automatic translation evaluation in this study suggest that automatic metrics such as BLEU, TER, and METEOR – while capable of reflecting quality differences – are insufficient for capturing the PE process or identifying specific error types (Daems et al. 2017a, 2017b). Therefore, this study only reports the results of human evaluation results.

4.3 User perception

User perception, including users’ positive and negative attitude or feedback towards technologies, is invaluable for providing insights into tools and workflows and revealing issues that would not be evident from the translations or process data (Bundgaard 2017). After finishing the PE tasks, the participants were asked to complete a post-task questionnaire to evaluate the tools they used.

The brief questionnaire focused on participants’ user experiences by evaluating each tool’s error-correcting capabilities across the five error categories (completeness, accuracy, language, culture, and style), as well as their reflections on the advantages and disadvantages of LLMs and NMT in supporting PE.

Students first assessed the error-correcting capabilities of each tool across various error types, which demonstrated their clear preferences. According to the results in Figure 2, DeepL, ChatGPT-4o, and DeepSeek-V3 received 39, 56, and 42 votes respectively. The highest number of votes was obtained by ChatGPT-4o, suggesting that student translators generally perceived it as the most effective tool in terms of error-correcting capability during C-E PE practices.

Figure 2:

Number of votes for each tool’s error-correcting ability.

From the perspective of the five error categories, ChatGPT-4o received the highest number of votes in the dimensions of completeness, language, and style, with particularly higher votes in the latter two. This suggests that students perceived ChatGPT-4o as more capable of assisting them in dealing with language and style types of errors during PE. In the dimension of accuracy, both ChatGPT-4o and DeepSeek-V3 received higher votes than DeepL, indicating that LLMs’ assistance was helpful for resolving accuracy errors. Last, in the culture dimension, DeepSeek-V3 received slightly more votes than the other two tools, implying that students viewed it as more effective in correcting culture errors.

In addition to voting for the three tools’ error-correcting ability in the five error types, students shared qualitative evaluations, mainly focusing on the advantages and disadvantages of each tool. Based on their evaluations, we calculated the theme distribution and frequency, and sorted them into five dimensions including assistance efficiency, information accuracy, user experience, translation quality, and context comprehension (Figure 3) to visually present student translators’ evaluation of advantages and disadvantages towards DeepL, ChatGPT-4o and DeepSeek-V3 in assisting PE practices.

Figure 3:

Frequency distribution of students’ evaluations of the tools’ advantages and disadvantages.

From the chart, we found that, in terms of assistance efficiency, ChatGPT-4o demonstrated the most notable advantage, followed by DeepSeek-V3, while DeepL was comparatively less efficient. Some students noted that when using DeepL to assist PE, the process was cumbersome, and the task flow would be easily disrupted when they had to search information online that DeepL couldn’t support. In the information accuracy aspect, LLMs did not outperform DeepL, primarily because they offered a wide range of information – some of which may be inaccurate – leaving the translator to make judgments. As for user experience, student translators generally found LLM-assisted PE to be highly convenient, and the information provided often inspired new ideas and perspectives. In terms of translation quality, DeepL was outperformed by the LLMs, with ChatGPT-4o showing a particularly clear advantage. In terms of context comprehension, DeepL lagged behind the LLMs as it was unable to accurately understand the context of source texts.

Overall, the data from total task duration, keyboard events, and pause length told us that there was no significant difference in three groups of student translators’ PE workload – encompassing total temporal, technical, and cognitive efforts. The only exception was observed in the popular science text tasks, where students in G1, assisted by NMT, exhibited a noticeably higher number of keyboard events compared to the other two groups.

In contrast, the data on error counts and error scores showed a clear advantage for LLM-assisted PE. Students in the G2 and G3 produced translations with fewer errors and lower error scores in total than those in the G1. This suggested that LLMs can effectively support students in generating higher-quality translations during PE tasks.

From the perspective of user perception, students rated ChatGPT-4o highest in terms of error correcting capability and assistance efficiency. DeepSeek-V3 was recognized for offering the best user experience, while DeepL was regarded as the most reliable in terms of information accuracy. These findings highlighted the importance of considering both objective performance and subjective user experience when integrating AI tools into PE practices.

5.1 Guidelines for C-E PE

This experiment selected four different types of Chinese source texts, with their MTs containing various types of errors, including completeness, accuracy, language, culture, and style. These errors posed considerable challenges for student translators in the PE process. Although they produced fewer errors with lower scores in PE tasks assisted by LLMs compared to those aided by NMT, some errors still persisted. By comprehensively analyzing the source texts, MTs, and screen recordings of the students’ translation processes, we identified three key aspects in which the assistance role of LLMs should be further leveraged to enhance the PE quality.

First, ensuring semantic accuracy. Machine translation systems often failed to accurately translate expressions containing culture-specific concepts. For example, in Text 3, the cultural term “叼空” is a regional colloquialism meaning “to squeeze time to do something,” yet the Youdao translated it simply as “manage,” which meant to succeed in doing something without the implication of time management, thereby omitting the cultural nuance of the source text. Similarly, “风水书” does not refer to an ordinary book, but rather to a type of Chinese almanac developed through centuries of Chinese people’s practices to guide the selection of auspicious dates. The reference translation used “fengshui almanac,” which accurately conveyed the concept, but if it was translated as “fengshui book” as Youdao may mislead readers into thinking that it refers to a general book about fengshui theory, being inconsistent with the context.

Second, maintaining logical consistency. Machine translation systems also frequently failed to deal with referential relationships. In Text 2, for instance, “梁任公,” “梁启超,” and “任公” all referred to the same historical figure. However, Youdao inconsistently translated these as “Mr. Liang Rengong,” “Liang Qichao,” or “Mr. Rengong,” thereby disrupting referential coherence. Similarly, in Text 1, “(鲁班锁)完全依靠自身结构的连接支撑” emphasized that the connection and supporting functions depend on the mechanism of Luban lock. However, the machine translation system translated it into “It is supported by the connection of its own structure”, which distorted the original logic.

Third, following the language rules. In Text 4, the sentence “所谓纪律习惯, 盖指多人聚集场面, 无待一条一条宣布” was translated as “The so-called discipline and habit refer to the fact that in a scene where many people gather, there is no need to announce one by one.” This translation contained two clauses without a clear syntactic connection, violating English cohesion norms. Furthermore, the subject in the second clause was also vague.

5.2 Methods for PE with LLM assistance

Experimental data suggested that LLM-assisted PE achieved higher translation quality than with NMT. Experimental observations and questionnaire results indicated some methods for effective LLMs integration in PE practices.

First, it is essential to fully leverage the respective strengths of different LLMs. Although the experimental results demonstrated that PE assisted by ChatGPT-4o and DeepSeek-V3 led to higher translation quality, certain differences between the two tools remained. In C–E PE tasks, student translators may choose tools based on text types, quality expectations, and personal preferences. For example: (1) When text type is not taken into consideration, both ChatGPT-4o and DeepSeek-V3 can effectively assist in reducing error counts and scores in PE tasks. However, these two tools also exhibit certain differences. For instance, compared to using DeepL, ChatGPT-4o is more effective in helping students reduce language type errors, whereas DeepSeek-V3 performs better in assisting the reduction of error scores in completeness aspect; (2) For texts requiring higher translation quality – whether in the popular science, prose, or academic work texts – DeepSeek-V3 helped student translators reduce accuracy errors (including mistranslation, repetition, ambiguity, and omission), while ChatGPT-4o contributed more to reducing style errors (including stylistic inconsistency and overly literal translation) in prose and academic work texts. (3) According to students’ evaluations of tool performance, ChatGPT-4o was considered the most efficient in terms of assistance efficiency, while DeepSeek-V3 provided the best user experience. Therefore, student translators may also take into account user perception and preference when selecting LLMs for supporting PE.

Second, it is important for translators to proofread the text themselves. In Text 2, the “前辈的学者” in the source text referred to elder scholars as opposed to the scholars of younger generation. However, the machine rendered it as “senior scholars”, which typically referred to scholars with higher academic rank or seniority, failing to match the intended contrast with “后生” (the younger generation), thus introducing semantic inaccuracy. Notably, among the students using ChatGPT-4o or DeepSeek-V3, only one student identified and corrected this error. Likewise, the LLMs themselves seldom found or addressed this issue during their interaction with students. In contrast, 80 % of the students who used DeepL successfully corrected the error. This suggested that effective use of LLMs not only involved crafting appropriate prompts but also needed translators to proofread their post-edited work in parallel with the source text. Blind reliance on LLMs’ revision suggestions may lead to missed errors in translations.

5.3 Implications for PE education

Translation students generally acknowledged the effectiveness of LLMs in PE, indicating the necessity of incorporating AI technologies into translation teaching to enhance student PE competence. Previous research defined PE competence as the knowledge and cognitive literacy required to revise machine-generated outputs based on task-specific objectives, encompassing MT knowledge, terminology management, discourse knowledge, documentation skills, intercultural awareness, error identification, and editing efficacy (Feng and Liu 2018; Koponen 2015; O’Brien 2002). Our findings further emphasized the importance of error correcting and critical thinking skills for translator training in the LLM era.

From the screen recordings of students’ translation processes, we observed that among the multiple translation suggestions provided by LLMs, at least one or more often contained relatively inaccurate expressions. For example, in Text 3, the phrase “鬼使神差” was rendered in the machine translation as “by some forces”, which was overly simplistic and resulted in cultural loss. The translation failed to capture the literary and stylistic information of the original expression. During student–LLM interaction, DeepSeek-V3 proposed more culturally sensitive or stylistically appropriate expression suggestions, such as “As if guided by ghosts and gods”, which retained cultural imagery, or the idiomatic English phrase “It is a twist of fate”. However, some students ultimately adopted a less appropriate revision like “driven by some inexplicable reasons”, which may obscure the intended meaning and diminish the stylistic effect.

At the same time, LLMs may also provide misleading suggestions. In Text 3, ChatGPT-4o advised the student to revise the translation of “无所为而为” as “a kind of effortless action – what the Daoists would call ‘doing without striving’”. This reinterpretation deviated significantly from the original meaning, which in context referred to Liang Qichao’s pursuit of learning driven by internal motivation rather than Daoist philosophy. Nevertheless, the student adopted the suggestion without critical thinking. Similarly, in Text 1, DeepSeek-V3 recommended translating “间不容发” as “fail-proof interlocking.” Yet, the term “fail-proof” typically denoted resistance to failure, which distorted the original text. These cases highlighted the risk of over-relying on LLM outputs without sufficient error correcting and critical thinking abilities.

Furthermore, some students expressed in the post-experiment questionnaire that, while neural NMT lacked the richness and immediacy of interaction that LLMs can offer, they appreciated the greater cognitive space provided for independent thinking. This observation underscores the value of pedagogical designs that balance AI assistance with opportunities for autonomous decision-making in PE tasks, and translation instruction should critically evaluate the role of AI, utilizing it as an auxiliary tool while maintaining human agency and critical awareness.

Moreover, prompt quality directly affects the quality of LLM-generated contents. Understanding prompt variation enhances ChatGPT interaction (Ekin 2023), and effective prompt design hinges on comprehension of LLM mechanisms (Polverini and Gregorcic 2024). Translation teaching should therefore incorporate additional knowledge of other areas, such as prompt engineering, enabling students to refine their “prompting intelligence” through PE practice.