Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition

2.1 Assessing the Capabilities of ChatGPT

Many research explore the capabilities of ChatGPT across many fields. Ogundare and Araya (2023) utilizes spontaneous quality (SQ) scores to compare the performance of ChatGPT on many natural language processing (NLP) tasks such as machine translation, machine summarization, question answering, and language generation, compared with other mainstream algorithms. George and George (2023) explores how ChatGPT can enhance e-commerce through chat and other sectors such as education, entertainment, finance, health, news, and productivity, and how ChatGPT can be used to create more personalized content for users and help businesses improve the efficiency and effectiveness of customer service. Rezayi et al. (2023) explores a new field of agricultural natural language processing by studying the effectiveness of pre-training transformer-based language models using food-related text corpora. Wang et al. (2023b) investigates ChatGPT’s ability to infer dynamic network structures from temporal text data, especially financial news. Khondaker et al. (2023) conducts a large-scale evaluation of ChatGPT on a wide range of Arabic NLP tasks. Wu et al. (2023b) evaluates ChatGPT’s ability in knowledge in the medical field on the Chinese National Medical Licensing Examination (CNMLE).

Wang et al. (2023c), taking document-level machine translation as an experimental platform, conducts an in-depth evaluation of ChatGPT’s discourse modeling ability in three aspects: the role of discourse awareness cue, the comparison of translation models, and the analysis of discourse modeling ability. Zhang, Ouni, and Eger (2024) explores ChatGPT’s ability to summarize and evaluate on cross-lingual cross-time summarization (CLCTS) tasks. Ronanki, Cabrero-Daniel, and Berger (2023) explores the application of ChatGPT for user story quality assessment and compares its performance to existing benchmarks. Sun et al. (2023) proposes a set of generic modules, attempting to break through the limitations of ChatGPT on various NLP tasks, including question answering, common sense reasoning, natural language reasoning, etc. Antoun et al. (2023) proposes a method for developing and evaluating ChatGPT detectors for French text, focusing on their robustness against extra-territorially based data and common attack schemes. Wang and Demszky (2023) explores whether ChatGPT can be a cost-effective supplement to expert feedback by acting as an automated teacher or coach.

2.2 Application of Large Language Models in Ancient Chinese

However, there is relatively little research on ChatGPT’s understanding of ancient Chinese. The research on the understanding of ancient Chinese by language model is mainly based on BERT model. Wang et al. (2022) and Chang et al. (2023) build a pre-trained language model of SikuBERT and SikuGPT for ancient text intelligent processing tasks based on BERT deep language model framework using the verified high-quality full-text corpus of Si Ku Quan Shu as a training set. Some researchers built GuwenBERT based on RoBERTa model trained on a large number of ancient literature corpus. Tsinghua University has constructed BERT CCPoem,^[1] a pre-training model based on BERT specifically for ancient Chinese poetry, which is trained on the complete collection of ancient Chinese poetry CCPC (Li et al. 2021) and can be used for intelligent poetry retrieval, recommendation and sentiment analysis. Wang et al. (2023d) introduces two language models, GujiBERT and GujiGPT, which are foundational models specifically designed for intelligent information processing of ancient texts. Chang et al. (2022) proposes SIKU-RoBERTa, pre-trained on the high-quality full-text corpus of SiKuQuanShu, and studies the part-of-speech tagging on ancient Chinese texts.

3.1 Data Preparation

The main data comes from Github open source project NiuTrans/classic-Modern,^[2] which is a very comprehensive ancient Chinese-modern parallel corpus, including a large number of classical ancient works. This dataset divides and displays each ancient book by chapters, and the text is stored in the txt documents under each chapter. These bilingual data are provided in sentence level, and three data formats are provided: original text, translation and bilingual.

In this study, the team mainly utilized the bilingual data of the book, Shi Shuo Xin Yu, including 36 chapters, to satisfy our needs in the experiments. And most articles in this book are about biographies of famous people in ancient China, which can be used for ancient-to-modern Chinese translation and person name recognition.

Due to the input length limitation of ChatGPT, we split the whole book into 3,923 sentences and each time we feed the model a sentence. For the ancient-to-modern Chinese translation task, we include all sentences, but for the ancient Chinese name recognition task, we randomly select a total of 300 sentences because of time constraint. Also we manually label Chinese names for the 300 sentences. The data is shown in Table 1 as follows.

3.2 Access to ChatGPT

ChatGPT, a pre-trained large language model, simply takes a text input following a prompt, then it provides a response to the question. ChatGPT supports HTTP requests in multiple languages. In this experiment, we access ChatGPT via the module openai in Python. There are many pre-trained models available in this module, like gpt-4o, gpt-4, gpt-3.5-turbo etc. In this experiments, we choose gpt-3.5-turbo and gpt-4o, because they are the most popular variants.

3.3 Translation of Ancient Chinese by ChatGPT

Once having access to ChatGPT in Python, we can simply design a sensible prompt to perform the ancient-to-modern Chinese translation task. We tried a variety of prompt input methods, such as “Please translate the following ancient Chinese:”, “Translate the ancient Chinese after the colon into modern Chinese:”, “Translate the ancient Chinese into modern Chinese” and so on. We found that it is necessary to include “translation” and “modern Chinese” in order to perform the ancient-to-modern Chinese translation robustly, otherwise the output of ChatGPT may have redundant and wrong information. In addition, we also found that even if we input only ancient Chinese sentences without any instruction, ChatGPT can sometimes return the correct translation results in modern Chinese.

After many trials, we choose a straightforward “将文言文翻译成现代汉语(translate ancient Chinese into modern Chinese)” as the prompt instruction for the translation task. This prompt almost never produces error information, and no superfluous information. We extract the text content that contains only the modern Chinese translation of the ancient Chinese. Due to the input length constraints set by OpenAI, it is unfeasible to command ChatGPT to translate the entire Shi Shuo Xin Yu book directly. Consequently, we use a systematic FOR loop to sequentially translate preloaded segments of the book.

To proportionately evaluate its performance, we adjust the size of the input – the amount of text that needs to be translated. We start off with a single sentence to observe how well the program can perform a basic translation task. Then, they slowly increase the input length to three, five, and eight sentences. This gradual scaling may challenge the program and test its limits, especially in maintaining cohesiveness and continuity in the translation of longer passages.

For the ancient-to-modern translation task, we evaluate the performance of ChatGPT with BLEU (Papineni et al. 2002) and BERT-Score (Zhang et al. 2019). BLEU, is the de facto standard for machine translation performance evaluation because it is easy to calculate regardless of the languages involved. BLEU compares n-grams of the candidate translation with n-grams of the reference translation and counts the number of matches; the more the matches, the better the candidate translation. Unlike BLEU that majorly makes use of token or phrasal level syntactic overlaps between hypothesis and reference text pieces, BERTScore on the other hand captures the semantic aspect by using the contextualized embeddings generated by the BERT model.

3.4 Person Name Recognition from Ancient Chinese Literature

Besides translation, we assess the competence of ChatGPT in recognizing names referenced within the ancient Chinese text. It is noteworthy that the effectiveness of ChatGPT in executing this task is contingent upon the prompt instruction. We utilize a trial-and-error approach to ascertain a suitable prompt instruction, “已知文言文的原文如下, 找出该文言文中的人名”, which means that “Given the following ancient Chinese text, find the people names in it”.

The Jieba module in Python is widely employed for word segmentation and rudimentary named entity recognition in Chinese texts. Hence, for our study on the recognition of people names in ancient Chinese text, we incorporate Jieba as a baseline method for comparison. SIKU-RoBERTa is a pre-training language model based on the high-quality full-text corpus of Si Ku Quan Shu, and a portion of Zuo Zhuan that has been word segmented and tagged is used as training sets to build a deep network model based on BERT for part-of-speech (POS) tagging experiments. We also include SIKU-RoBERTa as a baseline method for ancient Chinese names recognition. For the this task, we evaluate the performance of ChatGPT with commonly used metrics, like precision, recall and F1 score (Jiang, Banchs, and Li 2016).

4.1 Assessing the Efficacy of ChatGPT in the Translation of Ancient Chinese Texts

As mentioned in the experimental setup Section 3, we deploy ChatGPT to convert sentences from the ancient Chinese text, Shi Shuo Xin Yu, into modern Chinese in configurations of 1, 3, 5, and 8 sentences per query. These translations are then evaluated and contrasted using quantitative metrics such as BLEU, and BERT-Score. Table 2 illustrates the BLEU scores resulting from the translation of ancient to modern Chinese from ChatGPT and Baidu’s ERNIE-Bot under four different configurations: one sentence (1-sent.), three sentences (3-sent.), five sentences (5-sent.), and eight sentences (8-sent.) per individual query. In this table, we conduct an evaluation of the BLEU scores across three different levels of granularity: 1-g, 2-g, and 3-g. From Table 2, it is observed that ChatGPT is on par with ERNIE-Bot on ancient-to-modern Chinese translation in terms of 2-g BLEU score, where both GPT-3.5 and ERNIE-Bot achieve 0.20 when fed with three sequential ancient Chinese sentences (3-sent.). This suggests that contextual information enhances the translation quality from ancient to modern Chinese. Interestingly, the quality of translation does not exhibit improvement when the context increases to eight sentences, and a possible reason is the increased complexity of long piece of ancient Chinese text. Another important finding is that the BLEU scores are still very low, all models reaching 0 at 3-g BLEU in all four settings, which means that both ChatGPT variants and ERNIE-Bot still perform very poorly in ancient-to-modern Chinese translation.

Table 3 shares the same structure as Table 2, but displays the BERT-Scores, precision, recall, and F1 scores associated with the translation of ancient to modern Chinese by two ChatGPT variants and ERNIE-Bot of Baidu. From this table, both ChatGPT variants perform the best when translating three consecutive ancient Chinese sentences (3-sent.), with GPT-3.5 achieving 78.5 F1 score and GPT-4o achieving 81.65 F1 score. But Baidu’s ERNIE-Bot surpasses GPT-4o significantly, reaching 87.69 F1 score when translating five consecutive ancient Chinese sentences (5-sent.). This is understandable as ERNIE-Bot is released by the Chinese company Baidu, so it might be pre-trained by larger ancient Chinese corpus than GPT models.

To visualize the effects of input sentence number on the translation quality of GPT-3.5, Figure 1 presents the line plots of BERT scores versus input sentence number in the prompt. From this figure, it can be observed that BERT scores (recall, precision and F1 score) peak at 3. This can be explained by the fact that ancient Chinese is complex and context is very important for understanding the pronouns and references. But long context may further confuse the model, so the performance is best at a moderate length of 3 sentences.

Figure 1:

The BERT-scores of ancient-to-modern Chinese translation when ChatGPT (GPT-3.5) is fed with 1, 3, 5, and 8 consecutive sentences.

Error Analysis: To gain deep insights, we select and analyze a specific sentence as a case study. The original ancient Chinese sentence is “孙秀既恨石崇不与绿珠, 又憾潘岳昔遇之不以礼。”, which means that Sun Xiu both resented Shi Chong for not giving him Zhu Lv (name of a woman), and was dissatisfied with Pan Yue for his past discourteous behavior towards him. The groundtruth modern Chinese and translated texts from four settings are exhibited in Table 4. To aid readers understand the translated Chinese, we also provide an English version for all methods. As shown in Table 4, the translated modern Chinese texts from GPT-3.5 is closest to the groundtruth when feeding three sentences (3-sent.). The wrong points are highlighted by the underlines.

4.2 Assessing the Efficacy of Person Name Recognition in Ancient Chinese using ChatGPT

Person name recognition task can assess the capability of models on understanding ancient Chinese. For the person name recognition task, a total of 300 sentences are randomly selected, and a few examples are shown in Table 1.

Table 5 presents the performance of ChatGPT variants (GPT-3.5 and GPT-4o), SIKU-RoBERTa, and Jieba in recognizing people names from ancient Chinese texts in few-shot settings. From this table, on the ancient person name recognition task, ChatGPT variants significantly outperforms Jieba in zero-shot setting, with GPT-4o achieving 69.59 F1 score. But their performance is worse than SIKU-RoBERTa, which was trained on Zuo Zhuan for POS tagging. However, the precision of GPT-4o is at 94.93, mich higher than other models. The main problem of LLMs lie in the recall of ancient names.

For the few-shot setting, the performance of GPT-3.5 gradually improves as the number of demonstrations increases in the prompt. Figure 2 exhibits this increasing trend as well. As the number of instances increases from 0 to 10, the F1 score rises from 0.628 to 0.642.

Figure 2:

The performance of GPT-3.5 on person name recognition versus the number of examples in the prompt.

Error Analysis: To gain better understanding, we perform an error analysis of GPT-3.5 on person name recognition as shown in Table 6. From these cases, GPT-3.5 suffers from low recall rate in the first two examples. For the third example, GPT-3.5 recognized the wrong person name, and missed another name. For the last example, GPT-3.5 identified two more erroneous names.

4.3 Summary

From our experiments on ancient-to-modern Chinese translation, we can observe that ChatGPT performs well on ancient-to-modern Chinese translation in terms of semantic similarity, but suffering poor performance in terms of syntactic overlapping. On the person name recognition task, ChatGPT surpasses Jieba by a clear margin, showing good promise in extracting people name from ancient Chinese text.

During our experiments, we find that ChatGPT faces many challenges in both ancient-to-modern translation and person name recognition from ancient Chinese texts. Fully understanding ancient Chinese sentences rely heavily on the context of the sentence. Sometimes a sentence may omit the name of the person, or replace it with other names. Therefore, feeding multiple consecutive sentences in the prompt of ChatGPT enhances the performance as well. We leave it for future research.