Research on an English translation method based on an improved transformer model

1 Introduction

With the progress and development of society, the need for cross-language communication has increased [1]; therefore, translation between different languages has become particularly important. Traditionally, translation is done by humans [2], with high accuracy; however, with the development of globalization, the speed and cost of human translation cannot meet the current demand, so machine translation (MT) has emerged [3]. MT is a technology that enables the interconversion of different languages through computers, with high speed and low cost, and has been well used in many large-scale translation scenarios. To better serve the society, improving the quality of MT has become a very important issue nowadays. Lee et al. [4] introduced a neural machine translation (NMT) model that mapped the source character sequence to the target character sequence without any segmentation and used a character-level convolutional network with maximum pooling in the encoder part. By experimenting on a many-to-one translation task, the model was found to have high translation quality. Wu et al. [5] improved the NMT model using source and target dependency trees. The new encoder enriched each source state with a dependency relationship in the tree. During decoding, the tree structure was used as a context to facilitate word generation. The experiment found that the model was found to be effective in improving translation quality. Choi et al. [6] contextualized the word embedding vectors using a nonlinear bag-of-words representation of the source sentence and represented special tokens with typed symbols to facilitate translation of the words that are less suitable for translation through continuous vectors. The experimental results of En-Fr and En-De demonstrated the effectiveness of the model in improving the translation quality. Hewavitharana and Vogel [7] proposed a phrase alignment method that aims to align parallel sections bypassing nonparallel sections of a sentence and verified the effectiveness of the method in translation systems for Arabic English and Urdu English, which resulted in improvements up to 1.2 Bilingual Evaluation Understudy (BLEU) over the baseline. The current NMT has some problems, such as over-translation and under-translation, which lead to poor and ineffective translation, and further improvement and research are still needed. Therefore, this work studied the current mainstream Transformer model and innovatively improved it by combining it with a generative adversarial network (GAN). In the experiments, the reliability of the improved model in Chinese-English translation was verified by comparing the traditional Transformer model with the improved Transformer model. It was found that the translation results of the improved Transformer model were closer to the semantics of the source language and had smaller differences with the reference translations. The improved Transformer model is conducive to further improving the translation quality and effect and the better application of the Transformer model in Chinese-English translation.

2 Transformer model

With the development of artificial intelligence technology, MT has gradually developed from the earliest rule-based MT [8] to the early statistical MT [9], and the more common one now is NMT [10], which is mainly based on an “encoder-decoder” framework. NMT uses an encoder to map the source language sentence to a computable semantic vector and uses a decoder to decode the semantic vector to generate the target language sentence. Improving the translation effect of NMT is a key and difficult content in the current research. This work mainly studied the Transformer model.

Compared with the traditional NMT model, the Transformer model [11] completely abandons the recurrent neural network (RNN) structure and uses only the Attention mechanism to implement MT [12], which is good for reducing the computation and improving the translation effect. In the Transformer model, for an input (x ₁, x ₂,…, x _n ), it is mapped to z → = ( z 1 , z 2 , ⋯ z n ) through an encoder, and an output sequence (y ₁, y ₂,…, y _n ) is generated through a decoder. The overall structure of the model is shown in Figure 1.

Figure 1

The structure of the Transformer model.

“Attention” in the Transformer model refers to Scale Dot-Product Attention. Let the dimension of the input query be q and key be d _k and the dimension of value be d _v. Query, keys, and value are processed into Q, K, and V. The output matrix can be written as:

Q ∈ R m × d k , K ∈ R m × d k , and V ∈ R m × d v . The dimension of the output matrix is R m × d v .

Multi-head attention processing is used in the Transformer model. First, a linear mapping is performed on Q, K, and V · Q, K and V matrices whose input dimension is d _model is mapped to Q ∈ R m × d k , K ∈ R m × d k , and V ∈ R m × d v . Then, the result is calculated through Scale Dot-Product Attention. The above steps are repeated. Attentions obtained through h times of operations are put together to obtain multi-head attention. The detailed calculation formula is:

where W i Q , W i k , and W i V are parameter matrices.

Since the encoder and decoder of the model cannot capture the sequence order information, the position encoding is used in the Transformer model, which can be written as:

where pos refers to the position of a word in a sentence and i is the dimensional subscript of a word.

3 The improved Transformer model

In order to further improve the performance of the Transformer model, this paper improved it by combining GAN, which is based on the Nash equilibrium of game theory [13] and has extensive applications in image processing [14] and natural language processing [15]. GAN obtains the desired data through confrontation and gaming of a generator (G) and a discriminator (D) [16], using the back-propagation of D as the parameter update of G, thus enabling the model to learn what kind of utterance is considered a good translation.

The improved Transformer model uses the Transformer model as the generator (G) and the convolutional neural network (CNN) [17] as the discriminator (D). The goal of G is to generate a sequence from the initial state that maximizes the final desired reward, written as:

where θ refers to the parameter in the generator G, Y 1 : T refers to the generated target sentence, X refers to the source sentence, Y* refers to the given standard target sentence, and R D , Q G θ refers to the action-value function from source sentence X to target sequence.

The BLEU value of a sentence is used as a generator, and the n-element syntactic precision of sentence y _g is computed. Based on the target standard sentence y _d , reward Q ( y g , y d ) is calculated. The calculation formula of R D , Q G θ is:

where b ( X , Y ) stands for the base value, which is set as 0.5 for simple calculation, and γ stands for a hyperparameter. To obtain a stable reward, an N-time Monte Carlo search is used to obtain the action-value, and the formula is:

where T i refers to the length of the sequence at the ith Monte Carlo search, ( Y 1 : T , X ) refers to the present state, Y 1 : T N N refers to the sentence generated according to the policy G θ , and MC refers to the Monte Carlo search function.

The generator is optimized by updating the discriminator used as the reward function, and the relevant calculation formula is:

where E is an anticipation function. The derivative of the target function J ( θ ) to the generator parameter θ is:

The generator parameter is updated as follows:

where a h refers to the learning rate at the h step.

4 Experimental analysis

The previous sections introduced two NMT models: the traditional Transformer model and the improved Transformer model combined with GAN. In this section, the experimental setup was described, the evaluation indicators of the model performance were introduced, the experiments were conducted on the linguistic data consortium (LDC) dataset, and the experimental results were analyzed in detail.

4.3 Translation results

The BLEU values of the two models are shown in Figure 2.

Figure 2

Comparison of BLEU values between different models.

It was seen from Figure 2 that the BLEU value of the improved Transformer model had some improvement; the BLEU value of the improved Transformer model improved by 0.66 in NIST02, 0.36 in NIST03, 0.1 in NIST04, and 0.48 in NIST05, and NIST08 improved the most. The BLEU value of the dataset was 32.26 in the Transformer model and 33.09 in the improved Transformer model, i.e., the value improved by 0.83. The average BLEU value of the Transformer model was 40.47, while the average BLEU value of the improved Transformer model was 40.96, i.e., there was an improvement of 0.49. The above results verified the reliability of the improved Transformer model in improving the translation quality.

Then, the perplexity values of the two models were compared, and the results are shown in Figure 3.

Figure 3

Comparison of perplexity values between different models.

It was seen from Figure 3 that the perplexity value of the improved Transformer model was significantly reduced. Specifically, when using the improved Transformer model, the perplexity values of NIST02 reduced by 10.27, NIST03 by 9.16, NIST04 by 10.47, NIST05 by 9.45, and NIST08 by 10.98. The average perplexity value of the Transformer model was 26.89, while the average perplexity value of the improved Transformer model was 16.83, which was reduced by 10.06. The above results indicated that the probability of the model selecting the correct translation results somewhat increased, thus the model obtained results closer to the human translation.

The computational speed of the two models was compared, and the results are shown in Figure 4.

Figure 4

Comparison of computing speed between different models.

It was seen from Figure 4 that the computational speed of the Transformer model was 27,850 words per second, while the computational speed of the improved Transformer model was 25,890 words per second, which was only 7.04% lower than that of the Transformer model, indicating that the improvement of the Transformer model by combining GAN did not have a particularly significant impact on its computational speed.

The translation results of the two models were compared and analyzed, as shown in Tables 2 and 3.

Table 2

Translation example 1

Source language	欧盟办事处与澳洲大使馆在同一建筑内。
Candidate translation 1	The EU mission is in the same building with the Australian embassy
Candidate translation 2	The European Union office and the Australian embassy are both located in the same building
Candidate translation 3	The European Union office is in the same building as the Australian embassy
Candidate translation 4	The EU office and the Australian embassy are housed in the same building
The translation of the Transformer model	The European Union offices with the Australian embassy in the same building
The translation of the improved Transformer model	The EU office is housed in the same building as the Australian Embassy

Table 3

Translation example 2

Source language	经过处理后的“中水”将率先在城市绿化浇灌中使用。
Candidate translation 1	The treated reclaimed water will first be used in city greenbelt irrigation
Candidate translation 2	The treated reclaimed water will be first used to irrigate urban greenery
Candidate translation 3	The treated middle-water will first be used in watering the trees in and around the city
Candidate translation 4	The treated reclaimed water will be first used in urban green area irrigation
The translation of the Transformer model	The treated intermediate water will be the first to be used in urban green irrigation
The translation of the improved Transformer model	The treated reclaimed water will be first used in urban green area irrigation

It was seen from Table 2 that the Transformer model translated “欧盟办事处” as “The European Union offices,” while the improved Transformer model translated it as “The EU office,” which was more accurate. The reason for the above problem may be that the Transformer model did not consider “The EU office” as a proper noun, leading to a wrong translation.

In Chinese semantic, the term “中水” means recycled water. It was seen from Table 3 that the Transformer model translated “中水” directly as “intermediate water” without considering the specific semantics. In addition, the Transformer model translated “率先…” as “be the first to,” which was more inclined to the meaning of “the first time to,” the Transformer model translated it as “be first,” which was closer to the original meaning of the source language and the reference translations.

5 Discussion

Advances in artificial intelligence have led to the rapid development of NMT, which has become a new paradigm for MT [20]. Compared with statistical MT, NMT does not require steps such as word alignment and sequencing, and its translation process entirely relies on the self-learning of neural networks, which greatly reduces the complexity of the model and significantly improves translation performance. However, the current NMT has some problems, such as low-frequency words and unknown words [21]; therefore, the research on NMT is of great importance.

This article mainly focused on the Transformer model. In order to improve the translation effect, GAN was introduced to improve the Transformer model. Then, the translation performance of these two models was compared and analyzed on the LDC dataset. The comparison of BLEU and perplexity values showed that the improved Transformer model had larger BLEU values and smaller perplexity values, which indicated that the similarity between the results of model translation and human translation was high, i.e., the translation quality of the improved Transformer model was high. In addition, the comparison of the translation speed (Figure 4) suggested that the computational speed of the improved Transformer model only decreased by 7.04% compared with the traditional Transformer model, which indicated that the improved model did not increase the computational complexity to a great extent. Finally, the two translation examples (Tables 2 and 3) demonstrated that compared with the Transformer model, the translation results obtained by the improved Transformer model were more consistent with the meaning of the source language and matched better with the reference translations, which verified the reliability of its translation effect and its application feasibility in practice. At present, Chinese-English translation has a broad demand in many fields, and the improved Transformer model can provide simultaneous translation in multilingual communication in international conferences and provide services for cross-language retrieval in academic fields, which has great application values in scenarios such as foreign trade and overseas travel.

In this article, although some results have been obtained from the study of English translation based on the improved Transformer model, there are some shortcomings. For example, the amount of the experimental data was small and the translated sentences were short. In future work, experiments will be conducted on a larger dataset, and the performance of the improved Transformer model on English paragraph translation will be investigated.

6 Conclusion

This article improved the Transformer model by combining GAN and conducted experiments on the LDC dataset. The performance of the Transformer model and the improved Transformer model in English translation was compared. The results showed that:

1. The average BLEU value and perplexity value of the improved Transformer model were 40.96 and 16.83, respectively, which were superior to those of the Transformer model;

2. The computational speed of the improved Transformer model was 25,890 words, which only decreased by 7.04% compared to the Transformer model;

3. The translation results of the improved Transformer model were closer to the reference translations.

The experimental results verify that the improved Transformer model can improve translation quality effectively while ensuring computational speed. In order to realize better application in actual translation scenarios, the improved Transformer model will be further studied in the future to analyze its performance in translating long and complicated sentences and paragraphs.