A study of an intelligent algorithm combining semantic environments for the translation of complex English sentences

2.1 Baseline model

In this paper, two mainstream NMT models were used as baseline models for translating complex English sentences. One was a recurrent neural machine translation (RNMT) model, and the other was a transformer model based on a self-attentive mechanism. These two models are briefly described next.

The RNMT model [12] directly translated source sentences into target sentences. Before a source-side sentence entered the encoder, it was written as a word vector sequence:

where X is the source sentence, n is the total number of words in the sentence, and x _i refers to the word vector of the i-th word. The encoder of the RNMT model was a bidirectional RNN, and the forward RNN reads the source sentence from left to right and outputs the forward hidden state:

The calculation formula of h i → is

where f is the nonlinear activation function.

Similarly, the reverse RNN reads the source sentence from right to left and output the reverse hidden state:

The hidden state of the source statement was obtained after splicing the above two states:

Then, the decoder used a one-way RNN to predict the target sentence: {y ₁, y ₂, …, y _m }. The calculation formula of the i-th predicted word y _i is

where s _i refers to the current hidden state and c _i refers to the current source context vector. The calculation formula of c _i is

where a _ij is the attention weight, h _j is the hidden state of the j-th word of the source sentence, and e _ij is the matching degree between the i-th output word and the j-th input word.

The transformer model [13] also consisted of two parts, and it differed most from the RNMT model in that it used a multi-head attention mechanism [14] that represented every word by three vectors: Query (Q), Key (K), and Value (V). A multi-head attention mechanism contained multiple dot-product attentions. The calculation of the dot-product attention is

where d _k refers to the dimensionality of K. Then, multiple dot-product attentions are spliced and put into a feedforward neural network (FFN):

If there were training parameters, W ₁, W ₂, b ₁, and b ₂, FFN took rectified linear unit as the activation function, then

Finally, the transformer model used the positional encoding method to add location information, and the calculation formula is

where pos refers to the position of a word in a sentence and i is the vector dimension.

2.2 Translation algorithms that combine semantic environments

In MT, there are some common problems, such as unknown words, missed translations, and over-translations [15], which leads to poor translation quality. This paper mainly analyzed the problem of unknown words and designed an intelligent algorithm by combining the semantic environment.

Unknown words refer to words not covered by the word list. Models cannot correctly interpret unknown words in the translation process and uniformly replace them with “<UNK>,” which is prone to ambiguity and affects the translation results. This paper translated unknown words by replacing them with suitable words obtained through calculating semantic similarity based on the semantic environment. Based on the WordNet semantic network [16], the synonym of unknown words was found from WordNet after processing the source sentence by word segmentation. In the WordNet semantic environment, there are a variety of semantic concept relationships, and two relationships were involved in this paper.

1. Synonymy: words that are identical or similar on the level of meaning.

2. Hypernymy/hyponymy: if a word is a subclass of another word, there is a hypernymy/hyponymy relationship between them. Taking “car” and “vehicle” as examples, “car” is the hyponym of “vehicle,” and “vehicle” is the hypernym of “car.”

   The specific process of the intelligent algorithm combined with the semantic environment is as follows.

   1. For a word w whose part of speech was w _pos, all words whose part of speech was w _pos were screened out from WordNet and denoted as synsets_w. Then, known words were screened out from those words and added to the synonym set of w.

   2. If the synonym set was empty, all the hypernyms of synsets_w were found out and denoted as hyber_synsets_w. Then, all the synonyms of the hypernym set were found out, and the known words were added to the synonym set of w.

   3. If the synonym set was empty, all the hyponyms of synsets_w were found out and denoted as hypo_synsets_w. Then, all the synonyms of the hyponym set were found, and the known words were added to the synonym set of w.

   4. According to the above steps, the synonym set of w was found as the candidate replacement word set w i ′ . The calculation of semantic similarity was based on the ternary language model, and the calculation formula is

      (18) Score 3 -gram ( w i ′ , w i ) = p ( w i ′ | w i − 1 w i − 2 ) + p ( w i + 1 | w i ′ w i − 1 ) + p ( w i + 2 | w i + 1 w i ′ ) 3 .

      The similarity score of two words was calculated based on their distance in WordNet, and the formula is

      (19) Score WordNet ( w i ′ , w i ) = 1 path ( w i ′ , w i ) + 1 ,

      where path( w i ′ , w _i ) refers to the distance between two words. Finally, the calculation formula of semantic similarity of ( w i ′ , w _i ) is

      (20) sim ( w i ′ , w i ) = Score n − gram ( w i ′ , w i ) ⋅ Score WordNet ( w i ′ , w i ) .

      Then, the replacement word for the unknown word is

      (21) w best = argmax w ′ ∫ S sim ( w i ′ , w i ) .

   5. The replaced sentence was input into the NMT model for translation. After the result was obtained, the original unknown word was put back. In this paper, a word alignment model was trained by GIZA++. A bilingual dictionary that includes all the known words was selected. For word t _i in the translation result, if its aligned word s _j was obtained by replacement and the original unknown word was s j ′ , then the alignment relationship was determined by the bilingual dictionary. If (s _j , t _i ) was in the dictionary, then the alignment relationship was proved to be correct, and t _i was replaced by the translation of s j ′ .

3.1 Experimental setup

The Chinese Treebank 6.0 corpus was used, including 1,067 files, 20,367 sentences, 647,523 English words, and 963,461 Chinese words. The development set for the experiment was National Institute of Standards and Technology (NIST) 05 in the Linguistic Data Consortium dataset. The test sets were NIST 02, 03, 04, 06, and 08. The specific data are shown in Table 1.

The word alignment was implemented with GIZA++. The RNMT model was RNNSearch⁷ [17]; the decay constant was 0.95; the denominator constant was 10⁻⁶; the word vector dimension was 620; the hidden dimension was 1,000; the output layer used the dropout strategy; the dropout rate was 0.5; and the beam_size was 10 at decoding. The transformer model was the tensor2tensor model [18], with six encoders, six decoders, and eight multi-head attention mechanisms; the word vector dimension was 512; the hidden dimension was 512; the internal dimension of FFN was 2,048; the beam_size was 4 at decoding; and the length penalty was 0.6.

3.2 Evaluation indicator

The indicators used were BLEU and translation error rate (TER).

1. BLEU: its principle was comparing the reference translation with the MT. The value of BLEU was between 0 and 1; the larger the value was, the higher the translation quality was. First, the matching accuracy p _n of the n-gram was calculated:

where count_clip(N-gram) refers to the number of occurrences of n-gram in the reference translation.

There was also a length penalty term BP in the BLEU value:

where c is the length of the reference translation and r is the length of the MT.

Ultimately, the formula for calculating the BLEU value is

where N is the maximum order of n-gram grammar and w _n is the weight of co-occurring n-gram, generally 1/n. In MT evaluation, 4-tuple is usually used; therefore, N = 4 in this paper.

(2) TER: it is a common indicator used for evaluating the performance of MT. Its principle was evaluating the number of different words in MT and reference translation. The smaller the TER was, the smaller the error rate of MT was, i.e. the higher the translation quality was.

3.3 Experimental results

The BLEU values of the baseline models and the models combined with the intelligent algorithm on different datasets were compared, as shown in Table 2.

It is seen from Table 2 that before combining the intelligent algorithm, there was a gap in the BLEU value between the two baseline models; on the development set, the BLEU values of the RNMT model and the transformer model were 35.12 and 43.68, respectively, i.e. the transformer model was 8.56 larger than the RNMT model, and the situation was the same on the test set, indicating that the transformer model had better MT performance than the RNMT model. After combining the intelligent algorithm, the BLEU values of both algorithms showed an improvement; for example, in the development set, the BLEU value of the intelligent algorithm + RNMT model was 39.33, which showed an improvement of 4.21, and the BLEU value of the intelligent algorithm + transformer model was 44.61, which showed an improvement of 0.93; the situations were the same on the test set, indicating that the intelligent algorithm designed in this paper was effective in improving the translation quality. In addition, since the transformer model has performed better than the RNMT model before improvement, its improvement was not significant as the RNMT model.

The average BLEU values of different algorithms were calculated, and the results are shown in Figure 1.

Figure 1

Comparison results of average BLEU values between different algorithms.

It was seen from Figure 1 that the average BLEU value of the RNMT model and the transformer model was 35.18 and 42.14, i.e. the transformer model was 6.96 larger than the RNMT model. After combining the intelligent algorithm, the average BLEU value of the intelligent algorithm + RNMT model reached 37.5, which showed an increase of 2.32 compared to the RNMT model, while the average BLEU value of the intelligent algorithm + transformer model was 43.7, which showed an increase of 1.56 compared to the transformer model and an increase of 6.2 compared to the intelligent algorithm + RNMT model. The experimental results demonstrated the advantages of the transformer model for MT and the reliability of the intelligent algorithm combined with the semantic environment designed in this paper for improving translation quality.

The average TER was compared between different algorithms, and the results are shown in Figure 2.

Figure 2

The comparison of the average TER between different algorithms.

It is seen from Figure 2 that the average TER of the transformer model was 1.39 lower than that of the RNMT model (59.33 vs 60.72); the average TER of the intelligent algorithm + RNMT model was 59.37, which was 1.35 lower than that of the RNMT model; the average TER of the intelligent algorithm + transformer model was 57.68, which was 1.65 lower than that of the transformer model and 1.69 lower than that of the intelligent algorithm + RNMT model. The above results demonstrated that the MT obtained by the intelligent algorithm + RNMT model had fewer mistakes and higher quality.

Two complex statements were used as examples to analyze the details of the translation, as shown in Table 3.

In Table 3, the first example sentence contained an unknown word, “snow-covered,” which was not translated in the translation of the baseline system but was correctly translated after combining with the intelligent algorithm. In addition, the translation of the transformation model was more fluent than that of the RNMT model; for example, the translation of the word “fallen” was more contextual in the transformer model. In the second example sentence, which also contained an unknown word, “intelligence,” RNMT and transformer models did not translate it. In addition, the RNMT model translated “removed” to “带走,” which was not appropriate enough. In terms of word order and semantics, the results of the intelligent algorithm combined model were closer to the reference translation, which verified the reliability of the model in translating complex English long sentences.