Neural Machine Translation for Indian Languages

1 Introduction

Machine Translation (MT), an application area of Natural Language Processing (NLP) and a subfield of computational linguistics, facilitates automated translation of text or speech in a source natural language to corresponding text or speech in a different target natural language. Language incomprehensibility has wide-ranging adverse impacts on several aspects of human living, and the same can be reasonably alleviated with effective use of MT. Besides, the crucial idea of MT is to bridge communication barriers among people from different linguistic backgrounds.

Although MT-predicted translations differ from human-like translation, they are comprehensible and the translation process is free from human intervention. The effectiveness of the translation approach is manifested in its potential to ensure generation of semantically equivalent and grammatically sound target construct. An intellectual translation approach refrains from a word-for-word translation behavior and delves into conceptuality and crux of the languages, prior to translation.

Classical approaches of MT are broadly categorized into rule-based, corpus-based, and hybrid approaches. Rule-based approaches, namely transfer-based and interlingua-based approaches, rely on a set of predefined translation rules, and investigate the syntax, semantics, and morphology of the two languages to furnish target representation. Rule-based approaches resort to linguistic models to ensure comprehensibility of translation rules and to produce a syntactically and semantically sound translation that is well formed and less prone to grammatical errors [21]. However, interlingua-based approaches are inefficient, primarily because of their impractical and infeasible idea of using language-independent representation for translation.

Corpus-based or data-driven approaches, namely Example-Based Machine Translation (EBMT) and Statistical Machine Translation (SMT), dynamically build a translation model that is characterized by a set of translation rules learned from parallel corpus. A considerable number of instances in the corpus, and the ability of the translation approach to dynamically dig out excellent rules guide the result of translation to perfection. Given the test data (text to be translated), EBMT investigates the translation model to seek the optimal match using techniques of clustering or generalization. Parallel corpora, thesaurus for computing semantic similarity, bilingual dictionary, and syntactic parser are the key resources that are crucial to the functioning of EBMT. On the other hand, SMT employs Bayes’ theorem to reformulate the translation problem as a probability maximization problem. Considering S to be the source language sentence and T to be the target language sentence, SMT attempts to find translation T that maximizes P(T|S). Using Bayes’ theorem, P(T|S) can be rewritten as the right-hand side of Eq. (1):

With P(S) being fixed, the problem of translation boils down to finding a translation T, which maximizes P(S|T)*P(T). Language model, P(T), and translation model, P(S|T), are learnt from the parallel corpora and constitute indispensable components of SMT. The two components are exploited by the decoding algorithm to predict target sentences. The inability of the SMT system to exploit context information of the source sentence, system complexity, and use of many different independently trained components are the key factors that add to the inefficiency of SMT systems.

Although conventional approaches of translation have served the purpose for years, their underlying demerits and the need for better-quality translation enforce exploration of new, better-performing techniques. Neural Machine Translation (NMT), a fairly new and proficient approach to translation, incorporates the use of adequately trained large neural networks in the translation process. Encoder and decoder, which are the networks of Long Short-Term Memory (LSTM) units, constitute key components of the NMT system architecture. In a baseline system, encoder encodes the source sentence, one symbol at a time, and stores the entire encoding in its last hidden state. Encoded representation is fed to decoder for translation prediction. The comprehensibility, adequacy, and fluency of predicted translation are largely determined by the implementation approach used for decoder. Recent past has witnessed the use of Convolutional Neural Networks (CNNs) as well as Recurrent Neural Networks (RNNs) for implementing the decoder network [4; 23]. However, carrying encoded representation along the decoder network is a primary requirement of the decoding process, and simple RNNs are bad at it. An improved implementation strategy for decoder replaces simple RNNs with LSTM, which memorizes encoded information and carries enough of it along the network. Use of attention mechanism [14] furthers the effectiveness of translation by allowing decoder to access the entire pool of encoder states for translation prediction. The NMT system endows scalability and a radical shift from phrase-based translation, as in the case of SMT, to sentence-based translation. Unlike classical MT systems, NMT systems are end-to-end systems that discard the use of additional components for translation. Decoder in NMT exploits a comparatively larger context, comprising source as well as partial target text, for accurate translation prediction. NMT is better at generating more fluent translations with less syntactic and semantic errors.

Motivated by the advantages of NMT over classical MT systems and the promising results produced by NMT in recent years, we have investigated its effectiveness in the context of Indian languages. In particular, we have trained and tested NMT systems for English to Tamil, English to Hindi, and English to Punjabi translations. Predicted translations have been evaluated by employing human evaluators and Bilingual Evaluation Understudy (BLEU) evaluation [18]. Besides, we have comprehensively analyzed the implications at the performance of the English-Hindi NMT system, given changes in training data, epochs, and length of sentences in the test set.

The rest of the paper is organized as follows: Section 2 reviews relevant literature on translation of Indian languages. Section 3 details the system architecture for NMT. Section 4 describes details of different experimental setups. Section 5 describes system results and their comprehensive analysis. Section 6 concludes the paper and points the direction for future research.

2 Related Works

The era of NMT emerged in 1987 when English to Spanish translation was attempted using backpropagation neural network and a highly limited vocabulary [1]. The translation process used backpropagation mechanism for mapping from one language to another. The NMT system architecture has been subjected to a number of modifications after its resurgence in 2013. An RNN encoder-decoder NMT system facilitates encoding of a variable-length source sentence into a fixed-length vector and decoding of fixed-length vectors to obtain the target sequence [3]. Gated Recurrent Units replaced CNNs [8] for implementing hidden units of decoder. LSTM [22] offers improved propagation of the encoded source sentence along the decoder network, resulting in improved translation quality of longer sentences.

Use of neural networks on phrase-based SMT has been limitedly explored in the context of Indian languages, such as Tamil and Punjabi. Neural networks have been used to learn sets of ordered rules for Hindi to English translation, and the idea can be extended to other Indo-European languages by changing the dictionary used for literal translation [2]. A feed-forward backpropagation Artificial Neural Network (ANN) architecture uses nine separate modules for translating simple sentences of the English language into Hindi [9]. Bilingual dictionary, and storing word meanings and word features of a language pair have been implemented using ANN. A quantum neural network-based approach for English to Hindi translation learns the pattern of the English-Hindi parallel corpus using part-of-speech information of each of the word in corpus, and later uses the gained knowledge to perform translation [16]. Use of quantum neural network for reordering of words for parts-of-speech tagging and their alignment during the MT has been found to increase the translation accuracy to a significant extent. Translation accuracy can be further enhanced by increasing the size of the training corpus through automatic extraction of parallel texts from comparable corpora and adding them to the training data [17].

A direct translation methodology for Punjabi to Hindi translation exploits the syntactic and semantic similarity between the two languages [7]. Using a number of lexicons and the word-for-word translation approach, words in the source language are replaced by their target language equivalents. An extended web-based Hindi to Punjabi MT system facilitates website and email translation [5].

Ambiguities of content as well as some function words pose challenges to MT systems. A supervised method for resolving prepositional ambiguity in English to Tamil translation performs disambiguation by exploiting collocation occurrences and linguistic information of words [12]. English to Tamil translation confronts issues of non-availability of parallel corpora and morphological difference between the two languages [11]. Such issues have been tackled by incorporating linguistic knowledge in SMT. Formalism-based MT approach for English to Tamil translation uses synchronous TAG pairs derived from XTAG English grammar [15]. Proposed tag systems can be well extended to other Indian languages as well. Furthermore, integrating rule-based MT systems with functionality for complex sentence simplification has been found to improve the quality of English to Tamil translation [20]. Complex sentences, connected using connectives, are split into simpler sentences before feeding them to the translation system.

Owing to the diversities of Indian languages and exceedingly better performance of NMT, Google has recently employed NMT in multilingual translation of nine Indian languages (http://indianexpress.com/article/technology/tech-news-technology/googles-neural-machine-translation-for-indian-languages-heres-what-it-means/), namely Hindi, Bengali, Marathi, Tamil, Telugu, Gujarati, Punjabi, Malayalam, and Kannada. Google’s multilingual NMT system is characterized by simplicity, low-resource language improvement, and zero-shot translation features [6]. The zero-shot translation feature enables the system to do translation prediction for a previously unseen language pair. Google’s multilingual NMT system does not affect the basic encoder-decoder architecture but instead uses a special token at the beginning of source sentence to specify the target language.

The Shared Task Cum Workshop on Machine Translation in Indian Languages (MTIL; http://nlp.amrita.edu/mtil_cen/) offers research infrastructure for the development, evaluation, and comparison of the MT systems [13]. The specific objectives of the workshop were to design high-quality parallel corpora of Indian languages, to explore the role of state-of-the-art MT techniques in the context of Indian languages, and to cope with the issues of language divergence. A total of five teams participated in the workshop. CDAC-M was the top-ranked team in English-Malayalam, English-Tamil, and English-Hindi translation categories, whereas our NIT-M team was top ranked in the English-Punjabi translation category. Human evaluation score was preferred over BLEU score for ranking of the teams.

The factored SMT model of the CDAC-M team employs suffix separation (SS), source side reordering, and transliteration for all the four translation categories [19]. Reordering reorders the source side sentences as per the word order of target language. Reordering leads to better alignments and parallel phrase extraction, which eventually improves the translation quality. During SS, words are split into stem and suffixes, and a continuation symbol (@@) is added after the stem word. The continuation symbol helps combine suffixes after having performed the translation. Transliteration, a post-processing step, translates an out-of-vocabulary (OOV) word to the target language word. An unsupervised model based on expectation maximization has been used to train a transliteration model. Eventually, language modeling selects the best translation from the n-best transliterated output. Augmenting Moses (https://github.com/moses-smt)-based baseline system with pre-processing and post-processing steps have led to improvement in BLEU score for the English-Hindi and English-Tamil translation categories.

3 System Description

Data pre-processing, system training, and system testing/translation constitute key steps of system functioning, and the same have been elaborated in the following subsections. We have exploited the OpenNMT (http://opennmt.net/) system architecture, tuned its parameters, and trained and tested the system using corpora provided by MTIL organizers [13].

3.2 System Training

We have trained a sequence-to-sequence recurrent neural network model, using attention mechanism, for translation prediction. Training data are shuffled and sorted prior to training. Shuffling ensures that instances in the training batch uniformly come from different parts of the corpus. Sorting ensures uniform length of instances in the training batch. Training performance can be escalated with use of multiple Graphics Processing Units (GPUs) to train different batches of training data in synchronous or asynchronous fashion.

An epoch in training refers to one forward and one backward pass over all the training instances. The system is trained for some fixed number of epochs. An epoch comprises iterations, and in each iteration one forward pass and one backward pass are performed over a set of training instances. The system has been trained for 15 epochs in the first experimental setup and 19 epochs in the other three setups (refer to Section 4.2 for details of the experimental setups). A validation score, dynamically computed using validation data, helps in checking the convergence of training. The learning rate of the network decays by a factor of 0.7 if validation score improvement falls below 0.

The primary components of the system architecture are discussed next.

3.2.1 Encoder

A unidirectional sequencer has been used as encoder for encoding variable-length input sequence of the source language into fixed-size vectors. The architecture of encoder is characterized by a two-layer LSTM recurrent neural network having 500 hidden units in each layer. LSTM, unlike conventional backpropagation neural networks, remembers encoded source representation and carries enough of it along the decoder network.

To convert the input sequence into word embedding, encoder splits the input sequence into an array of words. Each word in the array is mapped to its index in the vocabulary, and the index of the <unk> word is used for OOV words. Sorting of the input sequences in a batch, prior to training, eliminates the need for zero padding. Thereafter, each index value is transformed into a vector of fixed length and different word vectors are combined to give a single fixed-length vector that is representative of the complete input sequence.

3.2.2 Decoder

Similar to encoder, a two-layer LSTM decoder, having 500 hidden units in each layer, has been used to decode a fixed-size source vector using input feeding and global attention mechanism. A decoder using global attention mechanism consults the entire pool of source states at each step of decoding. Not only the last hidden state but also the entire hidden states of source are considered as representatives of sentence meaning. Score function takes the current hidden state of decoder h_t and the source vector h¯_s as argument to determine the attention score of each source state. The score function used by the system is given by Eq. (2):

(2)score(ht,h¯s)=htTWah¯s.

The probabilistic attention score of a state is its score divided by sum of all the attention scores. Variable-length weight alignment vector a_t uses probabilistic attention scores and gives an estimate of the amount of attention to pay at different places in the source. A context vector c_t is then computed using source vector h¯_s and weight alignment vector a_t. The decoder uses its current hidden state information h_t and context vector c_t to predict attentional vector h¯_t, which eventually predicts the current word y_t. Figure 1 describes working of decoder in a nutshell [14].

Figure 1:

LSTM Decoder Using Context Vector and Current Hidden State for Translation Prediction.

Moreover, the system uses the input feeding approach to feed attentional vector h~_t to current hidden state, a mechanism that keeps the system informed about past alignment decisions.

Figure 2 illustrates the system architecture. Attention mechanism and input feeding are used to transform input sequence “A B C D” into target sequence “X Y Z” [14].

Figure 2:

System Architecture for NMT.

4 Experimental Design

This section contains detailed description about the experimental setup and corpora used for training and testing the translation effectiveness of the NMT system. OpenNMT, an open source toolkit, facilitates the required experimental framework, and provides a platform for training and deploying NMT models [10].

5 Results and Analysis

5.2 Performance Comparison of NIT-M and CDAC-M Systems

Figures 3–5 show the comparison of adequacy, fluency, and BLEU scores of NIT-M and CDAC-M systems for the three translation categories.

Figure 3:

Comparison of Adequacy Scores of NIT-M and CDAC-M Systems.

Figure 4:

Comparison of Fluency Scores of NIT-M and CDAC-M Systems.

Figure 5:

Comparison of BLEU Scores of NIT-M and CDAC-M Systems.

The three scores of the NIT-M team are lower than those of CDAC-M for the English-Tamil translation. This owes to the agglutinative nature of Tamil and the morphological divergence between the English and Tamil languages. The majority of Tamil words are formed by a combination of multiple words, a phenomenon referred to as agglutination. Agglutination and morphological divergence lead to generation of more unknown (<unk>) words, which lowers the quality of translation. As discussed in the Section 2, the CDAC-M system employs SS, source side reordering, and transliteration to cope with the challenges of agglutination and morphological divergence in the English-Tamil translation. However, our NIT-M system makes no attempt to handle such linguistic issues, hence the poor performance.

The remarkable performance of the NIT-M system for the English-Hindi and English-Punjabi language pairs owe to a large-sized corpus. Training using a large corpus ensures effective tuning of system parameters and generation of a sound translation model, which eventually leads to better predictions. Moreover, the performances of two teams are comparable in these two translation categories. Although the Tamil training corpus is also fairly large sized, agglutination and morphological divergence cause the NIT-M system to lag behind in this translation category.

Furthermore, for English-Punjabi translation, the human and BLEU evaluation scores are negatively correlated, with low BLEU score and high manual evaluation scores. This is attributed to the underlying working principle of BLEU, which relies on the precision of n-grams in the reference and candidate translation. Even minor lexical differences can cause a huge difference in n-gram precision, which eventually affects the BLEU score. However, such minor lexical differences are often considered insignificant from the perspective of human evaluation.

Moreover, the comparatively lower evaluation scores of the CDAC-M system for English-Punjabi translation can be attributed to the inability of preprocessing and post-processing steps to handle the language-specific constructs of Punjabi language.

5.3 Evaluation Results of Different Experimental Setups

Furthermore, we have examined different experimental setups, as mentioned in Section 4.2, to analyze a system’s translation performance with respect to number of epochs, size of training data, and average length of sentences in the test dataset. We have used a multi-BLEU (http://www.statmt.org/moses/?n=Moses.SupportTools) evaluator, using 1-g precision, to compare Gold Data and predicted translations.

Figure 6 shows the BLEU score versus epoch plot for English-Hindi MT. The highest BLEU score of 52.54 is attained at epoch 18 and the BLEU score curve converges after epoch 16 (say, epoch_converge). The decreasing curve between epoch 6 and epoch 7 is presumably because of decay in the learning rate of the network, which leads to poor translation prediction. The learning rate of network is decayed if the validation score improvement falls below zero. As the BLEU score versus epoch plot converges after a specific epoch (epoch_converge), the plot can be helpful in selecting the maximum number of training epochs. The number of training epochs should preferably not exceed the value epoch_converge.

Figure 6:

BLEU Score Achieved by NMT System at Different Epochs.

Furthermore, Figure 7 shows a performance comparison between the NMT system trained using complete training data (NMT-I) and the one trained using half the training data (NMT-II). The high BLEU scores for the former indicate that the system’s performance considerably improves with increase in the number of instances in the training data. As NMT-I has seen more number of sentence pairs during training, it predicts correct Hindi translation of many English words and generates less <unk> words in comparison to NMT-II.

Figure 7:

BLEU Score Achieved by NMT-I and NMT-II at Different Epochs.

To further analyze the system, we have selected the best-trained model, the one obtained at epoch 18 of the second experimental setup, and tested it using four distinct test sets. The four test sets (each of size 100) have been derived from the original test set provided by organizers, in such a way that the average lengths of sentences in these sets are 10, 15, 20, and 25. Sentences longer than 25 words have been accommodated in the fourth test set. Also, sentences having an average length of less than 10 words have been dropped. Figure 8 shows the BLEU scores achieved by the NMT system for the four test sets.

Figure 8:

BLEU Score Achieved by NMT System for Different Sentence Lengths.

It has been interesting and uncommon to observe that the performance of the NMT system (in terms of BLEU score) improves with increase in the length of test sentences. It probably owes to the context-analyzing ability of the NMT system. Use of attention mechanism and context vectors facilitate improvement in translation performance with increase in length of test sentences. At each translation step, decoder uses global attention to investigate the entire pool of source states. Context vector generated from source states, current hidden state of decoder, and partial target text are then used by decoder for translation prediction of the current word. Thus, the entire decoding process in NMT relies on the context of the source sentence. As the larger source (test) sentences are context rich, their predicted translations are better in quality.

6 Conclusion and Future Scope

NMT, a fairly new approach to MT, uses an adequately trained end-to-end neural network system for translation prediction. The ability to produce fluent translation, better context-analyzing abilities, and improved performance over the SMT system are some of the key benefits that motivated us to explore the usage of NMT in the context of Indian languages. In particular, we have trained and tested NMT systems for English-Tamil, English-Hindi, and English-Punjabi language pairs. Predicted translations were provided to MTIL organizers for human and BLEU evaluations whereby translation quality was evaluated on the grounds of adequacy, fluency, and an overall rating. Besides, different experimental setups have been designed to analyze the change in the translation performance of the English-Hindi NMT system with change in the number of epochs, training data, and length of test sentences. A close analysis of predicted translations guide us to the conclusion that NMT systems produce fluent translations and their performance improves with increase in training data and length of test sentences. Moreover, a translation performance versus epoch plot can be helpful in checking the convergence of system training.

The underlying working idea of NMT relies heavily on the size of the training corpus, which directs us to increase the number of instances in the training corpus. The effectiveness of translation is largely determined by attention mechanism and the score function used for computing the attention of each source state. The score function can be modified to increase the interaction between the source state vector h¯_s and the current hidden state of decoder h_t. Besides, a better grasp over the crux of target language constructs can help improve the comprehensibility, adequacy, and fluency of translation. Furthermore, a skillful and careful selection of values for system parameters such as number of epochs, hidden layers, GPUs, etc., can also add to translation quality.