Development of Telugu-Tamil Transfer-Based Machine Translation System: An Improvization Using Divergence Index

3.1 Dorr’s Divergence

Dorr classified cross-linguistic distinctions into several categories of divergences across English, Spanish, and German to account for divergences in the UNITRAN (UNIversal TRANslator) MT system. She divided divergences into two major types: syntactic and lexical-semantic divergences. Syntactic divergences are majorly divided into five: constituent order, preposition stranding, long-distance movement, null subject, and dative divergences. Lexical-semantic divergences are divided into seven: conflational, structural, thematic, categorical, demotional, promotional, and lexical divergences. Dorr’s description of divergences is restricted to English, Spanish, and German. There can be many other divergences when different language pairs are involved, and the study of divergence needs further exploration. Following Dorr, a few divergence studies have explored Indian languages, such as English-Sanskrit-Hindi MT [6], English-Sanskrit MT [15], English-Hindi MT [3], [7], Sanskrit-Hindi MT [23], Hindi-Telugu MT [21], and English-Bengali MT [2], to name a few. In most of the cases, Dorr’s divergences are noticed as rare phenomena and do not pose similar problems as far as Telugu and Tamil are considered. Further, Dorr discussed the divergences involved in interlanguage-based MT, i.e. UNITRAN; however, transfer-based MT requires further exploration.

3.2 Telugu-Tamil Divergence

The current research attempts to classify divergences into three major kinds: morphological, syntactic, and lexical-semantic divergences.

A. Morphological Divergence

Morphological divergence refers to differences that are found in inflectional and derivational (only productive) devices of words between a pair of languages in MT. Open word class categories such as nouns, verbs, and adjectives and closed word classes such as pronouns, number words, and nouns of space and time (NST) are studied between Telugu and Tamil to find out morphological divergences. Uninflected word classes, i.e. indeclinables and non-productive derivational word forms, are excluded here because they are listed in the lexicon and a straightforward mapping between them solves the problem in MT. For example, nouns in Telugu and Tamil are major word classes inflecting for number and case. The divergences in these classes are exemplified below.

Example A.1: Number Marking

There are two numbers in Telugu and Tamil, viz. singular and plural. Singular has no particular distinguishing marker, and plural is marked with the basic allomorph -ka in Tamil and -lu in Telugu. In Tamil, the plural marking is found obligatory with rational nouns^[2] and optional with irrational nouns^[3] [13]. In Telugu, the plural marking is usually obligatory with nouns to denote a plurality (see Table 1).

The divergence in number inflection is expressed as

Te.^[4]NN[-rational]-<plural> => Ta.^[5]NN[-rational]-(<plural>).

Example A.2: Case Markers and Postpositions

Telugu and Tamil use a wide variety of case markers and postpositions to indicate various syntactico-semantic relations between nouns and verbs. The major inflectional differences in case marking occur due to two reasons: (i) the choice of items in terms of inflections, viz. the oblique stem formation, case, and postposition, and (ii) the order of their presentation. For instance, an example for the direction to an interior location is shown in Table 2, where the order of suffixes in Telugu and Tamil may differ.

This divergence is explicated as below:

Te. Noun- ±Number- ±Stem-formative- ±Postposition- ±Case.

Ta. Noun- ±Number- ±Stem-formative- ±Case- ±Postposition.

B. Syntactic Divergence

Syntactic divergence here refers to syntactic structural differences that occur between pairs of languages. Divergences due to case mismatches, agreement, anaphora, negation, subordination, and clitics are noticed. Various syntactic processing and a robust TG are obviously required to overcome syntactic divergence. Some examples of syntactic divergences between Telugu-Tamil are given below.

Example B.1: Case Marker

Each case marker has a number of functions, and it is obvious that they lead to case mismatches in MT. For instance, Telugu and Tamil agree in the usage of dative case marker in various functions, viz. the beneficiary of an action, the goal of motion, and the experiencer subject [10], [26], among other functions. However, to express a possessive relationship between two inanimate nouns, one of the nouns of inanimate category carries the dative marker to express the locative function in Telugu. Here, the dative case marker relates two noun phrases that have the holonymic (HOL)-meronymic (MER) relationship. The word that is a holonym takes the dative case marker in Telugu and, on the contrary, the locative case marker is in use in Tamil as in the following example.

Divergence rule:

Te. NN[+HOL]-DAT NN[+MER] => Ta. NN[+HOL]-LOC NN[+MER].

Example B.2: Infinitive Construction

The use of infinitive construction in Tamil is wide ranging, whereas its use is restricted and obsolete in Telugu except in the augmentation of the main verb with an auxiliary verb [11], [13]. Infinitives (INF) are used as complements to different clauses in Tamil, whereas those in Telugu may use gerundivals (GER).

For instance, the infinitive clause occurs as a complement to desiderative verbs such as virumpu “want,” ācaippau “desire,” etc. in Tamil. In this respect, the dative-marked gerundivals act as a complement in Telugu as explicated in the example below.

Divergence rule:

Te. VM-GER-DAT VM[+desiderative] => Ta. VM-INF VM[+desiderative].

C. Lexical-Semantic Divergence

Lexical-semantic translation divergences are characterized by properties that are entirely lexically determined between languages. A concept expressed by a lexeme may not have the similar meaning in all contexts. Lexical-semantic divergence in Telugu and Tamil occur mainly due to lexical ambiguities and MWEs.

Example C.1: Lexical Ambiguities

For example, a lexeme that is used to express a concept in a language may not have the same meaning in all contexts. When it has multiple meanings, word sense disambiguation is required to overcome lexical ambiguity and to select an appropriate sense with its form in the TL.

For instance, the lexeme kuṭṭu in Telugu is ambiguous and expresses three different senses as given below.

Sense 1:kuṭṭu “to bite,” as in the context of cīma “an ant,” etc. The equivalent word in Tamil is kaṭi “to bite.”

Sense 2:kuṭṭu “to stitch,” as in the context of baṭṭalu “clothings.” The equivalent word in Tamil is tai “to stitch.”

Sense 3:kuṭṭu “to pierce,” as in the context of cevulu “ears” or body parts, etc. The equivalent word in Tamil is kuttu “to pierce.”

Example C.2: MWEs An MWE is a collocation of words that often come together with non-compositional semantics. These forms are sequences of two or more words that generally express a co-occurrence meaning. Telugu and Tamil show a number of conflational divergences as shown in Table 3.

4.1 Morphological DI

In Table 4, morphological DI is marked. For example, the number inflection of nouns (see Table 1) can be explained with different levels.

L1 (surface level) checks whether nouns in both languages inflect for number. If so, L2 (shallow level) indicates whether the number distinctions, such as singular and plural, are realized or not. L3 (intermediate level) inspects for the distinct marker for singular and plural. L4 (deep level) identifies differences in the number marking, if any, as explained in Table 1. That is, the distinction between rational and irrational nouns may play a role in their inflection of plural marking. L5 (deeper level) checks for the deeper linguistic feature in which the languages show any other difference.

Table 5 and Figure 2 illustrate the morphological divergence quantification for different linguistic levels.

Figure 2:

Morphological Divergence.

4.2 Syntactic DI

The syntactic divergence is quantified for different levels as expressed in Table 6 and Figure 3.

4.3 Lexical-Semantic DI

The approximate lexical-semantic divergence is calculated based on the entries in synset lexicons available in both languages. Ambiguous words are studied, and about 6706 words show ambiguity in Telugu when they are mapped with Tamil. About 6798 MWEs are identified in Telugu and are given appropriate equivalents in the TL, Tamil.

Figure 3:

Syntactic Divergence.

5.1 Handling Morphological Divergence

Morphological divergences are handled by the following modules.

1. Parser: Telugu-Tamil MT employs a simple parser expressing two types of relations. It shows ka¯raka (K) relations of nouns with respect to verbs and other non-ka¯raka (R) relations between other constituents. ka¯raka is the one that performs an action and has various participants to execute the action. An action in a sentence is denoted by a verb and participants are through nouns, and the relationship between nouns and a verb is a K relation. Apart from K relations, a sentence contains other types of relations as well. These relations are marked by other relations (R), such as constituents expressing purpose, reason, genitive, etc. For instance, Telugu uses a similar case marker, i.e. -to¯, to denote instrumental and associative cases. Parser finds out the exact case name based on the context using the database of rules. It makes use of ontological features of words to disambiguate their roles in sentences.

2. TG: TG in the current MT is equipped to solve certain morphological divergences. It contains a list of rules matching SL morphological features and transfers them into the expected TL. For instance, in Telugu, the dative case marking with NST denotes the direction when they occur with verbs of motion. In Tamil, the dative case marker is disallowed in such construction.

   A TG rule to handle this divergence is written as below.

   R13: NP((%NST<lcat="nst",case="o",cm="ki">))

   VGF((%VM<root="$motion", lcat="v">)) =>

   NP((%NST<lcat="nst",case="d",cm="0">))

   VGF((%VM<root= "$motion",lcat="v">)).

   TG rules contain SL features on the left side and TL features on the right side with the delimiter =¿ in the middle. It gets information from the morphological analyzer, POS tagger, chunker, and parser outputs of SL, and transfers them into the TL.

3. LT: LT solves certain morphological divergences. Other than lexical words, LT also contains a list of functional words and their equivalents in the TL. For instance, the divergence due to auxiliaries in compound verb formation is handled by LT by substituting the SL auxiliary to the expected TL auxiliary.

4. AGR modules: Morphological divergence due to the agreement is solved by AGR modules. For instance, the gender distribution in the singular is different in Telugu and Tamil. AGR modules aim at providing the exact TL gender.

5. MG: MG plays an important role in handling most of the morphological divergences in the current MT architecture. Once the SL morphological features are transferred to the TL by LT, the MG generates well-formed word forms of TL.

6. Postprocessor: Postprocessor is also involved in providing provision for solving certain morphological divergence. For instance, the number word “one” has special adjectival forms such as oru (used before consonants) and ōr (used before vowels) in Tamil. However, in Telugu, it does not have this distinction. In this case, postprocessor selects an appropriate number word “one” by looking at the initial sound of the following words.

5.2 Handling Syntactic Divergence

Syntactic divergences are mainly handled by TG. The input for TG is the shallow parsed SL text. As mentioned, TG rules contain two components: the SL structure on the left side and the expected TL structure on the right side. The following major tasks are handled by TG [8].

1. Insertion: Insertion of a new node is possible by TG indicated by the symbol “+.” For instance, Tamil optionally uses copula verb in verbless constructions. To insert a copula, the following rule is executed:

   R1: NP(({%NN}{%SYM<root="&dot;",lcat="punc">})) =>

   NP(({%NN}{})) +VGF<root="āku",lcat="v">

   ((+{%VM<root="āku",lcat="v"}+

   %SYM<root="&dot;", lcat="punc">}))

   As seen in R1, a new node VGF with the copula verb āku is inserted in TL. The agreement features of copula are provided by AGR modules.

2. Deletion: Deletion in TG rule empties nodes. For instance, in R1, the symbol SYM in NP is deleted and inserted again in VGF. Once a node is deleted, it is shown by {}.

3. Modification: Modification in TG involves a change in the node features of TL. For example, when the predicate expresses capabilitative mood, the subject is marked with the instrumental case marker (-a¯l) in Tamil, whereas the subject in Telugu is in the nominative case (0). This is written as a TG rule below:

   R2: NP<lcat="n",case="d",cm="0", drel="k1:$x">(({%NN}))∗

   VGF<lcat="v",tam="a_gala_$a">(({%VM})) =>

   NP<lcat="n",case="o",cm="āl", drel="k1:$x">(({%NN}))∗

   VGF<lcat="v",tam="a_gala_$a">(({%VM})).

4. Reordering: Reordering of nodes in TG is possible. For example, in Telugu, a reduplicated noun would have a dative marked on the second segment for distributive sense, whereas in Tamil, the dative is marked on the first segment of the noun.

   Te. iMṭ(i) iMṭi- kī “each house”

   Ta. vīṭṭ- ukku vīṭu “each house”

   R3: NP(({%NN<root="$x",lcat="n",case="o",cm="0">}

   {%NN<root="$x",lcat="n",case="o",cm="ki">}))

   => NP(({%NN<root="$x",lcat="n",case="o",cm="ki">}

   {%NN<root="$x",lcat="n",case="o", cm="0">})).

5. File handling: A list of items as files is maintained to write a global rule. For instance, nouns [+body part] occur with the postposition paina/ mı.da to denote surface location in Telugu. Such nouns are marked with the locative case marker in Tamil. A file with the list of body parts is maintained to handle such divergences.

   V1: R4:: "$x=bodyparts.txt"

   R4: NP(({%NN<root="$x",cm="paina">})) =>

   NP(({%NN<root="$x",cm="il">})).

5.3 Handling Lexical-Semantic Divergence

Lexical-semantic divergences are mainly handled by two modules, viz. MWE component and LT. A word with a number of senses will definitely make a problem in MT. In Telugu and Tamil, it is common to find out ambiguous words, and they need to be systematically resolved to obtain the appropriate equivalent. An exhaustive set of TG rules operating on the identification of the ambiguous words and disambiguating them by looking at the subject or the object nouns as suggested above is built. For instance, the following TG rules are samples to handle the different senses of Telugu word kuṭṭu in Tamil:

V1:R1::"$x=animate.txt"

R1: NP<root="$x",lcat="n"> VGF<root="kuṭṭu",lcat="v"> =>

NP<root="$x", lcat="n"> VGF<root="kaṭi",lcat="v">

V2:R2::"$y=inanimate.txt"

R2: NP<root="$y",lcat="n"> VGF<root="kuṭṭu",lcat="v"> =>

NP<root="$y", lcat="n"> VGF<root="tai",lcat="v">

V3:R3::"$z=bodyparts.txt"

R3: NP<root="$z",lcat="n"> VGF<root="kuṭṭu",lcat="v"> =>

NP<root="$z", lcat="n"> VGF<root="kuttu",lcat="v"> .