Supervised prediction of production patterns using machine learning algorithms

1 Introduction

Loanword adaptation (or repair) is the process by which a word’s original structure is altered to phonologically fit the borrowing language; that is, a word from a source language cannot enter a borrowing language without being phonologically adapted (Haspelmath 2009). However, there are some loanword patterns that cannot be explained by the phonological grammar of the borrowing language. One of those patterns involves what Peperkamp (2005) calls unnecessary repair, in which loanword repair occurs even when there are no illegal sequences in the native phonotactics of the borrowing language. Korean provides a particularly interesting case of this apparently unnecessary adaptation: a voiceless stop is permitted in the word-final position in the Korean native phonology (e.g., [pap˺] ‘meal’, [ap˺] ‘front’), so we would expect that words borrowed from English ending in a voiceless stop would be adapted as ending in a Korean voiceless stop, but it is surprising to find that a vowel is frequently inserted after the final stop (see Table 1). Previous studies have suggested that different factors affect the possibility of vowel epenthesis in this environment, including vowel tenseness, stop voicing, stop place, final stress, and word size (Jun 2002; Kang 2003; Kim 2018; Kwon 2017; Rhee and Choi 2001), as indicated in Table 2.

Since predicting the specific adaptation pattern in any case is not so easy even with the many possible features that have been proposed by prior research, the purpose of the present study is to identify which production pattern will be chosen if a new English word ending in a stop is adapted into Korean. That is, this study attempts to find out which linguistic factor serves to produce the optimal output. In order to discover the best predictor, a corpus was built by collecting loanword lists published by the National Academy of the Korean Language (2001, 2002, 2007a, 2007b, 2010). This corpus consisted of 540 Korean loanwords from English whose English source word ends in a stop; out of 540 English words with a coda stop, 264 were consistently adapted with final vowel insertion and 214 were always adapted without final vowel insertion, while 62 were variably adapted both with and without vowel insertion.

This study employed a classification technique as a predictive model (Hastie et al. 2009; Kuhn and Johnson 2013) and a classification task was conducted using support vector machines (SVMs; Cortes and Vapnik 1995). Introduced by Cortes and Vapnik (1995) for binary classification, SVMs have been increasingly used for research in various areas including linguistics, communication, bioinformatics, and so on (Razzaghi et al. 2016; Rodríguez-Pérez et al. 2017; Schohn and Cohn 2000; Sebastiani 2002; Yang 1999). SVMs are a popular choice for classification, with several advantages. They are less prone to overfitting, which occurs when a model learns the training data too well but fails to generalize to new data. SVMs find a balance between maximizing the margin and minimizing the training error, which allows them to generalize well to unseen data. Also, they are memory efficient and can handle datasets with a limited number of training instances.

Despite the strengths of SVMs, it is acknowledged that their trained parameters can be difficult to analyze and interpret. Thus, a different classifier, random forests (RFs; Breiman 2001), was also applied on the same data to properly interpret the contribution of linguistic factors and to see if similar variable importance results are obtained by both SVMs and RFs. RFs are one of the most versatile machine learning algorithms as they do not require many tunings of their setting. They have been applied to sociolinguistic data and to acoustic cue weighting in perception (Brown et al. 2014; Tagliamonte and Baayen 2012). Breiman (2001) proposed RFs as an ensemble learning technique that constructs a forest by growing independent decision trees for classification tasks and a decision tree is built by randomly selecting a subset of the data. This process is repeated multiple times and prediction is made based on a majority of votes (Liaw and Wiener 2002). The algorithm splits the dataset into two sets: the training set that comprises approximately 66.7 % of the data (in-bag set) and the testing set that consists of the remaining 33.3 % of the data (out-of-bag set). Then samples are selected from the dataset without replacement during the growth of the forest (Strobl et al. 2009). Since no study has conducted analyses based on classification in this specific context of Korean loanwords from English, the contribution of this study is that it provides methodology where we can classify different patterns of loanword adaptation and predict a pattern given a great deal of corpus data comprising established loanwords. The goal of the current study is to demonstrate how machine learning classifiers can be utilized to determine the primary phonological factors that account for variable adaptation of loanwords in Korean.

2 Methods

In this study, the adaptation pattern of vowel insertion was modeled using an ordinal fixed-effects logistic regression model implemented in the ordinal package (Christensen 2022) in R (R Core Team 2022) to assess the significance of the linguistic factors that have been proposed to affect vowel insertion. An ordinal regression model was employed since the analysis involved ordered dependent variable categories, with no vowel insertion (NVI) being the lowest and vowel insertion (VI) being the highest (see Table 1). The vowel insertion pattern was calculated by building a single model where the dependent variable was the adaptation pattern (NVI, OVI [optional vowel insertion], or VI) and the five context predictors were vowel tenseness (lax or tense), stop voicing (voiceless or voiced), stop place (labial or coronal or dorsal), final stress (unstressed or stressed), and word size (monosyllabic or polysyllabic). All fixed factors were coded using treatment coding with the reference level for the intercept being set to lax, voiceless, labial, unstressed, and monosyllabic. Ordinal logistic regression was run in this study instead of multinomial logistic regression since the former is more parsimonious in that it would return only one coefficient describing adjacent pairs of the outcome levels rather than a separate coefficient for every possible pair. This model, however, could not incorporate a random-effects structure (i.e., by-item intercepts) because the variance-covariance matrix of the parameters could not be calculated, which might be explained by the fact that the present data do not involve hierarchical structures with few grouping levels, where the estimation of random effects would become unstable, leading to decreased model reliability.

The current study employed the SVM algorithm in Python (Python Software Foundation 2022) to help determine which category of adaptation patterns a new data point belongs to if it is adapted into the borrowing language. This computational study involved two different types of classification, binary classification (NVI or VI) and ternary classification (NVI, OVI, or VI). The binary classification classifies all the data points as ending in a C(onsonant) or a V(owel) or both (for the OVI case). It classifies ending in a consonant as true (1) or false (0), and ending in a vowel as true (1) or false (0): that is, it has binary C (true or false that the word ends in a consonant) as well as binary V (true or false that the word ends in a vowel). In the binary C classification, the presence or absence of C is focused: if a vowel is inserted after the final stop and a word ends in a vowel then C is false, whereas if a vowel is not inserted and a word ends in a consonant then C is true; but if a word has both VI and NVI cases, then C is also true. Similarly, in the binary V classification, the presence or absence of V is focused: if a vowel is not inserted and a word ends in a consonant then V is false, if a vowel is inserted then V is true, and if a word belongs to the OVI case then V is also true. On the other hand, the ternary classification classifies the pattern of each data point as NVI, OVI, or VI. In this study, we expected to obtain better results with higher accuracy in the binary classification models rather than in those with ternary classification, taking into account the different chance levels.

The production patterns following the English final stop were fitted using the sklearn package in Python (Pedregosa et al. 2011). We fitted separate models to two classification types, the binary classification (NVI or VI) and the ternary classification (NVI, OVI, or VI). Each model had class as the outcome variable (i.e., NVI, OVI, or VI) and the linguistic attributes as the predictor variables (i.e., lax, tense, voiceless, voiced, labial, coronal, dorsal, monosyllabic, polysyllabic, unstressed, or stressed; see Table 3). For this study, data preprocessing was conducted for experimental setup since SVMs handle categorical variables, for which a dummy variable is created with case values as either 0 or 1. SVMs require each data case to be represented a vector of real numbers and hence the original loanword list was converted into numeric data. A categorical variable consisting of two levels is represented by a set of two dummy variables, and a variable consisting of three levels is represented by three variables. That is, a two-category attribute such as {voiced, voiceless} was represented as (0, 1) and (1, 0); a three-category attribute like {labial, coronal, dorsal} was represented as (0, 0, 1), (0, 1, 0), and (1, 0, 0). Each attribute set was randomly divided into two subsets: a training subset comprising 75 % of the data and a testing subset comprising the remaining 25 %. All models were fitted using a linear kernel since it is the simplest among the kernel functions used in SVMs and its simplicity makes it a suitable choice when dealing with limited training data. The set type of SVMs that was chosen was nu-SVR, which is relatively robust to outliers and allows adjusting the flexibility of the model by using the nu parameter. Model performance was evaluated using a five-fold cross-validation. To avoid the challenges of working with a limited dataset, we employed Monte Carlo cross-validation (Kuhn and Johnson 2013; Picard and Cook 1984). This method consisted of repeating the train/test process 100 times for each model with random train/test splits each time. After conducting the 100 iterations, we calculated the average of the classification rates from each iteration to obtain the overall classification rate.

3 Results and discussion

The regression model built for the corpus analysis found significant effects of several linguistic factors. Figure 1 shows the predicted probability of vowel insertion patterns classified as VI, OVI, and NVI depending on each predictor. First, tense pre-stop vowels significantly increased the likelihood of each pattern relative to NVI (see Table 5 in Appendix). The model shows that NVI patterns (reference category) were predicted to be significantly more common than OVI patterns at baseline (i.e., tenseness = lax, voicing = voiceless, place = labial, size = monosyllabic, and stress = unstressed); VI patterns were predicted to be significantly less common than OVI patterns at baseline. These results are shown by the significant positive estimate values (β = 3.397, p < 0.001 for NVI vs. OVI; β = 4.456, p < 0.001 for OVI vs. VI). Importantly, tense pre-stop vowels significantly increased the predicted probability of VI patterns relative to other patterns, compared with the reference likelihood (β = 3.323, OR = 27.74, p < 0.001). Second, words ending in voiced stops also significantly changed relative rates of VI patterns in relation to words ending in voiceless stops. There was a significant increase in the relative rates of VI patterns in words ending in voiced stops, compared with the reference condition rates (β = 2.215, OR = 9.16, p < 0.001). The model also found a significant effect of stop place; that is, there was a significant increase in the relative rates of VI patterns in words ending in coronal and dorsal stops, compared with the reference rates, that is, the rates of VI patterns to other patterns for words ending in labial stops (β = 3.572, OR = 35.57, p < 0.001 for labials vs. coronals; β = 2.034, OR = 7.65, p < 0.001 for labials vs. dorsals).

Figure 1:

Predicted probability of vowel insertion patterns in the regression model classified as vowel insertion (VI; dark blue), optional vowel insertion (OVI; mid blue), and no vowel insertion (NVI; light blue) depending on each linguistic factor (vowel tenseness in A, stop voicing in B, stop place in C, word size in D, and final stress in E).

The SVM model for ternary data in Table 6 (see Appendix) shows overall classification rates of 71 %, which represents well above chance classification. The ternary model of the classification matrix in Table 7 shows that the five-attribute classifier of vowel tenseness + stop voicing + stop place + word size + final stress is the most accurately classified at 78 %, while the one-attribute classifiers of stop voicing and word size are the worst at 56 %. The models for binary data, on the other hand, show overall classification rates of 78 %, which is higher than for the ternary one. Two four-attribute classifiers are the most accurately classified at 86 % for both binary C and binary V: vowel tenseness + stop voicing + stop place + final stress, and vowel tenseness + stop voicing + stop place + word size for binary C; vowel tenseness + stop voicing + stop place + final stress for binary V. The model for binary data also shows that the one-attribute classifier of word size is the worst at 62 % for binary C and 58 % for binary V, respectively.

It is evident that classifiers incorporating the vowel tenseness attribute demonstrate strong performance, achieving classification rates exceeding 77 % for the binary C model. This finding would suggest that tenseness of the vowel preceding the English final stop holds a phonological/phonetic importance in the Korean grammar. Additionally, it can be observed that increasing the number of attributes resulted in higher accuracy rates across the three models (see Table 6). The increase in accuracy percentages with the addition of more attributes is not so surprising, as including a greater number of explanatory variables naturally enhances the fit of the model in SVM analyses (Srivastava and Bhambhu 2010). Generally, more context in any multivariate analyses tends to lead to better performance of the model (Altman 1992; Cunningham and Delany 2007; Vapnik 1995). However, an exceptional case is found in the models for binary data: the five-attribute predictor (vowel tenseness + stop voicing + stop place + word size + final stress) is less accurately classified than the four-attribute predictor (vowel tenseness + stop voicing + stop place + final stress and vowel tenseness + stop voicing + stop place + word size), with 85 % versus 86 % (see Table 7). This result shows that the presence of a larger number of variables does not necessarily guarantee a more satisfactory outcome. A possible reason for this could be that each cue points toward different directions: in the model for binary V, all five attributes except word size might be pointing in the same direction so that the four-attribute classification has a stronger effect and is better accurately classified than the one containing all the possible cues.

As shown in Figure 2, which illustrates the classifiers with the top three overall classification rates for each of the three models, the most accurate classification rate of the ternary model was 7 % lower than that of the two binary models (78 % vs. 86 %). The best result (86 %) was obtained from both binary C and binary V models in the classifier of vowel tenseness + stop voicing + stop place + final stress. Thus, we can assume that this classifier has the greatest impact on vowel insertion patterns when an English word ending in a stop is adapted into Korean. Specifically, the classification results revealed a substantial effect of both vowel tenseness and stop voicing on the vowel insertion phenomenon. Classifiers including the tenseness attribute are found to consistently achieve greater accuracy than those containing the other attributes. All 16 classifications containing vowel tenseness showed high accuracy both in ternary and binary models; they were all higher than 70 % in the ternary model and 77 % in binary C model (see Table 7 in Appendix). Although the binary V model does not exhibit the exactly same pattern as the other two models, it still generally yields very high accuracy level; out of the 16 classifiers that include vowel tenseness, 12 had accuracy higher than 79 %, and four range from 70 % to 73 %.

Figure 2:

Classifiers with the top three overall classification rates in the support vector machine models of the data for ternary, binary C(onsonant), and binary V(owel).

In this study, an RF classifier was also applied on the same production data to see if similar results are obtained by both SVM and RF classifiers. We made use of the classical RF algorithm implemented in the R package randomForest (Liaw and Wiener 2002). RFs were fitted to the data to predict the production pattern with three levels (NVI, OVI, and VI), and the dependent variable was modeled as a function of linguistic factors containing vowel tenseness, stop voicing, stop place, final stress, and word size. Since the mtry parameter in the RF algorithm determines the number of predictors used in constructing each decision tree and experimenting with various mtry values is recommended to enhance the model’s performance by optimizing mtry, the train() function from the caret package was utilized to tune the mtry parameter in this study (Kuhn 2008). To optimize mtry, a ten-fold cross-validation with five repetitions was conducted and the final RF model exhibited a high classification accuracy of 80.75 % for the three different production patterns. The confusion matrix in Table 4 shows the model’s ability to distinguish the three patterns. The columns in the table represent the model’s predictions while the rows represent the observed categories in the data. Instances that are accurately classified are situated on the diagonal of the table and misclassifications are found off the diagonal. Furthermore, the fourth column presents the balanced accuracy. The model accurately classified NVI and VI patterns with an accuracy of approximately 87 %, which is remarkable for unseen data. The accuracy for the OVI pattern, however, indicates that this type was not effectively learned by the model, with an error rate of 50 %. A possible reason for the poor performance of the model with reference to predicting OVI could be that the model predominantly predicted OVI instances as VI (n = 70); the majority class appears to influence the model’s prediction in such cases.

In order to compute the predictor importance in the RF model, the varImp() function from the caret package was employed (Kuhn 2008). In this study, permutation-based variable importance was employed to assess the differences in predictive accuracy of the model. The method of estimating variable importance using RFs is different from more conventional analyses like regression in that permutation-based variable importance measures the impact of each predictor by randomly permuting its values and observing the change in the model’s performance (Breiman 2001). In other words, if a certain predictor (e.g., vowel tenseness) is connected to a specific level of the response variable (e.g., VI), randomly shuffling the values of vowel tenseness reduces the connection between VI and the predictor. This analysis directly ranks the relative importance of several linguistic factors in predicting the different patterns of vowel insertion. The estimated relative importance of predictors is shown in Figure 3, where predictors that are closer to zero are considered to have minimal contribution in predicting the dependent variable. These results show that the predictors can be categorized into three distinct groups based on their importance. Primary predictors consist of vowel tenseness and coronal stop place, where vowel tenseness was the most important predictor of dependent variable, followed by coronal stop place. Random permutation of the values for these predictors noticeably decreased the accuracy of the model’s prediction of production patterns. Moreover, stop voicing and dorsal stop place are also important predictors, although their importance is relatively lower compared to the predictors in the main group. The remaining predictors, word size and final stress, constitute the third group and are the least important in the RF model. Final stress is specifically associated with a mean decrease in accuracy that is very close to zero, indicating that it has little importance as a predictor in this model. Overall, vowel tenseness is the best predictor of production patterns in the data, whereas other predictors are less informative as predictors. These findings from RFs are consistent with the SVM results in that the vowel tenseness predictor greatly contributes to the classification accuracy in both RF and SVM models.

Figure 3:

The estimated relative importance of variables in random forests considering all predictors. Greater values of mean decrease in accuracy rates are predicted to have a more substantial impact on the model’s classification accuracy.

In this study, we trained classifiers of SVMs and RFs to get an insight into the factors that can contribute to the prediction of vowel insertion following an English word-final stop when a word is borrowed into Korean. From the SVM results, we have found that binary classification (NVI or VI) achieved more satisfying results than ternary classification (NVI, OVI, or VI), with the average accuracy of the ternary model being 7 % lower than that of the two binary models (78 % vs. 71 %; see Table 6). More importantly, vowel tenseness was found to consistently achieve greater accuracy than other predictors. Similar results were obtained by the RF model, with vowel tenseness the most important predictor of production pattern, as random permutation of the values for this predictor noticeably decreased the accuracy of the prediction of the model. Predictors other than vowel tenseness were found to be less informative, although factors such as stop voicing and stop place also held some variable importance. These findings support the need for a behavioral experiment to see if the current results can make the correct predictions with respect to the behavior of nonce items.

The contribution of the current study is that it provides methods where we can classify different adaptation patterns and determine which predictor is most influential in classification. In this study we show that using machine learning classifiers in the investigation of loanword adaptation presents several advantages and opportunities for further study. Machine learning techniques offer quantitative methods for analyzing large datasets and identifying subtle phonological patterns. These classifiers can also be trained to predict the outcomes of loanword adaptation for new entries into a language. By uncovering predictive patterns in existing loanwords, we have developed models capable of anticipating how future loanwords will be adapted. This interdisciplinary approach could potentially open up new avenues for research and provide valuable insights into the dynamics of language contact.