English grammar intelligent error correction technology based on the n-gram language model

3.1 Construction of error correction model based on n-gram

The situation of English grammar errors is complicated, and the intelligent error correction technology designed by the research is mainly aimed at article errors, preposition errors, noun errors, and verb errors in English grammar [16]. If n words form a sentence S , then the string of S can be expressed as S = w 1 w 2 w 3 … w n − 1 w n , and the probability of string S is P ( S )

In equation (1), the probability of occurrence of word w i is jointly determined by word w 1 w 2 w 3 … w i − 1 before word w i , so these words are the preamble of word i [17]. When the vocabulary set size is L and the length of the preamble is i − 1 , the word at i will have L i − 1 different preambles, which is too large to calculate. Map w 1 w 2 w 3 … w i − 1 above to the equivalence class E ( w 1 w 2 w 3 … w i − 1 ) with certain rules, so that P ( w 1 w 2 w 3 ⋯ w i − 1 ) = P [ w i ∣ ( w 1 w 2 w 3 … w i − 1 ) ] reduces the number of free parameters.

Partition equivalence class method: Two preambles w i − n + 2 … w i − 1 w i , w k − n + 2 … w k − 1 w k can be mapped to the same equivalence class if and only if the nearest n − 1 word is the same [18]. Then, E ( w 1 w 2 w 3 … w i − n + 2 … w i − 1 w i ) = E ( v 1 v 2 v 3 … v k − n + 2 … v k − 1 v k ) , if and only if ( w i − n + 2 … w i − 1 w i ) = ( v k − n + 2 … v k − 1 v k ) . When a language model meets the above conditions, it is called an n-gram language model [19]. The related phrases before and after a given word can be obtained by window movement in a sentence, and its mathematical description is

In equation (2), i indicates the position of a given word in a sentence; k is the window size; w refers to the word in position i .

Table 1 provides examples of moving window values. n-gram fragments have different lengths when window sizes are different. An error candidate set is a set composed of two parts, such as an error in a sentence and the corresponding candidate modification answer [20]. The zero probability problem occurring in n-gram is solved by linear interpolation smoothing technique; that is, the higher-order model is combined with the lower-order model in a linear way, and the higher-order n-gram model is linearly interpolated by the lower-order n-gram model, so as to estimate the probability of the higher-order model [21]

In equation (3), ∑ i λ i = 1 and λ i are calculated by the expectation maximization (EM) algorithm. First, training data and hold-out data are determined, the initial language model is built from the training data, and the initial λ i is defined. The EM algorithm is used for iterative optimization λ i to ensure that the probability of Held out data is maximized [22]

Equation (4) is a probability calculation equation for Held out data. The n-gram combination model (Combine N-gram, Cn-gram) of (A n = 1 , 2 , 3 , 4 , 5 ) is obtained by the linear interpolation smoothing algorithm. The probability calculation equation of this model is shown in equation (5)

λ N in equation (5) corresponds to λ i in equation (3), both of which are calculated by the EM algorithm. The moving window is combined with the error candidate set to design an English syntax error correction technique based on the error candidate set. The results of error correction technology are evaluated by precision, recall, and F1 values

N correct in equations (6)–(8) refers to the number of grammatical corrections that are correct; N predicted refers to the number of grammatical correction errors; and N target refers to the number of errors originally present in the grammar. The syntactic error correction model based on the Cn-gram model can effectively locate the possible errors in sentences and has shown good performance for processing simple sentences. However, when dealing with long sentences, the model’s precision, recall rate, and F1 value all decrease. Therefore, syntactic analysis is introduced in this study to make the model better able to deal with errors in English long sentences.

3.2 Construction of grammar error correction model based on Cn-gram and syntactic analysis

In this section, a Parsing and Cn-gram Grammatical Error Correction (PCGEC) is proposed, which combines Cn-gram and syntactic analysis. In essence, syntactic structure is a process of interrelation between words [23]. A dependency ties two words together, one being a core word and the other a modifier. Dependency can be used to describe the grammatical relationship between two words and can be further subdivided into various types [24]. The entered sentence corresponds to a dependency graph G = ( V , A ) , which is a directed multi-graph. Wherein, V = { 0 , 1 , … , n } , each node i corresponds to an input word w 1 . Dependency tree T = ( V , A ) is a dependency graph, whose specific expression is

In equation (9), A , B , C represents a dependency arc from w A to w B ; w C represents core words; w B represents a modifier; C is the type of relationship of the arc of dependence; and L is a set of dependency arc relationship types. The following assumptions exist in the dependency tree corresponding to the sentence: (1) The interaction and correlation between dependency arcs only occur in some specific structures (sub-trees) [25]. (2) The other arcs of dependence are independent of each other. Then, the fractional value of a dependency tree can be decomposed into the sum of the fractional values of several sub-trees, and the calculation equation is as follows:

In equation (10), x represents the sentence; p represents a subtree permitted by an independent hypothesis; d represents the feature vector of a sentence; p contains one or more arcs of dependency in d ; w is the characteristic weight vector; f ( x , d ) is the aggregated syntactic feature vector corresponding to ( x , d ) ; and Score ( x , p ) denotes the score value contributed by subtree p . The syntactic analysis model proposed in this study is introduced into the three-seed tree structure, which consists of one dependency arc, two sibling dependency arcs adjacent in the same direction, and two dependency arcs of the grandparent relationship [26]. The calculation equation of the dependency tree is

In equation (11), Score dep , Score sib , and Score grd , respectively, represent the scores corresponding to the three seed trees, and their specific expressions are

In equation (12), f dep , f sib , and f grd , respectively, represent the eigenvectors corresponding to the three seed trees. W dep , W sib , and W grd represent corresponding feature weight vectors, respectively. The flow of the syntax error correction algorithm based on syntax analysis is shown in Figure 1.

Figure 1

Syntax error correction algorithm flow based on syntax analysis.

In Figure 1, the first step is to present the set of sentences to be corrected and perform word segmentation on the sentences, that is, to decompose an input text stream into words, phrases, symbols, or some meaningful elements. Afterward, the data obtained from word segmentation are annotated with part of speech, which determines whether each word is a noun, verb, adjective, or other parts of speech. Syntactic analysis mainly refers to performing dependency syntactic analysis on the data annotated with part of speech to obtain the dependency tree corresponding to the sentence. Afterward, based on the instances in the incorrect candidate set C, first-order and second-order sub-trees are extracted from the complete dependency tree. The frequency of sub-trees is calculated based on the tree library and converted into scores. The error candidate set is taken with the highest score in the sub-tree, the error item is replaced, and the sentence is outputted after the completion of error correction.

The syntax error correction model based on parsing has the following shortcomings: (1) It relies too much on a dependency tree. Currently, the dependency tree database is not enough, and the workload required to establish the dependency tree database is large, and the corresponding relationship of each dependency tree also has different differences. (2) The local error rate and recall rate in sentences are not too high. (3) The precision of syntactic analysis has a great influence on error correction performance. Therefore, in this study, the n-gram algorithm and syntactic analysis model are integrated to improve the error correction performance of the overall model. This study focuses on the automatic correction of English compound sentence grammar. Currently, there has been a lot of research on compound sentences, such as the hierarchical study of long and difficult sentences and the study of related words. When building a compound sentence model, we should give full play to the advantages of the Cn-gram word model and avoid the shortcomings of the Cn-gram word model for long-distance constraints. This article adopts a new way of thinking, that is, by splitting the compound sentences, analyzing the syntax, and then combining the results. First, the sub-sentences of compound sentences are classified according to semantic relations, and the related words and guiding words are used as the basis of classification. On this basis, Cn-gram is used to model each segment independently. Finally, the model is combined, that is, the final complex sentence modeling result. Among them, the connective words and guiding words in the compound sentences serve as the link to connect each clause, so that they can complete the Cn-gram model independently. The probability calculation process of the combined model is

In equation (13), P 1 represents the probability that W i ( w ) occurs after knowing the previous word of the specified word W i ( w ) ; P 2 represents the probability that a word before W l ( w ) occurs when the word after W l ( w ) is known; and P 3 represents the probability of W l ( w ) appearing in the middle of a given word W i ( w ) with one word before and one word after it. Assuming that the words before and after the candidate words have the same influence on the candidate words, the score calculation equation is

In equation (14), λ , μ are the values determined for a large number of training through the training set. The probability calculation equation of the entire complex sentence is

In equation (15), S i represents a clause or clause and α i denotes the weight representing the probability of each clause is trained in the training set. In a pun complex sentence, the conditional possibility of two connective words is α i . For single related words and clauses, after synthesizing the components in each clause, select α i as the conditional probability of the sentence components of the guiding words or related words and clauses. The overall operation flow of the error correction algorithm based on PCGEC is shown in Figure 2.

Figure 2

Overall operation flow of error correction algorithm based on PCGEC.

In Figure 2, first, the corrected sentences are segmented based on the set of sentences to be corrected, and the data obtained from the segmentation are annotated with part of speech. Afterward, dependency syntactic analysis is performed on the data annotated with part of speech to obtain the dependency tree corresponding to the sentence. Based on instances in the incorrect candidate set, first-order and second-order sub-trees are extracted from a complete dependency tree. Sub-tree frequencies are calculated based on the tree library and converted into scores. The error item is replaced with the instance in the error candidate set that corresponds to the sub-tree with the highest score. Errors in the N-gram error correction process are corrected in the order of nouns, articles, and prepositions, and a voting strategy is used to rate the N-gram. The probability of the corrected sentence is calculated in the N-gram model, and the sentence with a higher probability is selected to output the corrected sentence.

4.1 Performance verification of syntax error correction algorithm based on Cn-gram

The training data and test data as shown in Table 2 are studied and selected, and the practical application effect of the designed syntax error correction technology is verified by the Windows 7 operating system. In Table 2, error types include article error, preposition error, noun error, subject–verb agreement error, and verb form error. According to the length of sentences in the test data, it is divided into two parts with the same number, the Long part and the Short part.

In the n-gram model, when n is 2, the word w i in the position i is only affected by the previous historical word w i − 1 , which is denoted as the Bi-gram model. In the n-gram model, when n is 3, the word w i in the position i is only affected by the first two historical words w i − 1 and w i − 2 , which is recorded as the Tri-gram model. The comparison experiment of the Bi-gram model, Tri-gram model, and n-gram model-based error correction algorithm Cn-gram is set up.

In Figure 3, the correct rate of Bi-gram and Tri-gram is 0.2935 and 0.2899, respectively, both of which are lower than that of Cn-gram (0.3011). The recall of Bi-gram and Tri-gram is 0.4342 and 0.4401, both lower than Cn-gram (0.4832). The F1 value of Cn-gram is 0.3710, 2.08% higher than that of Bi-gram and 2.15% higher than that of Tri-gram. Figure 3(b) shows that in the correction of prepositions in English grammar, the correct rate of Bi-gram is 0.0482, and the correct rate of Tri-gram is 0.0476, both lower than that of Cn-gram (0.0567). The recall of Bi-gram is 0.1690, slightly higher than Cn-gram (0.1678); The recall of Tri-gram is 0.1589. The F1 value of Cn-gram is 0.0750, which is 0.97% higher than Bi-gram and 1.14% higher than Tri-gram. To sum up, Cn-gram has good error correction performance for articles, that is, the Cn-gram model is suitable for the local description of sentences.

Figure 3

A comparative study on the correction results of (a) article errors and (b) preposition errors.

From Figure 4, the correct rate of Cn-gram is 0.2597, higher than that of Bi-gram (0.2409) and Tri-gram (0.2403), respectively, increasing by 1.88 and 1.94%. The recall of Cn-gram is 0.5088, higher than Bi-gram (0.4538). The recall of Tri-gram is 0.4608, which increased by 5.5 and 4.8%, respectively. The F1 value of Cn-gram is 0.3302, higher than Bi-gram (0.3147) and Tri-gram (0.3158), respectively, higher by 1.55 and 1.44%. Figure 4(b) shows that in verb error correction of English grammar, the correct rate of Bi-gram is 0.1493, and the correct rate of Tri-gram is 0.1387, both lower than the correct rate of Cn-gram is 0.1602. The recall of Bi-gram and Tri-gram is 0.2701 and 0.2681, both lower than Cn-gram (0.2795). The F1 value of Cn-gram is 0.1981, which is 0.58% higher than Bi-gram and 1.73% higher than Tri-gram. The above results show that the Cn-gram algorithm has better error correction performance for nouns than for verbs. Short test set and Long test set are processed by the Cn-gram model, and the influence of sentence length on the error correction performance of the Cn-gram model is compared. The specific results are shown in Figure 5.

Figure 4

A comparison of the results of correcting (a) noun errors and (b) verb errors.

Figure 5

A comparison of the (a) correction results of article errors and (b) preposition errors under different sentence lengths.

From Figure 5, when correcting English grammar articles in the test set with a Long sentence length (Long), the precision rate of the Cn-gram model is 0.2697, the recall rate is 0.4742, and the F1 value is 0.3438. The precision of the Cn-gram model is 0.3111, the recall rate is 0.5032, and the F1 value is 0.3910 when correcting the articles of English grammar in a Short test set with short sentence length. Figure 5(b) shows that when correcting prepositions of English grammar in the Long test set, the precision rate of the Cn-gram model is 0.0472, the recall rate is 0.1590, and the F1 value is 0.0730. When correcting prepositions of English grammar in the Short test set, the precision rate of the Cn-gram model is 0.0574, the recall rate is 0.1679, and the F1 value is 0.0852. These results show that the Cn-gram model has better performance in article and preposition correction for English with short sentence length.

Figure 6 shows that when correcting nouns in English grammar in the Long test set, the precision rate of the Cn-gram model is 0.2315, the recall rate is 0.4533, and the F1 value is 0.3064. When correcting nouns in English grammar in the Short test set, the precision rate of the Cn-gram model is 0.2757, the recall rate is 0.5268, and the F1 value is 0.3401. It can be seen that in noun error correction, the precision rate of the Short test set is 4.42% higher, the recall rate is 7.35% higher, and the F1 value is 3.37% higher than the results of the Long test set. Figure 6(b) shows that when correcting verbs of English grammar in the Long test set, the precision rate of the Cn-gram model is 0.1513, the recall rate is 0.2201, and the F1 value is 0.1798. When correcting English grammar verbs in the Short test set, the precision rate of the Cn-gram model is 0.1599, the recall rate is 0.2595, and the F1 value is 0.1985. It can be seen that in verb error correction, the precision of Short test set is 0.86% higher, the recall rate is 3.94% higher, and the F1 value is 1.87% higher than the results of Long test set.

Figure 6

A comparison of the correction results of (a) noun errors and (b) verb errors under different sentence lengths.

4.2 Performance verification of the syntax error correction model based on the PCGEC model

Next, the performance of the syntactic error correction model based on the PCGEC model is verified. The experimental environment and training data sets used in this study are the same as those used in the previous section. In Figure 7, for article errors, PCGEC’s precision, recall rate, and F1 value are higher than Cn-gram and syntactic analysis model (SAM). The precision of PCGEC is 2.33 and 5.1% higher than that of Cn-gram and SAM, respectively. The recall rate is higher by 0.69 and 3.73%; F1 values are higher by 1.29 and 6.88%. For preposition errors, PCGEC is 2.07 and 3.27% more accurate than Cn-gram and SAM. The recall rate is higher by 1.68 and 0.02%, respectively. F1 values are 2.11 and 3.33% higher.

Figure 7

A comparative study on the correction results of (a) article errors and (b) preposition errors.

In Figure 8, for noun errors, PCGEC has higher precision, recall rate, and F1 values than Cn-gram and SAM. The precision of PCGEC is 2.17 and 4.58% higher than that of Cn-gram and SAM, respectively. The recall rate is higher by 0.21 and 12.03%, respectively. F1 values are 6.74 and 7.31% higher, respectively. For verb errors, PCGEC’s precision, recall rate, and F1 value are also higher than Cn-gram and SAM. The precision of PCGEC is 11.07 and 4.03% higher than that of Cn-gram and SAM, respectively. The recall rate is 2.18 and 6.13% higher, respectively. F1 values are 7.73 and 5.07% higher, respectively.

Figure 8

Comparison of the results of correcting (a) noun errors and (b) verb errors.

Figure 9 shows the comparison results of precision, recall rate, and F1 values of the three models on Short and Long data sets. In the Short data set, the precision of PCGEC is 16.11 and 5.11% higher than that of Cn-gram and SAM for subject-verb agreement errors. Recall rates are 8.26 and 7.19% higher; F1 values are 13.15 and 6.15% higher. On the Long data set, the precision of PCGEC is 21.15 and 5.05% higher than that of Cn-gram and SAM for subject–verb agreement errors, respectively. The recall rate is higher by 13.15 and 5.79%; F1 values are higher by 15.14 and 5.31%. Therefore, PCGEC performs better than Cn-gram and SAM. Among them, the difference in subject–predicate agreement is obvious. The error-correcting effect of nouns and articles is very similar. The second is the verb and subject–verb agreement, and the last is the preposition. There may be the following reasons: First, there is little difference between articles and nouns, and errors in articles are mainly manifested as lexical errors, while errors in nouns are mainly manifested as syntactic errors. Second, the form of the verb changes with the context, mainly due to syntactic errors. Third, subject–verb agreement error is a kind of grammatical error. Fourth, there are many types of prepositions, and the preposition phrases are very complicated.

Figure 9

Comparison results of precision, recall rate, and F1 values of the three models on Short and Long data sets. (a) The precision of three models on the Short dataset. (b) The recall of three models on the Short dataset. (c) The F1 of three models on the Short dataset. (d) The precision of three models on the Long dataset. (e) The recall of three models on the Long dataset. (f) The F1 of three models on the Long dataset.

Figure 10 shows the comparison results of precision, recall rate, and F1 value of the three methods on the complete test set. Among them, the precision of PCGEC is 19.10 and 5.41% higher than that of Cn-gram and SAM for subject–verb agreement errors, respectively. The recall rate was 9.55 and 10.77% higher, respectively; F1 values were higher by 12.65 and 1059%, respectively. Through the above experimental analysis, the following conclusions can be drawn: (1) The Cn-gram model is very powerful for the local description of sentences, and it is very effective for local sentence errors (lexical errors), but it is not effective for syntactic errors. (2) The SAM method can analyze the structure of the sentence and the relationship between various elements in the sentence, so it has a significant performance in grammar errors, but is weak in vocabulary. (3) The combination of these two methods can effectively improve the correction effect of vocabulary and grammar errors meanwhile.

Figure 10

Comparison results of (a) precision, (b) recall rate, and (c) F1 value of the three methods on the complete test set.

This study used word move’s distance (WMD) and improved move’s distance (IMD) to measure the measurement of perturbed samples. The larger the WMD distance, the smaller the similarity. On the contrary, the deviation in word meaning is relatively small. IMD mainly considers the distance of movement between pinyin and determines the degree of semantic deviation. Figure 11 shows the line graph of the test results. When the sample size reaches 2,000, WMD is used to measure the generated sample size, and the obtained sample sizes are all between 0 and 0.2, while other methods are all between 0.4 and 0.6. When calculating IMD offset, the proposed method also has better performance than other methods.

Figure 11

Generated adversarial sample test results. (a) WMD distribution of adversarial sample size generated by different methods. (b) IMD distribution of adversarial sample size generated by different methods.