A study on automatic correction of English grammar errors based on deep learning

4.1 Experimental setup

The Transformer model was realized by the open-source tensor2tensor. There were eight heads in the multi-headed attention. As to the FNN, the dimension of the hidden layer was 2,048, the gelu function was used as the activation function, dropout was 0.1, and lr was 0.01. The Adam optimizer was used. The batch_size was 20, and the epoch number was 40. The maximum length of the sentence was 50. The beam size was 8. The length penalty parameter was 0.6. The training ended when the model had no effect improvement for three consecutive epochs on the development set. In CNN, the word vector dimension was 300, the size of the convolutional window was 3 × 3, the pooling window was 2 × 2, the first convolutional layer was mapped with 128 features, the second convolutional layer was mapped with 256 features, and the dimension of the hidden layer was 128. For adversarial training, the RMSprop algorithm was used [18], the initial learning rate was set as 0.0003, and the batch_size was 128. When training the discriminator, 5,000 samples were randomly sampled as positive samples, and the corresponding negative samples were created.

4.2 Experimental data set

Training sets: ① Lang-8: it is a corpus established by extracting from the social network Lang-8 and has multilingual versions, as shown in Table 1. The English version is extracted as the training set.

② CLC FCE: It includes 1,244 exam scripts written based on test papers of candidates who participated in exams of English for Speakers of Other Languages, containing original texts, labels, error comments, etc. There are 80 types of grammatical errors and 1.36 million data.

③ NUCLE: It includes about 1,400 papers written by National University of Singapore undergraduates, which have been annotated and corrected by professional English teachers. The percentage of incorrect sentences in the corpus is 42.4%, involving 28 types of grammatical errors.

Test sets: ① CoNIL-2014: It is a standard data set of GEC, which includes 1,312 sentences and 28 types of grammatical errors. The evaluation index is F _0.5.

② JFLEG: It is a common data set of GEC, which can help correct grammatical errors and produce more fluent language. It includes 747 difficult sentences. The evaluation indexes are GLEU and F _0.5.

4.3 Evaluation indicators

1. F _0.5: F refers to the weighted harmonic mean of precision and recall rate; F _0.5 refers to that the precision is twice as important as recall rate. The samples are divided according to the confusion matrix shown in Table 2.

   Then, the precision (P) of the model is

   (11) P = TP TP + FP ,

   and the recall rate (R) is

   (12) R = TP TP + FN .

   The calculation formula of F _0.5 is

   (13) F 0.5 = ( 0.5 2 + 1 ) × P × R 0.5 2 × P × R .

2. MaxMatch (M ²): In GEC, an error sentence may have multiple results after corrections. If every answer is considered as a branch of a node, then the whole process of error correction can be regarded as a connected graph, and the evaluation of the error correction result can be regarded as the evaluation of the graph, i.e., scoring the edge of the node that can reach the final correct answer. For source-side sentence:

   (14) S = { s 1 , s 2 , … , s n } ,

   let the corrected sentence output by the model be

   (15) E = { e 1 , e 2 , … , e n } ,

   and the manually labeled sentence be

   (16) G = { g 1 , g 2 , … , g n } .

   The specific calculation formulas are as follows:

   (17) P = ∑ i = 1 n ∣ e i ∩ g i ∣ ∑ i = 1 n ∣ e i ∣ ,

   (18) R = ∑ i = 1 n ∣ e i ∩ g i ∣ ∑ i = 1 n ∣ g i ∣ ,

   (19) F 0.5 = ( 0.5 2 + 1 ) × P × R 0.5 2 × P × R .

3. GLEU: In a translation task, BLEU is based on the similarity between sentences. The specific calculation formulas are as follows:

where P n is the precision rate of n-gram level words, BP is the length parameter, and c and r are the lengths of reference translation and machine translation, respectively. However, unlike machine translation, in the GEC task, an untranslated word may not be wrong, and changes can be made to the words in the source sentence; therefore, in order to evaluate GEC, BLEU is modified to GLEU, and the specific calculation formulas are as follows:

where C , R , and S refer to the candidate set, reference set, and source-end sentences, P n ′ is the accuracy rate after correction, and γ is the penalty rate, which is used to penalize incorrect answers that appear in the source sentence but not in the reference set.

4.4 Experimental results

The experimental results on CoNIL-2014 are shown in Figure 1.

Figure 1

GEC results of the model on CoNIL-2014.

It is seen from Figure 1 that the P-value, R-value, and F 0.5 of the basic Transformer model were 61.21, 30.91, and 51.18, respectively; the P-value of the GAN-combined Transformer model was 63.27, which was 2.06 larger than that of the Transformer model; the R-value of the optimized model was 34.52, which was 3.61 larger than that of the Transformer model; and the F 0.5 of the optimized model was 53.87, which was 2.69 larger than that of the Transformer model. The experimental results showed that the performance of the Transformer model on the GEC task was improved to some extent after combining GAN, which suggested that the improvement was effective.

The experimental results on JFLEG are shown in Figure 2.

Figure 2

GEC results of the model on JFLEG.

It is seen from Figure 2 that on the JFLEG data set, the GLEU value of the Transformer model was 52.96, and the GLEU value of the GAN-combined Transformer model was 61.77, which was 8.81 larger than the Transformer model, indicating that the similarity between the results obtained by the GAN-combined Transformer model and the manually annotated reference set was higher. The results suggested that the GAN-combined Transformer model was superior.

In order to further understand the application possibilities of the GAN-combined Transformer model, a student’s English essay was used as an example. The grammatical errors in the essay were automatically corrected using the model designed in this article, and the original text is as follows.

Technology is all around us, changing the way our live. The emergence of ships, trains, planes, etc. has made transportation more convenience. No matter where you want to go, as long as you find a suitable means of transportation, you can get to there quickly. On the same time, the development of science and technology has promoted the birth of computer, which has further narrowed the distant between people, allowing people to communicate to distant friends without leaving home, and can also realize online shopping, entertainment, work and so on. The development of technology have enabled us to live a better life.

The correction results of the model are shown in Table 3.

It is seen from Table 3 that the GAN-combined Transformer model corrected tense errors, pronoun errors, and singular and plural errors, but some errors were not detected, for example, in the sentence “which has further narrowed the distant between people,” the word “distant” should be corrected to “distance.” The above results suggested that the GAN-combined Transformer model had some shortcomings compared with manual annotation, which need to be improved in future research to further improve its performance in GEC.