Do writing performance and examination grading correlate in an EMI university setting?

1.1 Previous research

In our previous studies, we showed that German students at the Radboud University of Nijmegen, the Netherlands, with Dutch as their study language, systematically perform less well than Dutch domestic students (Zijlmans et al. 2016, 2020). However, not all studies observe a negative effect of studying in a non-native language. De Vos et al. (2020) collected data about grades, the number of study points (EC),^[1] and dropout rates of Dutch and German students in parallel Dutch and EMI tracks in the psychology bachelor at the same university, during their first year of study. The research aim was to determine whether study success was related to studying in one’s first (L1) or second (L2) language. The group difference in ECs obtained was not significant. Also, the dropout rates were not significant, although admittedly higher than for the Dutch students. For our present study, the relevant conclusion was that studying in the L2 can be disadvantageous when it comes to grades. However, is such a general effect visible in concrete grading processes? Are lower grades merely due to the use of an L2? Or are lower grades related to the lower L2 proficiency levels?

Vander Beken and Brysbaert (2018) focused specifically on the relationship between language and the ways of assessing content knowledge. After reading disciplinary literature in either their L1 Dutch or in L2 English, students performed at the same level on a true/false recognition test in both languages but lower on a free recall test in L2 English. This outcome suggests that students’ performance may be underestimated if they are only tested with essay-type L2 exams.

These studies on international students’ academic success focus on grades, grade point average (GPA), and European credits (EC). However, Rose et al. (2019) point out that GPA and course completion – ECs – are based on subject knowledge, but also attendance, active participation, and group work. They, therefore, consider GPA and ECs as measures too indirect to examine the relationship between language proficiency and academic performance. Additionally, their study shows that besides English language proficiency, academic skills were statistically significant predictors of students’ final grades in a course in International Business. Their data analyses also revealed that students from all language proficiency levels were able to pass the course.

In Santos’ (1988) study, native English (NE) speaking and non-native English (NNE) speaking university lecturers (178 in total) rated two English essays from NNE university students. Their content appreciation was not influenced by the “quality of the language”. The language “errors” in the NNE essays were judged as generally “not irritating” and not hindering comprehensibility. The overall conclusion was that the lecturers were willing to appreciate the content beyond the perspective of a stringent native speaker norm.

Research on the relation between English language proficiency and study results, however, often assumes a native English perspective as the norm (English as a Native Language = ENL). Given that English is used worldwide as a Lingua Franca (ELF), comprehensibility has been put forward as a more relevant criterion in evaluating English proficiency. Santos’ use of the words “deficiencies” and “irritating” is perhaps unfortunate, but he revealed academic readers focus on content and successful communication rather than on form. ELF researchers argue that ENL should not be taken as the norm when studying the use of English worldwide. Mauranen and Metsä-Ketelä (2006) designate Jenkins’ work in 2000 as “the first major description of ELF as a kind of language in its own right rather than as a deficient form of English” (Mauranen and Metsä-Ketelä 2006: 3). ELF is best defined as “an additionally acquired language system which serves as a common means of communication for speakers of different first languages” (the website of VOICE, quoted by Jenkins 2011: 928).

All in all, studies on the effect of NNE on study results show contradictory and inconclusive results. Whatever the outcomes of various studies are, however, the Englishization of higher education seems to have reached a point of no return. The aims of EMI, to develop content knowledge and academic English language proficiency simultaneously, however, are not always met.

1.2 Research questions

This study aims to contribute to a better understanding of how English functions within international academia. Rose et al. (2019) point out that studies examining the association between language proficiency and academic performance are scarce in EMI contexts. Within EMI university programs, where the need for academic and professional productive language skills has increased (Pitkänen 2018), we investigated the role of English as an Academic Lingua Franca. Academic English proficiency manifests enormous variation, and the perspectives on the communicative appropriateness of these manifestations are currently changing (Mauranen 2018b). So far, most research has been done on spoken data, and to a lesser extent on written data. Shchemeleva (2022) investigated research articles by NNE scholarly writers (she used the SciELF corpus). These writers were professional NNE users, different from student writers. Our data consists of texts written by first-year students who have not had much earlier contact with English academic literature. Their answers to an exam question were written in October. These texts represent NNE writing language skills at the start of their studies. We wanted to investigate NNE communication in an early stage as communication difficulties might negatively affect the study’s progress.

We concentrate on productive skills typical of dense short texts written in ELF by students in a time-limited examination. These texts have been graded by the lecturers of the course in question. The English proficiency level of these texts was graded by four English language experts. We address three research questions to find out more about the relationship between written NNE and content grades. The first question concerns their correlation:

1. Do the Common European Framework of Reference (CEFR) English levels of written examination answers assigned by language experts correlate with the grades assigned by content course lecturer(s)?

We chose to express language proficiency at the CEFR levels (Council of Europe 2001). Firstly, although we are aware of criticisms regarding the validity and relevance of the framework, using these levels is commonplace in Europe in university admission demands and in evaluating students’ language proficiency skills. Our choice does not mean that we support this current use of CEFR levels. Secondly, the higher CEFR levels are related to more cognitively demanding tasks (Hulstijn 2015), such as writing an academic text. Thirdly, although parts of the CEFR are written from the viewpoint that interaction takes place with native speakers (NSs), the general description that the interaction should be “without strain for either party” (Council of Europe 2001: 33) is well applicable to the interaction between interlocutors with different L1s.

Our written answers to an open examination question provide us with the data we need to answer our first research question. From an ENL perspective, there should be a significant correlation, from an ELF perspective not, assuming that all students have sufficient English skills to provide a comprehensible answer.

In addition, we investigate what features are recurrent in the ELF writings of German students in this specific academic context:

1. What are linguistic and textual features, different from standard native use, of written examination answers by German students, and do these features resemble findings in other ELF research?

We counted their occurrences (frequencies) to get an idea of their distribution, persistence, and relevance. These features are not by definition specific ELF features only, but features of the writings produced by the German students.

Thirdly, we wanted to know whether some features may disturb comprehensibility in written ELF discourse. In this context of writing answers in examinations, this would mean that the content expert cannot assess the answer positively, due to a lack of clarity, caused by inadequate ‘command’ of the language needed for the purpose. Or can the content experts accept and assess written answers diverging from ENL when these are intelligible to them (Jenkins 2011)? The role of comprehensibility is addressed in the third research question:

1. Which features may disturb comprehensibility, in terms of inadequacies, omissions, or ambiguities, and thus negatively influence the grading of a written examination question?

In the context of the “Englishization of higher education”, we need to know which minimum standard norms and practices should be applied, that is justified by the specific context and purpose (Jenkins and Leung 2017).

2.1 Data

2.2 Rating the texts

2.3 Coding procedure

We inserted the texts and the LEs’ comments in the program ATLAS.ti. We did not supply the LEs with codes or a codebook in advance, based on specific features that had appeared in former research. We wanted the LEs to mention all aspects they felt as striking. We transferred the LEs’ comments into codes. Finally, we revisited each text repeatedly via the tool ‘intercoder agreement’ in search of overlapping codes, thus determining the final set of codes. We want to point out that the same fragments regularly got different codes, as the LEs interpreted the writer’s answers differently and/or suggested different linguistic explanatory categories. We eventually distinguished five coding categories, namely (i) grammar, (ii) vocabulary, (iii) academic register, (iv) structure and coherence, and (v) content and comprehensibility. Spelling as a separate category was removed, as we used typed transcripts of the handwritten texts. We do not know which incorrect spelling was originally done by the students, or the result of a misinterpretation by the typists.

3.1 The relation between English language proficiency levels and grades

To answer our first research question we allocated a rounded mean CEFR level to each text. Eventually, we had six texts at B1, 28 texts at B2, and 12 texts at C1. Table 3 contains the cross-tabulation of CEFR levels by grades.

Table 3:

Cross-tabulation of CEFR levels by grades.

Grade	10	8	7	6	4	2
CEFR
B1 n = 6	5		1
B2 n = 28	17	5		4	1	1
C1 n = 12	10	1	1

There is no significant correlation between the CEFR levels allocated by the LEs, and the grades assigned by the content expert (r = −0.042). The correlation between grade and each LE separately ranged between −0.037 (LE3) and 0.019 (LE4). Interestingly, the correlation between the automatically calculated overall Grammarly score and the average CEFR level was moderately positive (r = 0.676, p = 0.01).

Also available to us were the ECs obtained by each student at the end of the first year of study. This gives us a broader view of the effect of language proficiency on study results, as course completions increase the number of ECs. We did not find a correlation between the CEFR levels of the texts in the first study trimester and ECs at the end of the year (r = 0.014). This is illustrated in Figure 1. If students obtain fewer than 42 ECs, they drop out after the first year. This was the case for two students at the B2 level and one student at the C1 level.

Figure 1:

Number of ECs obtained by CEFR level.

3.2 Features of ELF writing

We present the features most frequently mentioned by the LEs in this section. We will report their occurrences in the five coding categories we distinguished. Note that at times the label of ‘non-standard’ usage is a matter of individual interpretation and judgment of the LEs, as we left this for their interpretation. Table 4 contains the top 10 features.

3.2.2 Vocabulary

Codes in the category ‘vocabulary’ occurred 207 times including prepositions (40 times) and linking words (see also coherence). The code ‘choose (a) different word(s)’ occurs 96 times.

The LEs differentiate between words that they qualify as inappropriate for academic English and words that have a different meaning than the one the student probably intended (then for than). Often, the students’ word combinations are rather inventive, like the phrase languageless thinking in (3):

(3)

However, this theory has to be crityzed due to the fact, that there is empirical evidence of languageless thinking, for example thinking in pictures. (512)

One LE assigned ‘not existing word’, the three others did not even mark this phrase. It seems to be a fully comprehensible example of a “neologism that makes use of morphological productivity” (Mauranen 2017: 6). Another category is the use of full German words or anglicized German words (code ‘word from L1’ occurs 15 times), as in after this hypothesis which is a literal translation of nach dieser Hypothese in German, and therefore for dafür, where for that is meant. In (4) below, the German word Stamm is used instead of tribe. Often, students resort to descriptive writing. LEs indicate 32 times that words and phrases are vague and thus affect comprehensibility.

(4)

There was an example given with mens of a stamm in Africa which can not count more than three so if somebody asked them about a number above three they cannot tell, but is it than that they also don’t know about the numbers because they are that much influenced by there language? (675)

To obtain an overall perspective of lexical proficiency, we added up comments on ‘wrong’, vague and inappropriate academic wording per text and compared them to the assigned CEFR level, as can be seen in Figure 2. We seem to see a relation between the number of word codes and the CEFR level, in particular between levels B1 and B2 on the one hand and C1 on the other. This pattern is blurred by the outlying value in C1.

Figure 2:

The relation between the sum of perceived inappropriate academic wording, vague wording, and ‘wrong’ word codes, and the CEFR level.

3.3 Comprehensibility: an exemplary text example

We addressed the potential negative effect of comprehensibility disturbances in the third research question. Some examples were discussed in the above section. Our data contains texts graded the maximum 10 by the content expert, while LEs indicate that at least for parts of the text, comprehensibility is problematic. We selected one complete answer to demonstrate the absence of a relation between potential comprehensibility and the actual grade.

The LEs assigned a B1 and commented that, apart from spelling and grammatical divergences, the answer was hard to follow when one did not know the correct answer to the question. The content expert graded the answer with a maximum score of 10. The answer below contains the markings (gray) and CEFR evaluation of LE3.

There was one senior lecturer for the subject course who gave the lectures, compiled the exam, and supplied an answer model. Several assistant lecturers corrected the answers based on this answer model. Given the comments of LE3, the maximum grade of 10 by the content expert can only be understood when the grading is driven by ticking the box for a set of key concepts mentioned in whatever format. We did not have access to the answer model. The following elements must have been part of it as they occurred in most answers, and can also be found in the above answer:

1. The relation between language and thought according to Whorf;

2. The strong version and the word determine or synonyms;

3. The weak version and the word influence or synonyms;

4. An indicative example for each of the versions;

5. The strong version is rejected and the weak version is still current.

Our LEs noted comprehensibility problems but never to such an extent that they rejected the text. Only fragments appeared less comprehensible to them. Five students with B1 level scored a grade of 10; one student scored a grade of 7, which is still sufficient. We conclude that no serious communicative disturbances are being signaled in this type of text. Informed content interpretation by the content expert seems to outweigh actual semantic coherence and overrule non-standard grammatical and lexical features. Even our LE, not a content expert, comments to feel that the student understood the concept.

4.1 The relation between English proficiency levels and grades

In relation to our first research question, we found no correlation between proficiency and grades, a clear outcome. The current study might on the other hand be called limited in how written NNE was evaluated. Scholars have pointed out that the CEFR scales do not primarily address language skills, particularly not at the higher levels. The descriptors of level B2 and above contain academic skills and are very general when it comes to grammatical and lexical ‘mastery’ (Hulstijn 2015). Moreover, the CEFR framework takes ENL as the standard and assumes that interaction takes place with native speakers (McNamara 2012), whereas native English is unrepresentative for the use of English as a lingua franca worldwide (Jenkins and Leung 2017). In 2018, there has been a revision of the descriptors, no longer assuming the approximation of native speakers.^[5]

Our choice was nevertheless to use the CEFR framework as it is often the label with which the students enter the university. The CEFR is used worldwide standardly by many second language lecturers, test and learning materials developers, and researchers. At the same time, our study reveals the limitations of the framework. Interpretations differ when LEs float away from the ‘can-do approach’ to arguments based on grammar and lexis in assigning a certain level: “these grammatical features indicate the B1 level.” This can be seen in the differences between the raters: LE1 based her grading sometimes more on grammar, LE2 is a native English speaker who used the ENL standard for grammar, lexis, and formulations, and LE3 and LE4 looked much more at what the students can express. Note, however, that apart from their different focus, there is a high level of consensus among the experts. Also, we found a moderately positive correlation between the experts’ CEFR score and the total score by Grammarly. The latter is based on ENL spelling, grammar, and lexis. This seems to imply that language teachers link CEFR levels for a significant part at least to formal aspects of language.

4.2 Features of ELF writing

The LEs provided us with a list of potential linguistic and textual features of written ELF. Two issues stood out: (i) vocabulary/academic register, and (ii) grammar. Although grammatical accuracy is not a central issue in the ELF literature and there is a general tendency to focus on performance and pragmatics, we are also interested in which grammatical divergences from the ENL norm might lead to incomprehension (if at all) on the part of the recipient of the message.

4.3 Communicative disturbances

The main form feature that might have caused opacity in our data in the eyes of our LEs is sentence length in combination with non-standard word order. The problem of long sentences with unclear meaning occurs at all proficiency levels in our data. However, sentence length in itself does not necessarily obscure meaning. The LEs appreciate complex sentences with effective use of linking as evidence of advanced ELF proficiency and as adding to comprehensibility. This idea is in line with the findings of Tiryakioglu et al. (2019). Their results indicate that there is a relationship between the non-native proficiency level and the composing processes in the L2. The formulation process is performed more efficiently when a learner/user is more proficient in that language.

We studied written communication where there is no possibility to resolve misunderstandings and communicative breakdowns. The evaluation by intended readers is crucial in signaling the occurrence of disturbances in communication in the sample of texts. The grades assigned by the content experts do not signal such disturbances in relation to NNE features. There appears to be a sufficient level of mutual comprehensibility. We conclude that the content experts, having the model answer in mind, are not bothered by formulations in English that were considered less adequate by the LEs, while understanding these subject-specific texts might be more difficult for non-content experts. The main conclusion of this study is that evaluators let content prevail over non-native language forms.

One may argue that the language background of the content expert is a significant factor in our study. The content experts are typically ELF speakers. Björkman (2013) observed many of the same NNE features in the language of lecturers as in the language of students (Björkman 2013). In our study, the content experts roughly share the linguistic background of their students, as German and Dutch are closely related languages. This might explain why the content expert is not distracted in reading ELF.

In an earlier publication, we reported that university lecturers – the content experts in the present study – said to read ‘cooperatively’ when reading non-standard Dutch by German students. Mauranen (2017) used the term “augmented cooperativeness between participants” (Mauranen 2017: 8). This applies particularly in the setting of an examination. Often, answers lack cohesion, contain lexical gaps, and the formulation lacks accuracy (see example above), and still, the assessor was willing to uncover the student’s intent in a supportive way. In spoken language, the conversational partners frequently use pragmatic strategies to achieve communicative effectiveness and to prevent or repair disturbances (Björkman 2013). In writing, the recipient is not present, so it is in the writer’s interest to be as clear and explicit as possible, especially in the setting of an examination. Nevertheless, even when the student does not succeed in the eyes of the LEs, the content expert as an assessor is so cooperative as to interpret the student’s message as informative as possible. University lecturers unilaterally accept, if not condone, instances of English that diverge from ENL but that are nevertheless intelligible to them (Jenkins 2011). By ticking the box for words and phrases representing the key concepts, the assessor accumulates evidence for a student’s understanding of Whorf’s hypothesis.