Tackling the elephant in the language classroom: introducing machine translation literacy in a Swiss language centre

2.1 Who is talking about MT?

It is interesting to briefly consider the culture of omertà surrounding MT in Higher Education. It could be linked to negative perceptions of MT, mainly that MT use is akin to cheating. As numerous researchers have mentioned, this is how many language teachers have felt about MT for decades (see “MT as Academic Dishonesty” in Jolley and Maimone 2022: 28). At the same time, however, language teachers now seem to be spearheading the development of MT literacy in many national contexts (Delorme Benites et al. 2021; Hellmich 2021), and many teachers have now gained experience in incorporating MT in the classroom (for recent examples see Vinall and Hellmich 2022).

If we look at the data gathered from all Swiss universities in 2021 and 2022 by the DigLit project,^[2] language students are more likely than regular students and other users (mainly staff) to have already had instruction in MT. More than half of the language teachers who took part in the survey responded that they usually give their students some information about MT (“how it works, its potential and its risks”) and, for about 11 %, this was “an explicit part of [their] teaching” (DigLit survey 2021–2022). In contrast, only a quarter of university teaching staff in other subjects presented any information about MT in their classes. Nevertheless, none of the language teachers had received any explicit instruction on using MT: 5.5 % of the respondents indicated that they did not remember but 94.5 % answered “no” (DigLit survey 2021–2022). It would thus be interesting to discover what these teachers show their students. Moreover, the question “How confident are you that you understand how MT systems work?” generated very diverse results. Language teachers had to rate their confidence from 0 (not confident at all) to 100 (perfectly confident) and the median fell at exactly 50 (see Figure 1).

Figure 1:

Answers to the question “How confident are you that you understand how MT systems works?”: Lowest values: 0,0,0, 1,2; highest values 85, 85, 90, 98, 100; median = 50 (DigLit survey 2021–2022; language teachers n = 50).

On the basis of these results, the DigLit project has been offering MT literacy training to Swiss university language teachers (four sessions in spring-autumn 2022; more to follow in 2023). One of the main observations made by the workshop coordinators was that the participants had a strong need to talk about MT and share their experiences. Candid discussions repeatedly broke out, leading the coordinators to recognise that the participants needed this space to speak about these issues with their colleagues. For some, this was the first time they had done so.

Both our data and our experience thus show that even if there is some discussion and information exchanged about MT at Swiss universities, it is not enough. This is why we decided to implement a mandatory twenty-minute session on MT for all students attending language classes at UniNE in spring 2022. We based the content of our short information session on the extensive work done to date on MT literacy.

2.2 Machine translation literacy

The term ‘machine translation literacy’ was introduced by Bowker and Buitrago Ciro (2019) to describe the core skills needed by lay users of MT systems. These include the ability to comprehend the basics of how MT systems process texts, appreciate the wider implications associated with the use of MT, create or modify a text so that it can be translated more easily by an MT system (‘pre-editing’) and modify the output of an MT system to improve its accuracy and readability (‘post-editing’) (Bowker and Buitrago Ciro 2019: 88). Similar to the wider field of digital literacy, MT literacy therefore relates to cognitive skills, rather than techno-procedural ones (Bowker 2020). As Bowker states: “Using machine translation is easy; using it critically requires some thought” (2020: 28). To this end, MT literacy is about developing the critical thinking skills needed to assess whether, when and why to use this technology and how to interact with it (Bowker 2020).

In the context of language learning, MT literacy poses challenges for teachers and learners alike. For language teachers, MT literacy means recognising that NMT systems prioritise fluency (Loock and Léchauguette 2021), which renders obsolete the bad model approach of using MT to raise awareness of correct grammar and style through error identification (Yamada 2019a). Moreover, NMT is now capable of producing output that corresponds to the levels pre-A1 to B2 descriptors for overall written production employed by the Common European Framework of Reference for Languages (CEFR) (Delorme Benites and Lehr 2022). This calls for new approaches to language instruction, assignments and evaluation, particularly in light of the fact that students use MT even when explicitly asked not to (for a discussion, see O’Neil 2019).

For language learners, MT literacy involves developing an awareness of the strengths and limitations that result from how MT systems work. For example, although NMT output is highly fluent, it continues to exhibit errors of accuracy, including mistranslations and incorrect terminology (Yamada 2019b). However, the quality of NMT output makes it very difficult for students to identify and correct MT errors (Loock and Léchauguette 2021). In addition, while studies have shown that language learners may be positively primed by syntactic constructs in MT output (Resende and Way 2021), lack of lexical diversity in MT output due to algorithmic biases may result in ‘machine translationese’, which Vanmassenhove et al. (2021: 2203) describe as an “artificially impoverished language”. This in turn could have a negative impact on L2 production.

In light of the above, it is imperative that language learners develop an enhanced MT literacy so they can critically exploit the strengths of MT as a language learning tool, be aware of its limitations, and protect themselves from the possible negative consequences of its use. The question our study seeks to answer is what is the minimal instruction needed to begin fostering MT literacy.

3.1 The UniNE language centre (LC) and the ILCF

Like most universities, UniNE – a French-speaking university in Neuchâtel, Switzerland – offers its students extracurricular language classes to foster their communicative skills in foreign languages or to improve their French. Among other programmes, the UniNE LC holds classes in German and English open to all UniNE students and staff. In spring 2022, the six English courses (from B1 to C1 according to the CEFR) and six German courses (A2 to C1) hosted a total of 138 students.

At UniNE, French as a foreign language classes are managed by the Institut de langue et civilisation françaises (ILCF) at the Faculty of Arts and Social Sciences. We collaborated with the teachers of the ‘Français-midi’ classes, which consist of five different levels of French classes intended for non-native and exchange students (A1-C1 level). In spring 2022, 99 students attended Français-midi.

The MT literacy sessions were therefore held in 17 classes, with a cohort of approximately 237 students who participated in the action research project.

3.2 Interventions

We wanted to create the shortest intervention possible with the maximum effect. We thus planned our presentations carefully, considering several important factors: content, format, timing, presenters and the overall information network on MT at UniNE.

3.3 Evaluation

In order to determine whether the interventions had the desired effect, we devised a comparative framework. The spring 2022 cohort was asked to fill in an online form with approximately 45 multiple-choice and open questions. Some of the questions were new and designed for this task (e.g. “Has the training changed the way you use MT?”), but most of the questions mirrored those from the 2021–2022 DigLit survey. This allowed us to contrast the cohort’s answers to a control group.

The students were asked to complete the questionnaire, which was implemented on Qualtrics^[4] in English and French, on their computer or cell phone at the end of a class in week 13 or 14 of spring semester 2022. As mentioned earlier, by the end of the semester, a few students usually drop out of extracurricular classes due to exam stress and preparation load. Together with some students who were unwell and other students who had to leave early, this explains why less than half the cohort completed the questionnaire (105 responses out of a total of 237 students).

4.1 Change in the way students use MT

One of the most important questions in our survey was about the students’ perception of the training: had it helped to change the way they use MT? Only a third of the respondents answered in the affirmative and slightly less than half said that nothing had changed (see Figure 2).

Figure 2:

Answers to the question “Has the training changed the way you use MT?” (our data).

At first glance, this was very disappointing, especially as the discussions during the interventions had been lively and the students had expressed interest in and amazement at some of the information shared. However, as Sections 4.3 and 4.4 show, compared to our control group, the students in the cohort demonstrated a greater MT literacy, namely in understanding how to use MT more effectively and some awareness of its risks. We have no way of knowing whether they had this knowledge prior to the interventions, but, as Section 4.2 shows, this is not likely given that they had never received any instruction in MT previously.

Moreover, when looking at the follow-up open question (“If you chose yes, please, tell us how”), the answers show that some students understood some key points: (1) never to look up single words (“Considerer [sic] davantage le contexte”, “Je traduis des phrases complètes avec un peu de contexte”, “Mettre plutôt des phrases complètes”, “Je n’utilise que des phrases” [“I take more consideration of context”, “I translate full sentences with a bit of context”, “put full sentences instead”, “I only use sentences]); (2) better awareness of data protection (“du point de vue de la protection des données”, “Je fais plus attention à la nature des textes que je traduis” [“Regarding data protection”, “I am more careful at the type of texts that I translate”]); (3) applying pre-editing tips (“De manière plus simplifiée ou alors éviter les expressions qui ne sont pas traductible [sic] littéralement” [“In a more simplified way or to avoid expressions that cannot be translated literally”]); (4) certain functionalities of the tool, such as the option in DeepL to right click on a word to display additional translation options (“Plus attentive aux possibilités de traduction” [“I am more careful of translation possibilities”]). These were directly linked to our main goal (see Section 3.2.1) and demonstrates that some information was retained and applied by the 10 % of students who answered this open question (n = 13/total = 105).

4.2 Training or guidelines about MT

We wanted to see if any of the students had already “had any explicit instruction on using MT” and if they considered our short session enough to say they had received some training. This did not seem to be the case as 55 % of respondents considered they had never received any explicit instruction on using MT (see Figure 3). Only 16 % said they had and, when answering the follow-up question (n = 11/total = 105), they all mentioned our interventions.

Figure 3:

Answers to the question “Have you ever had any explicit instruction on using MT?” (our data).

This was surprising, as the main goal of our interventions was to give “explicit instruction” to students on using MT. However, this result started to make sense when we looked at the responses to the question about whether their institution (in our case, UniNE) had “guidelines about the use of MT?”. Nearly half the cohort (46 %) answered that their university had guidelines on using MT and only 22 % did not know. As shown in Figure 4, these results stood in complete contrast to the control group.

Figure 4:

Answers to the question “Does your institution have guidelines about the use of MT?” (our data; DigLit survey 2021–2022, all users).

These results were particularly unexpected, as UniNE has no guidelines on the use of MT. This led us to conclude that some of the students misunderstood our interventions. We intended them as instructive and a first step towards MT literacy, but it would appear that nearly half the students considered them as guidelines on the use of MT. This could have been reinforced by the choice of presenter: Sara Cotelli Kureth, who is head of the UniNE LC, gave the majority of interventions, and it is possible that her position of authority in the LC^[5] led the students to consider what was said as actual LC guidelines. In the next phase of our project aimed at implementing MT literacy at the UniNE LC, the task of delivering the presentations will be handed over to the language teachers to avoid such misunderstandings. Asking the teachers to take on this role will also allow them to embed this theoretical knowledge into more hands-on tasks that foster MT literacy for all students and reinforce it by using MT in the classroom regularly. The DigLit project has thus started to create teaching material based on our short presentations that language teachers at UniNE will be asked to use as early as spring semester 2023.

4.3 Use of MT as a dictionary

When we looked at the question on the use of MT as a bilingual dictionary, the students appear to have understood this key point quite well. Figure 5 compares our data with the DigLit survey’s responses to the question: ‘How/Why do you use MT?’.

Figure 5:

Answers to the question “How/Why do you use MT?” (our data; DigLit survey 2021–2022, all users).

Only 20 % of our cohort indicated that they still use MT to translate individual words or phrases, a percentage that is significantly lower than what was found in the DigLit survey (58 %). While this initially seemed very encouraging, the results in fact contradict the responses to several other questions. In point of fact, when asked at the end of the survey to give an example of how they use MT to help with their language learning, some respondents provided responses that indicate that they still appear to think at a word level and use MT for single word searches: “Traduire un passage obscur lors d’une lecture ou un mot difficile en visionnant un film en VO” [“Translate a difficult passage when reading or a difficult word when watching a film in the original language], “Traduire des termes techniques ou inconnus dans un texte” [“Translate technical or unknown words in a text”] (our emphasis).

In addition, we asked the students which tool they use to perform certain tasks: MT or a bilingual dictionary (see Figure 6). This question, which was not in the DigLit survey, was based on responses to a survey conducted by O’Neill (2019).

Figure 6:

Comparison between our data and O’Neill’s data (2019) with regard to which tool students use to perform certain tasks.

Our results show that 43 % of our cohort use MT to look up individual words, which directly contradicts the findings in Figure 5 above. However, we suggest that this second figure cannot be accepted at face value. When analysing data from our DigLit survey, it became apparent that MT users have an inadequate understanding of what an MT system is. When asked to provide the name of the MT system that they use, the students named a large number of online bilingual dictionaries and corpora (Leo.org, wordreference, Linguee, etc.) (Cotelli Kureth et al. to be published). This unfortunately casts doubt on the reliability of Figure 6, as it could be unclear to students what the difference between MT systems and online dictionaries is, even if this was very briefly mentioned in the interventions.

It is therefore important in future presentations and/or tasks to introduce a substantial focus on tools for language learning. Fostering MT literacy alone is not enough, and language teachers should be aware that their students may have a very shallow understanding of the difference between the various tools, which may result from the way different companies market their products. DeepL and GoogleTranslate, for example, allow users to make lists of searches that some students actually use to learn vocabulary (Bin Damash 2020), and Linguee, a bilingual automatically aligned corpus, presents itself as a dictionary. It is thus very important for teachers to foster digital tool literacy, which encompasses MT literacy, to allow students to adopt the right tool for the correct type of search.

4.4 Pre-editing

Another focus of the interventions was to give information on MT-friendly writing to induce users to pre-edit rather than only post-edit. This seems to have been well understood by the cohort as 84 % mention they now modify a text before putting it in the MT system (see Figure 7). Only 23 % of the control group had done so (DigLit survey 2021–2022).

Figure 7:

Answers to the question “Do you ever make modifications to the original text before running it through the MT system?” (our data; DigLit survey 2021–2022, all users).

The responses to the following two questions (a multiple-choice question to check how often users changed the input and an open-ended question to see what type of content they changed) would indicate that this part of the training had worked well. The first follow-up question shows that half of our cohort often changed the output and about a third rarely did so (see Figure 8). Still, a significant number of users understood that they could make changes to the input when they needed to.

Figure 8:

Answers to the follow up question, “I modify the original text: always, often, rarely” (our data).

For the second follow-up question, users mention paraphrasing when they encounter problems, by saying things in their own words (“dans [leurs] propres mots”) or by adding context (“je rajoute des phrases pour contextualiser si j’ai l’impression que les termes employés ne correspondent pas à la situation recherchée” [“I add more sentences to give more context when I feel that the words provided do not fit the proper context”]).

These results indicate that a short presentation is sufficient to change our students’ use as far as pre-editing is concerned.

4.5 Awareness of risks

Finally, we wanted to see whether the awareness of the risks associated with MT use had changed after our intervention. The DigLit survey was able to confirm such a change when we isolated the data for first-year BA translation students at the Zurich University of Applied Sciences who had received an in-depth ninety-minute training by two members of the project. Their awareness of the risks was significantly higher than other users. This, unfortunately, was not the case for our cohort, as shown in Figure 9.

Figure 9:

Answer to the question “On a scale of 1–100, how would you rate the risks of using MT in the following domains” (our data; DigLit survey 2021–2022, all users).

As the results indicate, UniNE students do not recognise that using MT could entail high risks for a writer’s reputation or result in ethical problems or miscommunication and misunderstandings. This clearly suggests that our input, which focused mainly on data protection, was not sufficient to trigger any change in the students’ behaviour and perception. Personal safety was the only point that had slightly higher results for our cohort, but the difference does not seem significant as it is too small and there are no changes to the risks associated with academic integrity, which we also touched upon in our interventions.

This point will need to be developed in any future presentation as an awareness of the risks is crucial to a critical use of MT. We feel that a longer presentation will likely be necessary for all students, one that features some of the aspects that worked well in our interventions and additional ones that have emerged from the evaluation above.