Critical AI literacy for applied linguistics and language education students

3.1 Focus group data

The data of this study derives from a focus group conducted with learners from the 4th year of the Bachelor degree in English Studies at a Spanish EMEMUS university that emphasises the development of disciplinary knowledge alongside academic literacies. These literacies are taught through workshops, modules, and sometimes tailored language support programs that support the development of skills like academic writing and critical reading. The group of learners who participated in the focus group were recruited through the module, Discourse and English language mass media, during the 2023/2024 academic term. In this module, the students analyse English language mass media communication using corpus-assisted discourse analysis (CADS) for the first time in the degree. The module sees students examining the representation of minorities and people with diverse sexual orientations, discourses related to political ideologies, the emergence of populism, the extreme right in English speaking countries, and discourse and identity, inter alia. Through this module, the students hone their undergraduate research abilities, analyse and evaluate language data collected from a variety of sources (mostly newspapers), and gain insight into the ways in which discourses around a variety of topics are found in mass-media texts. The module also presents opportunities for indirect advanced language learning.

As the module progresses, learners are required to work in groups and select a research topic that they will investigate in depth. They are given the freedom to choose the ideological issue they wish to analyse, the time frame (span of years) for the analysis, and the newspapers on which they focus (usually from the US, the UK, or both). To do so, students use corpus methods and language data analysis to gain insights derived from frequency patterns, facilitating the identification of prevalent and less common discourses within societies. The adoption of corpus methods in this module is premised on the view that querying corpus data enhances our comprehension of the materialisation of discourses in texts and the role of language in the representation of objects, individuals, concepts, and social issues (Pérez-Paredes 2024). This module is designed to be a culmination of the learners’ undergraduate journey through applied linguistics and language education research. They are encouraged to draw on knowledge and experience from other modules in the programme and make use of all of their linguistic expertise.

Three learners took part in the focus group which took place in June 2024. Their profiles are shown in Table 1. They are similar in that they performed well in the module and showed advanced critical skills when analysing the language used in English-speaking mass media. However, Participants 1 and 2 showed a cline towards quantitative language data analysis and had completed the final degree project on the characteristics of AI-generated discourse. Participant 3, on the contrary, was invested in qualitative analysis methods and had not yet started her final degree project. The participants were informed about the nature of the research activity prior to their involvement. Each student willingly provided their consent to participate, understanding the objectives and methods to be employed. We adhered to the ethical guidelines prevalent in social science research, ensuring confidentiality and the right to withdraw from the study at any point without any repercussions.

As the focus group progressed, the participants were asked a series of questions pertaining to applied linguistics, corpus linguistics, language education, and artificial intelligence. These questions addressed:

1. Their experiences of using corpus linguistics to analyse English texts quantitatively and qualitatively;

2. Their perception of the skills involved in language data analysis;

3. Their perception of the impact of AI on language education and applied linguistics; and

4. Their experience of using ChatGPT for language learning and undergraduate research tasks.

The rationale for engaging with questions of corpus linguistics in particular derives from two main lines of reasoning. First, the participants had all studied corpus linguistics recently, thus, this was a topic on which they could speak readily, allowing for conversation to flow. Second, as corpus linguistics is the most computational approach to language study and learning in which these learners engaged throughout their degree, we wanted to explore how participants would compare, contrast, or conflate corpus linguistics, AI, and GenAI. Such an insight would allow us to see where corpus literacy and CAIL align and how corpus literacies could be used to develop CAIL. Throughout the focus group, the discussion was reoriented towards AI, in order to keep the focus on learners’ understandings of AI and GenAI, the ways in which they use AI and GenAI, their perceptions of AI and GenAI, and their engagement with the ethics of AI and GenAI.

As part of the focus group, participants were also asked to engage in an evaluative task, critiquing output from ChatGPT 4. The GenAI tool was asked to propose a response to an essay topic for the module. The participants were asked to reflect on ChatGPT’s response to the essay question, “What is the role of language in discourse(s)?”, and evaluate it in terms of quality. Figure 1 shows a screenshot of the output from ChatGPT. In our view, the AI-generated reply is vague and omits most of the contents and the insights from the compulsory readings of the module. As such, the response ignores the critical concepts involved in the analysis of discourses from a CADS perspective (e.g., Baker 2023; Partington 2006; Pérez-Paredes 2024).

Figure 1:

ChatGPT 4 reply to the prompt “What is the role of language in discourse(s)?”

We recognise that the small sample size (N = 3) undermines the generalisability of the findings we share in Section 4. As such, any arguments made should be viewed in light of what they are: part of an exploratory study endeavouring to situate CAIL within applied linguistics and language education disciplinary confines. As such, it is never our intention to generalise and it is important to note that this study is driven by a qualitative agenda that allows us to conduct in-depth analyses of situated learner discourses that help us to make sense of emerging CAIL within a controlled context. In preparation for analysis, the focus group was automatically transcribed, then manually revised, anonymised, and checked for accuracy by the researchers. The transcript was parsed by speaker turn and the analysis centres only on turns produced by the participants. In total, the focus group lasted 97 min and produced a total of 7,698 words, after removing the interviewers’ language from the count.

3.2 Analytical approach: corpus linguistics and critical grounded theory

For the analytical approach, we adopt a method outlined in Curry and Pérez-Paredes (2023) for interview and focus group analysis. This approach combines corpus linguistics and critical grounded theory with the criteria for AI literacy drawn from Marx (2024). First, the interview transcript was saved as a text file and analysed using Sketch Engine (Kilgarriff et al. 2014). Keywords and key terms were computed using a corpus of interviews and focus groups with English-Medium Instruction (EMI) lecturers from a Spanish university as a reference corpus – for details on this corpus see Pérez-Paredes and Curry (2023). This corpus was selected as it is composed of discussions about university education in the same EMEMUS Spanish context. Therefore, general language of university education would be less likely to appear key and, given that these data were collected in 2019, there were no discussions of AI. This means that language relating to AI and GenAI was more likely to appear as keywords and key terms. This is valuable as these are discourses we are interested in analysing. Once key words and terms were computed, an alpha cutoff of 0.05 was used to determine statistically significant keywords. This relatively low threshold was applied as the data for the target corpus (i.e., the focus group with learners) was quite small (7,698 words). Keywords and multi-word terms were then extracted with a total of 39 significant keywords and four significant key terms identified. These words are presented in the Appendix A. These key words and terms were used as field codes, following Curry and Pérez-Paredes (2023). To apply the field codes, each turn in which any of these words occurred was tagged with key words and terms. In Example 1 below, the keywords parameters, text, like, and know were tagged as field codes.

Example 1 (Participant 3)

probably to know which are the useful variables set like, that, like the parameters you have to use in order to find what you want to find, because you know it’s there, but you don’t know how to reach it. So, to have the tools to reach the amount of text you need that’s ―yeah.

Not all instances of field codes reflected discussions of AI or GenAI, e.g., like in Example 1 is simply a feature of spoken discourse. Yet, other words, such as text and parameters were effective at signalling sites in the discourse in which discussions of AI and GenAI emerge.

Using the field codes as a way into the discourse, CAIL codes, based on Marx (2024), were then applied. These codes are outlined in detail in Table 2.

Once these codes were applied, focused codes were then developed to tease apart the different themes and ways in which issues of CAIL were being evoked in the discussion of applied linguistics and language education. Table 3 presents an overview of these focused codes, linked to their overarching AI literacy code.

These focused codes were iteratively developed through the coding process, and applied to every turn in which their associated themes were found to occur. As Dafouz and Smit (2020) note, when using qualitative coding to understand a complex concept, there is an inevitable degree of overlap in the codes. We share their view that such overlap is a strength of qualitative coding as it affords insight into related and intertwined concepts. To ensure consistency of coding, Stemler’s (2004) consensus estimates were used to iteratively and collaboratively apply AI literacy codes and generate focused codes. Two of the authors coded the entire transcript and negotiated the final set of codes.

4.1 Critical thinking

Critical thinking is the most frequent code applied to the focus group data, accounting for 34 % of all CAIL codes. This prominence also exhibits small variability across the participants, representing 29 %, 36 %, and 37 % of codes applied to discourse produced by Participants 1, 2, 3. In total, 34 turns, accounting for 4,343 words, are attributed with a CT code (either independently or in combination with other codes). Only 5 % of CAIL codes account for CT alone, indicating a propensity for discussions around critical thinking to co-occur with other themes. Most prominently, CT co-occurs with PA in 16 % of the tags applied to turns, followed by EA at 14 % and TU at 9 %. This kind of co-occurrence is evident in Example 2 in which participants demonstrate critical thinking skills while discussing the affordances of ChatGPT for supporting essay writing.

Example 2 (Participant 2)

“The fact is that ChatGPT is not an expert on this field. […] But because, as we said, like, we know how to answer questions and ChatGPT doesn’t, really. Like, it for, I think it’s a good tool, like if you read this then you can propose a good answer. But ChatGPT didn’t answer your question at all, but at least I don’t like the answer.”

CT also co-occurs with multiple other codes, albeit to a lesser degree. For example 5.8 % of all tagged turns contained CT, ET, and PA tags, 4 % contained CT, ET and TU tags, and the co-occurrence of all four tags accounted for just 3 % of tags. Example 3 showcases a complex engagement with critical thinking as the learner evaluates the potential quality of responses but, at the same time, does not question the ethical issues with getting ChatGPT to write a response. The learner also indicates some ideas of the practical uses of GenAI tools as a writing support but the assertion of what the tools ‘know’ may indicate a limited understanding of how GenAI tools work.

Example 3 (Participant 1)

“I think it will be it will create a great response just for the question of the language. I think it’s kind of simple question. So ChatGPT will be able to generate a response based on what it knows about language […] However, I don’t think that the same result will be from the second question. I think that the sense, or the of the meaning, the meaning of representation in that sentence will be kind of a problem for ChatGPT to understand what this question really means.”

Thus, questions of critical thinking abound in these data, as the participants endeavour to assess the reliability and validity of GenAI tools, question or fail to question their role in their studies, and consider issues of over-reliance.

Overall, 25 % of all focused codes pertain to CT01 and 11 % to CT02, rendering these two focused codes the most prominent focused codes in the data. Many different focused code combinations occur in the data, the most prominent being combinations of CT codes as well as CT and PA codes. In some instances, CT01 and CT02 co-occur, indicating a complex engagement with criticality among the participants. In Example 4, one participant demonstrates a critical perspective on analytical approaches, indicating that ChatGPT could not conduct an in-depth study of language that meets the rigorous expectations of corpus linguistics, which reflects wider discussions in the literature (e.g., Curry et al. 2024). Yet, they also propose that ChatGPT can support learners in operating research tools, which is not always the case as ChatGPT has a propensity to also create issues in the learning process. Thus, the participant shows criticality from an analytical perspective, but when viewed as a supporting resource, more limited criticality is exhibited. As such, questions of quality may need to be addressed with learners, focusing not only on the importance of quality knowledge in analyses, but in every aspect of the research process – a central facet underpinning learners’ understanding of AI, as reflected in Miao and Shiohira (2024).

Example 4 (Participant 3)

“I think that for example in the case that, for instance, a researcher forgets to use something like, for instance, I don’t know, a database, you can ask ChatGPT to give you a sort of list to ―of how to use it as such. And for that it can be useful for the researcher. But for corpus linguistics to make a full ―a deep search for a topic or something like a database does, I think that it cannot be that useful but it can also offer some answers that can be, you know, resourceful.”

In terms of CT and PA, CT01 and PA03 co-occur most often. When they do, participants discuss the need to be critical of GenAI tools when applying them as part of the analytical process. They draw on their wider knowledge of linguistics, citing scholars like Fairclough and Foucault, as means to explain their nuanced views on the use of GenAI, as demonstrated in Example 5. In that example, the participant critiques output from ChatGPT, assessing its quality favourably but determining the need for more information.

Example 5 (Participant 2)

“I would go for the ―I don’t remember right now, excuse me, if it was Fairclough, Foucault ―the CDA perspective in which language and discourses and, well, especially the aim, once we have them, was to have like a critical awareness of what discourses could ―were able to do with our identities and our ideologies and social structures and the ability they have to modify them.”

Overall, the participants demonstrate a high degree of critical thinking when discussing and evaluating the GenAI tools. Notably, they often draw on wider disciplinary knowledge to substantiate their view and offer explanations and nuance. There remain issues with their engagement with GenAI tools, as they exhibit a lack of criticality at times, both in terms of their use of GenAI tools and their understanding of how such tools work. Given the potential affordances of knowledge transfer across disciplinary confines for advancing thinking and learning (e.g., van Peppen et al. 2022), developing conceptual links between different areas of applied linguistics, language education and GenAI – as the learners instinctively do – may prove fruitful in helping learners map their critical perspectives against new and emerging GenAI tools.

4.2 Ethical awareness

The ethical implications of using AI in education environments was also a recurring topic among the participants. EA occurred in 25 turns, accounting for 3,328 words. When combined with other tags, questions of EA were raised mostly in combination with CT (14 %), while appearing less frequently alongside TU (8 %) and PA (8 %). The participants demonstrate mixed feelings towards the ethical considerations of using GenAI in educational contexts. Their perspectives range from seeing AI as a clear threat to both human cognitive development and the ability of humans to access or retain job positions to the opposite view, suggesting that research should focus on how we can work alongside AI instead of analysing the potential problems GenAI tools may pose.

For EA01, the participants highlight the importance of being able to access and understand the data on which AI and AI-based tools are trained. In their view, it is an ethical duty that users develop sufficient critical thinking skills to be able to discern useful from non-useful feedback. They argue that such a practice is fundamental for language research generally, and for being able to make sense of GenAI produced output specifically. In this way, participants appear to see greater value in using GenAI as a sort of companion tool to amplify information rather than using GenAI to generate text, as Example 6 illustrates.

Example 6 (Participant 3)

“[…]when I’m studying and I see that a summary that I have about a unit or something is, like, short, I […] look for information [in ChatGPT] to amplify what I have and that’s all for now ―I have also used it to translate the text and then to do the evaluation of the translation made by ChatGPT.”

When faced with the question of to whom or what AI-generated data belongs, the participants struggle with finding a definitive answer, as in Example 7.

Example 7 (Participant 3)

“It’s a kind of a difficult question to answer, because you don’t know if it’s the creator as such of the AI […] does ChatGPT owns text? I don’t really know about that.”

The literature remains similarly unsure as to how we can answer these questions effectively. The interplay between GenAI and both the data it uses as sources and the data it outputs has been the subject of discussion on ethical and legal grounds (e.g., Eshraghian 2020). Draxler et al. (2024), for example, note that people tend to avoid assigning authorship to AI, especially when GenAI has been used as a ‘companion tool’ rather than the primary source of output. Given the immense complexity involved in addressing the question of data ownership in using GenAI, it is imperative that it form part of situated CAIL.

In terms of the quality of texts produced by GenAI (EA02), the participants seem confident in their ability to correctly distinguish AI-generated text from human-generated text, generally alluding to “mistakes” in AI pattern selection and organisation, which, in their view, are visibly different from the ones humans may make. This is evident in Example 8.

Example 8 (Participant 3)

“[…] we can distinguish [translations made with ChatGPT] by the errors that they made as such because […] they don’t relate to context […] as well as we humans do. […] For example, humans, we make errors depending on […] how exhausted we are. We have made an error that then, when we revise the text, we see it and we change it. But for ChatGPT I translated the text and click like four, five times and it made the same error.”

Regardless of their possible personal reluctance towards AI, they seem to deem GenAI tools useful for a handful of specific tasks, such as generating a short piece of text. They declare their clear preference towards managing the selection, cleaning and compilation of a corpus ‘by hand’, which, in their view, promotes their research skills and ensures an ethical engagement with the data.

For EA04, the participants seem to be aware of a growing dependence on GenAI. The ways in which GenAI could negatively affect humans were discussed in depth in several turns, with participants showing clear differences in perspectives. Participant 1 believes the capabilities of AI are yet to evolve and that the focus of AI-related research should be directed towards what they can offer rather than the problems they may entail. This suggests the current impact of AI tools in educational settings is not as dire as some may initially regard. In contrast, Participants 2 and 3 openly label AI as a “threat” to human cognitive development, especially in the case of very young users, as Example 9 illustrates.

Example 9 (Participant 3)

“I feel that it’s making us lazy, kind of non-thinking students.”

This same view is echoed by Participant 2, who implies that indulging in the yearning for constant stimulus and instantaneous feedback might potentially hinder human cognitive development, as people would not make “a real effort” in using their own capabilities to perform tasks when they can reach a potentially comparable result in mere seconds. From an educational perspective, such a use of GenAI risks undermining efforts by teachers to develop process-driven and pedagogically-situated approaches to using AI in education (Yang et al. 2024).

The participants express deep concerns about how the existence of GenAI tools may affect future generations. For the unprepared users, they fear that GenAI will become a substitute for critical thinking processes, especially those related to written comprehension and production. In their view, children are not sufficiently educated at home to deal with the proliferation of different devices and tools available to them. Aside from the impact of GenAI on young users, Participant 2 was also concerned about the perceived threat that AI poses to certain jobs, especially those related to linguistic production, naming translation as an “at risk” area. Participant 3, in contrast, believes that there is no such threat, trusting that the quality and reliability of human translation will always overcome automatic GenAI translation. These perspectives underscore even further the sheer importance of providing an ethics-driven accessible and comprehensive CAIL to potential users of GenAI tools, especially at a very early age. Such literacy would provide the necessary tools for humans to manage both the potential tasks GenAI can perform reliably and, even more importantly, to critically interact with GenAI production instead of uncritically accepting.

The question of whether AI can be regarded as human also emerged from discussions with participants (EA05). While such a question has traditionally been a philosophical consideration reserved for science-fiction literature, the human-like production of GenAI has brought it into our realities. The participants agree in not considering GenAI production as human, suggesting that the process of thinking requires more than putting together words to form coherent sentences, demonstrating a capacity to make sense of AI through their applied linguistics lens. For the participants, the very act of creating something is inherently human, as GenAI is only able to replicate what other humans have already produced. As such, they argue that AI cannot form opinions by itself, unless prompted to it by a human beforehand.

4.3 Practical applications

The analysis of the focus group reveals a blend of enthusiasm, scepticism, and reflective insight about integrating AI into practical applications. Discussions span issues of data analysis and problem-solving. 22 turns tagged as PA were analysed, totalling 2,502 words. PA codes account for 22 % of all codes applied to the data. These codes occur alone only in 2 % of tagged turns and most often co-occur with other codes, including CT in 16 % of tagged turns, EA in 8 % of tagged turns, TU in 9 % of tagged turns, CT and EA in 8 % of tagged turns, CT and TU in 4 % of tagged turns, and EA and TU in 5 % of tagged turns. PA also occurs with all other codes in 3 % of tagged turns. Thus, PA is a highly interconnected theme.

The participants’ opinions collectively highlight the promise and challenges of using AI for practical applications. Key takeaways include the importance of robust support during the learning curve, the indispensable role of human oversight, and the value of hands-on projects in making AI tools meaningful and relevant. These insights underscore the view that an effective integration of AI in language education university programs requires thoughtful design, ethical vigilance, and a balance between technical capabilities and human expertise. The human-centred nature of their perspective reflects a more general trend at the intersection of AI and education (e.g., Miao and Shiohira 2024)

For PA01, the participants’ reflections show the growing utility of AI in addressing complex challenges, particularly in language analysis and research. Tools like Sketch Engine exemplify how software facilitates textual analysis and pattern recognition, simplifying what might otherwise be arduous manual processes. The participants appreciated the systematic capabilities AI tools provide, allowing for an expanded comprehension of language use and discourse analysis by uncovering implicit meanings. However, they also highlighted the steep learning curve, suggesting that effective integration of AI-driven data analysis and the use of complex corpus analysis tools requires substantial initial investment in understanding and adapting their use to specific contexts, such as applied linguistics and language education, as illustrated in Example 10.

Example 10 (Participant 1)

“We worked with Sketch Engine a few years ago in another subject. We didn’t get anything at all, and it was not really clear what we were dealing with and I think that for the first time we now are able to understand what Sketch Engine is capable of.”

The initial struggles with Sketch Engine suggest a steep learning curve, particularly when tools are not adequately explained or contextualised. This capacity to critically engage with such tools demonstrates their willingness to persist in iterative learning when working with technology – a capacity directly transferable to their CAIL.

The participants’ reflections highlight the cognitive shift that AI necessitates when engaging with the variables involved in the analysis of mass media texts written in English (PA02). Participants described the iterative process of refining their queries and datasets, which is critical for generating meaningful results. They noted that understanding and managing variables like time frames (years or months) or sources such as different newspapers or academic references impacts the reliability of their analysis – an issue of disclosure as identified in Yang et al. (2024).

Their reflections demonstrate how tools for data analysis such as Sketch Engine encourage a more structured and critical approach as they necessitate a research methodology that facilitates critical analysis and makes users more meticulous in their planning and execution phases. For example, they value how the ability to manipulate large datasets and refine search parameters (e.g., time frames, genres, etc.) allows for detailed and precise linguistic analyses. Participant 1 noted a newfound appreciation for Sketch Engine after gaining a clearer understanding of its functionalities, despite earlier struggles in previous years (see Example 10). Once again, this knowledge and these skills developed in applied linguistics are directly mappable to CAIL, meaning that there are opportunities for transfer.

For PA03, AI’s role in support functions, like generating references or summarising texts, has been lauded by the participants for its efficiency. For example, Participant 3 mentioned the use AI to amplify study notes and evaluate translations, indicating its utility in supplementary academic tasks. While these applications streamline workflow and reduce redundancy, the reflections also point to a dependency on human oversight. The importance of validation and cross-referencing AI outputs with established sources was emphasised by the students as an important means of ensuring academic rigor and accuracy.

For PA04, the importance of validating AI-derived insights with authentic sources and human judgment is a recurrent theme. Participant 2 raises a critical concern: without a credible source, they wonder how AI outputs can be trusted. This highlights the importance of providing AI with high-quality, reliable data. The reflective question about relying solely on AI-generated ideas emphasises the ethical and practical necessity of human involvement in AI-driven processes. Participant 1 reflects on the ability to infer meaning from what is explicitly stated and what is omitted, particularly in text analysis. This insight reveals an appreciation for AI’s nuanced capabilities, such as outlining implicit ideas or even bias in textual data. This perspective aligns with AI’s strength in processing large datasets to uncover patterns and meanings beyond human intuition. However, the same participant hints at the risks of over-reliance, particularly when AI tools such as ChatGPT fail to offer the “real source or sources”. This is echoed by Participant 2, as Example 11 demonstrates.

Example 11 (Participant 2)

“I think it depends on how you use this tool, but if you don’t have the real source, how can you really contrast the information?”

AI’s analytical capabilities are recognised by the participants as a double-edged sword. On one hand, it aids in generating initial insights or hypotheses by synthesizing vast amounts of data. On the other hand, participants identified its limitations in critical areas like discourse analysis, where nuanced understanding is required. For instance, while AI can produce general summaries or responses, it struggles with deeper contextual or theoretical interpretation, often offering disjointed or overly generalised conclusions. This highlights the complementary rather than replacement role of AI in scholarly analysis.

For PA05, participants reflected on AI’s potential to perform increasingly complex tasks, from constructing corpora to assisting in translation. They speculated about its growing capability to outpace human performance in specific areas, such as machine translation. However, scepticism persists regarding AI’s ability to replicate human expertise, especially in domains requiring critical thinking and contextual understanding. The reflections suggest a cautious optimism about integrating AI into professional environments, where it could function as a tool for augmentation rather than replacement. This perspective aligns with broader trends advocating for AI literacy to maximise its benefits while addressing its limitations (Yang et al. 2024).

Overall, the participants appreciated the relevance of their hands-on project, describing it as a “real project” and recognising the applicability of their learning to authentic contexts. The elective nature of the module seems to have attracted motivated individuals who value the subject matter, resulting in a generally positive perception. However, the participants acknowledged that interest in the module and AI tools may vary. Participant 2 emphasised that the module was engaging mainly for those who found the analysis of discourse interesting. The participants agreed that, while interest-driven learning is valuable, broader educational strategies should also aim to make AI accessible and appealing to a wider audience, given its increasing relevance in diverse fields. Taking such thinking on board, there may be a case of both disciplinarily situated CAIL that complements a more generic CAIL.

4.4 Technical understanding

Finally, TU, which pertains to what the participants know about AI systems, how they are trained, how they function, and their limitations, account for the most infrequent number of tags in the data. 18 % of all AI literacy tags were TU tags and these occurred across 18 turns in 2,823 words. TU occurs alone in 4 % of the tagged turns. Therefore, it mainly co-occurs with more prominent themes, including CT in 9 % of tagged turns, EA in 8 % of tagged turns, PA in 9 % of tagged turns, CT and EA in 4 % of tagged turns, and EA and PA in 5 % of tagged turns. TU also occurs with all three other tags in 3 % of tagged turns.

TU04 is the most common focused code, followed by TU03 and TU02. These focused codes also co-occur with focused codes from other CAIL codes, including EA04, CT01, and PA02, most prominently. Thus, once again, the role of technical knowledge in the participants’ AI literacy is intertwined with questions of ethics, critical thinking and practical applications, illustrating the complex nature of CAIL and the need to address these issues holistically.

When combined with EA04, TU04 is tagged to utterances in which participants demonstrate a partial understanding of how GenAI tools and their LLMs work, drawing distinctions between human thought processes and that of AI, as Example 12 presents. In this case, the participant draws on their metalinguistic knowledge of LLMs, conceiving their processes as ones designed to produce a string of words. The participant challenges this process as not matching human intelligence, also signalling an understanding of how humans produce text.

Example 12 (Participant 1)

“Yeah, ChatGPT work with, as far as I know, with prompts and datasets and it’s supposed to kind of learn about humans from all those samples […] It’s just ―it’s not a brain who is thinking ―it’s kind of mixing words that ChatGPT knows that exist, knows that tend to appear in similar context or in the same context.”

TU03 occurs with critical thinking in utterances in which participants offer a critical perspective on AI while also revealing a rather limited understanding of its mechanics. However, this combination, as Example 13 demonstrates, signals an effective critical perspective that accounts for the participant’s lack of knowledge. Thus reflecting on a lack of knowledge can prove a useful means in highlighting gaps in knowledge around AI that can curtail inappropriate usage.

Example 13 (Participant 1)

“In my case when ChatGPT appeared I didn’t understand what it really was, and I, at the beginning I didn’t use it. I found it really difficult. I, to be honest, I kind of ―I mean, I didn’t understand how it work, how I was supposed to ask, or how the ChatGPT was supposed to answer the question, and I didn’t know if the response was ―were reliable enough.”

Elsewhere in the focus group, participants demonstrate a greater understanding of data, more generally, indicating the kinds of technical knowledge (TU02) needed to put tools to use (PA02) – specifically corpus linguistics tools. In Example 14, one participant highlights the need for linguists to understand exactly what their data represents to perform effective analysis. This demonstrates a data literacy typical of the rigor of applied linguistics. Contrasting Example 13 with 14, it is clear that these participants have a complex understanding of language data and issues of representation and rigorous analysis. Using criteria established in the domain of text analysis, for example, may prove an effective means of illuminating challenges with GenAI when developing CAIL.

Example 14 (Participant 1)

“You might think what you are thinking or what you are imagining about your topic is enough, and then you just find around 50 texts and you know that’s not enough […] So it’s clear that you have to be as cautious as possible in order to, you know, that if you mess up that stage you are going to mess up all the project.”