EU AI Act: what could AI literacy mean for medical laboratories? – Opinion Paper on behalf of the section digital competence and AI of the German Society for Clinical Chemistry and Laboratory Medicine (DGKL)

Article 4 of the EU AI Act on “AI literacy”

The EU AI Act applies to all AI systems used in the European Union, regardless of the country of origin of the AI system (see Art. 2 (1a) EU AI Act). The only exceptions are purely private, “non-professional” use (see Art. 2 (10) EU AI Act) of AI systems and the use of such systems in the context of research and development (see Art. 2 (6) EU AI Act). There are also other exceptions, such as those relating to the national security of member states (see Art. 2 (3) EU AI Act). This means that the EU AI Act also applies to AI systems used in medical laboratories. To use AI systems, users must demonstrate “AI literacy.” Article 4 of the EU AI Act sets out various requirements for this:

1. Measures must be implemented to ensure the AI literacy of the company’s own employees and, where applicable, contracted people.

2. Employees must have a sufficient level of AI literacy.

3. The training courses should be context-dependent, depending on technical knowledge, experience, education, and training, as well as the AI system used. The context here also includes the people for whom the AI systems are to be used (e.g., patients).

This raises three key questions that a laboratory should answer when developing a training program:

1. Which AI systems does the laboratory already use and what is the intended purpose of those systems (defined by the provider)?

2. Who will use and operate the system? Which knowledge is already available and which knowledge is required to use the tool effectively?

3. For whom (e.g., patients) will the system be used? What potential biases may occur?

The first question raises the difficulty of defining an AI system. Whether a system is an AI system within the meaning of the EU AI Act is based on a case-by-case assessment of the respective system regarding the definition of an AI system under Article 3 (1). Unfortunately, this definition is very broadly formulated. The definition is as follows:

‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. [2]

The EU Commission has therefore published a 16-page “Guideline” on the interpretation of the definition [3] but explicitly points out that this is not binding and that the exact legal interpretation is the responsibility of the European Court of Justice. The authors of this article consider the following quote from the Commission’s guideline on the definition of AI to be particularly crucial in relation to laboratory medicine:

5.2. Systems outside the scope of the definition of an AI system: […]

(40) Recital 12 also explains that the AI system definition should distinguish AI systems from “simpler traditional software systems or programming approaches and should not cover systems that are based on the rules defined solely by natural persons to automatically execute operations. [3]

This means that rule-based systems defined by laboratory personnel or clinical decision support systems that operate based on static rules are unlikely to fall within the scope of the EU AI Act.

It is also important to consider the overlap between the EU AI Act, the Medical Device Regulation (MDR) and the In vitro Diagnostic Device Regulative (IVDR). AIB 2025-1/MDCG 2025-6 was recently published on this topic, according to which all products that correspond to classes A, B, C, and D of the IVDR and for which a notified body must be involved are considered high-risk AI systems [4]. Interestingly, AI systems classified as in-house devices under Art. 5 (5) IVDR are not considered high-risk AI systems (see MDCG 2025-6, p. 6, Table 1). In addition, there is another important overlap between the EU AI Act and the GDPR, as health data is considered particularly sensitive private data and therefore a high level of protection must be ensured when using AI tools to improve healthcare.

As there is currently no precise wording regarding the implementation of the aforementioned legal provisions, the authors of this article point out that the interpretations of the legal text presented here can only be regarded as a suggestion. The interpretation of the text is carried out exclusively by the European Court of Justice.

Module 1: general knowledge on the EU AI Act

To understand why certain content about AI needs to be taught, it is necessary to understand the purpose of this training and the most important aspects of the relevant legislation. Additionally, laboratory staff should be informed why the EU AI Act also applies to medical laboratories, as explained in the introduction of this article.

The EU AI Act has a major, fundamental objective, as explained in Article 1:

The purpose of this Regulation is to improve the functioning of the internal market and promote the uptake of human-centric and trustworthy artificial intelligence (AI), while ensuring a high level of protection of health, safety, fundamental rights enshrined in the Charter, including democracy, the rule of law and environmental protection, against the harmful effects of AI systems in the Union and supporting innovation.

The aim of the EU AI Act is therefore to safeguard the health, safety, and fundamental rights of the people in the European Union. Even though many people currently use AI systems without much criticism, the EU AI Act rightly focuses on the necessary protection of these fundamental values.

The EU AI Act takes a risk-based approach, dividing AI systems into different risk classes. AI systems that are banned in the EU are distinguished from systems with high, moderate, low, or no risk. The ban on certain AI systems has also been in power since February 2, 2025. Considering the overlap between the MDR, IVDR, and the EU AI Act, it can be assumed that most AI systems used in medicine will be classified as high-risk AI systems (see introduction, all system that require involvement of a notified body). Another important aspect of the EU AI Act is the distinction between providers and deployers and the associated obligations that must be fulfilled to comply with the EU AI Act. In addition to AI systems that are specialized for specific tasks, the EU AI Act distinguishes between so-called “general purpose artificial intelligence” (GPAI), which includes chat-based systems such as ChatGPT or Gemini. In this case, other requirements must be met. Finally, training should be provided on the basics of data protection when dealing with AI systems.

Module 2: fundamentals of artificial intelligence

To gain a basic understanding of artificial intelligence, it is essential to clarify some of the most important terms. Training should therefore first distinguish between rule-based systems, such as rulesets in a Laboratory Information System (LIS) or in a clinical decision support system (if they are expert rules (hard-coded) without dynamic adaptation of the system), and genuine AI systems. Rule-based systems have long been used in laboratory medicine and are unlikely to fall under the provisions of the AI Act (see introduction). In order to understand the terminology used in the field of AI, basic knowledge of the different types of machine learning, such as supervised learning, unsupervised learning, and reinforcement learning, should be taught. Deep learning techniques, as a subfield of machine learning, are particularly important for further understanding AI systems such as large language models (LLMs), so the basic structure of neural networks, their fundamental function, and examples of their application, such as image recognition, should be discussed.

Today’s debate about AI revolves primarily around generative AI models. It is important to provide employees with basic knowledge of foundation models, especially large language models. It is helpful to structure this knowledge around the development process of an LLM (pre-training, supervised fine-tuning, reinforcement learning, reasoning), as this also provides a basic understanding of the possibilities and limitations as well as the fundamental functioning of LLMs. Based on this knowledge, phenomena such as hallucinations, sycophancy, alignment faking, and security issues such as jail breaks or prompt injections should then be addressed. Problems such as biases in the training data, which lead to biases in the model results, also need to be discussed, as this can lead to relevant problems in the care of different patient groups (e.g., gender or ethnicity bias). Finally, the fundamentals of the function and possibilities, but also the limitations and security concerns of agentic AI systems should be addressed.

Module 3: context-dependent training

Once the fundamentals of the EU AI Act and a general understanding of the various possibilities of AI models and systems are known, AI literacy should be tailored to the tools used in the laboratory in a context-dependent manner. It is important to take into account the existing knowledge of employees and to address the specific algorithms used in the AI tools. It is crucial to emphasize that AI tools may only be used for the purpose intended by the manufacturer. Based on this, the basic functioning as well as the strengths and weaknesses of the AI system can be discussed. This requires information from the manufacturers, which is also required of them in exactly this form by the EU AI Act. Based on this information and fundamental knowledge of how AI models work, the outputs of the tools used in practice should be critically examined. In this context, reference should be made to the high relevance of the “human-in-the-loop” principle, which is explicitly required by the EU AI Act for high-risk AI systems. The basics of evaluating AI systems should also be discussed (especially if AI systems are to be developed as in-house systems). Finally, training should also include content on the possibilities and obligations of quality and risk management for the AI systems used.

In Figure 1, we have attempted to present a framework that is helpful for classifying AI systems and their requirements in terms of regulation, knowledge, and control by employees. While most AI systems currently are mainly used privately (base of the pyramid in Figure 1, “cellphone-based AI”), such as the apps from OpenAI, Google, or Meta AI in WhatsApp, laboratory staff will increasingly be confronted with AI-based IVDs in the future. Some laboratories will also develop their own AI tools, and universities and specialized institutes will conduct research in the field of AI for laboratory medicine. Depending on the application, different requirements must be placed on AI literacy. In the case of AI- or machine learning-based studies in laboratory medicine, the EFLM working group recently published a checklist that can serve as a basis for the design of corresponding studies in the field of science [9]. The IFCC has also issued corresponding recommendations in 2023 [10]. Since there are no official guidelines on AI training yet, thinktanks such as working groups of national and international professional societies should seize this opportunity and contribute possible training content and foundations for technical, professional, and ethical frameworks.

Figure 1:

AI systems and their relevance for Article 4 of the EU AI Act. The pyramid shape is intended to symbolize the frequency of AI applications.