Capable exam-taker and question-generator: the dual role of generative AI in medical education assessment

Literature search

The literature search and screening process followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guideline [64]. Scopus database was searched in September 2024 with keywords of “Generative Artificial Intelligence,” “GAI,” “ChatGPT,” “GPT,” “Bard,” “Bing,” “Claude,” “Gemini,” “DALLE,” “Midjourney,” and “Stable Diffusion,” as well as “medical examination,” “medical exam,” “medical assessment,” “medical test” in title-abstract-keywords, from 2022 to 2024. The records obtained were examined to eliminate any duplicates. Once the duplicates were removed, the titles and abstracts of the retrieved studies were screened to identify those met the inclusion and exclusion criteria (Table 1). Subsequently, the full texts of the identified studies were retrieved, and those inaccessible to the full text were excluded from further analysis. When necessary, papers from reference were manually searched.

Data analysis

Data from the included studies were extracted into Microsoft Excel spreadsheets. The extracted characteristics of the studies included: title, authors, publication year, medical examination name, examination type, specialty, question type, country or region of the examination, AI model, prompt strategy, accuracy rate, passing score, error type, quality measure, and language interacting with AI, etc. Narrative analysis and descriptive statistics were used to analyze the contents of the included studies. When calculating the average accuracy of AI responses to exam questions in a specific medical specialty, only studies with at least 10 questions were included, provided there were at least five such studies. Studies that did not clearly specify the number of questions were excluded. The difficulty distributions of exam questions were assumed similar among studies. Average accuracy was calculated by dividing the total number of correctly answered questions in all exams by the total number of questions. The 95 % confidence interval of the accuracy was estimated using the binomial distribution.

Ethical review

This review was conducted based on published studies; therefore, no ethical review was required.

Searched records

The searching strategy resulted in 119 studies. Then, 2 duplicate records, 49 irrelevant studies, and 2 inaccessible ones were removed; and 3 manually searched records were added. Ultimately, a total of 70 studies were included in this review (Figure 1).

Figure 1:

Literature screening diagram.

AI tools’ performance in different types of medical exam questions

Single-best answer MCQs, choose-n-from-many, true or false, and open-ended questions are possible question types in medical examinations. Single-best answer MCQs are very popular in various medical exams, so AI’s ability to answer this MCQs has attracted the interest of many researchers. Meta-analysis of the published studies shown that ChatGPT-3.5 had an overall accuracy of 61.1 % in Levin et al.’s study [51], and an overall accuracy of 58 % in Liu et al.’s study [4], which were quite similar, while ChatGPT-4 had a higher accuracy of 81 % [4] (Table 2).

Choose-n-from-many is a variant of single-best answer MCQs, which has two or more correct answers in answer options. In Haze et al.’s study, AI’s ability to respond to this kind of question was inferior to answering single-best answer MCQs. For example, ChatGPT-4 had an accuracy of 69.8 % in answering choose-n-from-many questions, compared to an accuracy of 83.7 % in answering single-best answer MCQs [39]. However, in Hirano et al.’s study, ChatGPT-4 Turbo/ChatGPT-4 Turbo with vision had similar accuracy in answering single-best answer MCQs and choose-n-from-many questions [34] (Table 3).

True/false question is also a variant of MCQs, which has only two options. Sadeq et al. reported that AI’s performance in true/false questions was lower than that in MCQs [6]. For example, ChatGPT-3.5 obtained an accuracy of 23.1 % in answering true/false questions, while it achieved an accuracy of 62.9 % in answering MCQs, and similar trends were observed in GTP-4, Bard, Bing, Claude, Claude Instant, and Perplexity [6]. However, in another study, Sood et al. found that GPT-4 had an accuracy of 83 % in answering true/false questions, better than answering MCQs, and so did GPT-3.5 (Table 3) [33].

For open-ended questions, ChatGPT-3.5 obtained 66.5 % accuracy in community medicine [65], 73.6 % in family medicine [66], and 77.4 % in psychiatry [9]. ChatGPT-4 achieved 75 % accuracy in otolaryngology-head and neck surgery [5], 81.0 % in family medicine [66] (Table 3). Both ChatGPT-3.5 and ChatGPT-4 seemed to have a better performance compared to answering MCQs, and both exceeded the common passing threshold of 60 %.

Performance of AI in addressing MCQs across various specialties

Popular AI tools were employed in answering MCQs, and their performance varies across different specialties. Table 4 presents the results categorized by specialties. It reveals that ChatGPT-3.5 and ChatGPT-4 were most used AI tools in medical examinations. ChatGPT-3.5 performed best in psychiatry, with an average accuracy of 74.6 %. Its second-best performance was in general surgery, reached 70.6 % accuracy; then in neurology (61.8 %), internal medicine (61.6 %), and emergency medicine (54.9 %). Its worst performance was in pediatrics as well as gynecology and obstetrics, with an average accuracy of 53.6 %. While ChatGPT-4 performed better than ChatGPT-3.5, it also performed best in psychiatry, with an average accuracy of 90.1 %; followed by internal medicine (84.0 %), general surgery (81.2 %), neurology (78.9 %), pediatrics (78.7 %), emergency medicine (78.3 %), gynecology and obstetrics (76.8 %). ChatGPT-4 performed worst in osteology, with an average accuracy of 67.4 %. Detailed performance of AI tools across various specialties were in Supplementary Material 1.

Prompt strategies in answering questions

In many studies, the original examination questions were directly input to the AI tool [7], 8], 16], 17], 21], which simulated the humans taking the exams. However, a number of studies employed lead-in prompts from simple to complex. The components in these lead-in prompts could be classified as basic components and advanced ones (Figure 2). The most commonly used basic component was requiring AI to select one correct answer for MCQs [15], 34], 41], 44], 71]. The other basic components were to specify specialty field [5], 10], 12], 13], 40], 41], 66], 67] and question type [12], 28], 36], 41], 47], 66]. The advanced components included assigning a professional role in an expertise field [12], 32], 33], 35], 41], explaining and/or justifying its answer [12], 33], 67], 69] or not explaining or justifying its answer [13], 71], 72], identifying learning objective [67], 69], chain-of-thought strategy [32], 34], 41], and few-shot strategy [40]. Roos et al. [10], Wu et al. [40] and Torres-Zegarra et al. [69] employed structured prompts that compiled basic and advanced components in a clearer way (Figure 2). More detailed analysis of prompts in response to medical exam questions were in Supplementary Material 2.

Figure 2:

Components of prompt in answering and crafting medical exam questions. MCQs, multiple-choice questions.

Analysis of incorrect AI answers

Several studies analyzed in detail the types of incorrect answers yielded by AI (Table 5). Guillen-Grima et al. [28] analyzed wrong answers based on Taxonomy of Medication Errors by National Coordinating Council for Medication Error Reporting and Prevention [73], which has 9 categories. They identified that in a total of 24 incorrect answers, 10 could cause medication errors (category A), and 8 could not cause harm to patient (category B–D), while 6 could cause harm to patient (category E–H), and none would cause death (category I).

Herrmann-Werner et al. [12] categorized incorrect answers according to the revised Bloom’s taxonomy, which has six levels regarding to cognition challenge: “remember, understand, apply, analyze, evaluate and create”. In a total of 68 wrong responses, the most incorrect answers were at the “remember” and “understand” level, with 29 and 23 incorrect answers respectively. Maitland et al. [18] classified wrong answers into 8 types in accordance with clinical thinking and reasoning. In 51 incorrect answers, the factual error was the most common one, followed by omission error [18]. Wang et al. [41] divided incorrect answers into open-domain and closed-domain hallucination. They found that GPT-4 had less open-domain and closed-domain errors.

Prompt strategies in generating questions

Prompts used to generate medical exam questions were various (Figure 2). The basic components were the question types [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], subject matter or topics [53], [54], [55, , 62], 63], number of questions [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], number of answer options [59], 62], 63], and requirement to provide correct answers [53], 55], [60], [61], [62], [63]. The key components included aligning to learning objective [56], 57], 63], targeted at specific audiences [53], 55], 56], 60], 63], specifying question difficulty level, such as easy or difficult [58], knowledge-based [55], 60] or clinical case-based [55], [56], [57]; for generating clinical cases or case-based questions, the patient details [59] or clinical vignette details were required [57], 58]. The advanced components included providing examples (few-shot strategy) [55], referring to uploaded file as the question source [54], 60], 61], and specifying the output format of questions and answers [55], 57], 59], 61]. Kıyak et al. [58] employed a well-structured prompt framework to generate MCQs, which consisted of the above-mentioned basic, key and advanced components. Detailed analysis of prompts in generating medical exam questions were in Supplementary Material 2.

Quality assessment of AI-generated medical questions

While AI’s performance in answering medical questions could be evaluated by comparing its answers to reference answers, there are no standard criteria for assessing AI’s performance in generating medical questions. Thus, researchers proposed their own quality measures to evaluate the quality of AI-generated questions (Table 6).

The commonly used quality measures were clarity [54], 58], 60], 62] or ambiguity [61], and correctness [55], 58], accuracy [54] or appropriateness [60]. Measures such as appropriateness [55], 58], suitability [60], validity [56] or instructional alignment [61] were used to judge the degree a question aligned to a topic, content or intended learning objective. For the difficulty level of the questions, some researchers used Likert scale to measure the question difficulty [54], 56], 61], 62], while some other researchers put AI-generated questions in real exams to measure the difficulty [57], 63]. Besides, AI-generated questions in real exams were also assessed by discrimination index [57], 58], 63]. When comparing AI-generated questions to those created by humans, the quality of the AI-generated questions were almost as good as human-generated ones, either by human judgement [60], 63] or by test results [57].

Principal findings

This narrative review highlights that generative AI tools, particularly large language models, demonstrated capabilities in answering and creating medical examination questions. AIs’ performance varied when answering different types of questions, and probably performed best when answering open-ended questions. AIs’ performance also varied when answering different specialty questions, with the best achievement in psychiatry for both ChatGPT-3.5 and ChatGPT-4 [9], 12], 26], and the worst achievement in osteology for ChatGPT-4 [11], 23], 49] and in pediatrics as well as gynecology and obstetrics for ChatGPT-3.5 [39], 46], 49], 70]. When guided by appropriate prompts, AI tools could generate suitable medical exam questions [53], which were comparable to questions created by humans [57], 59], 60], 63].

AI tools’ performance is influenced by question types, specialty knowledge and prompts [20], 39], 67]. MCQs, choose-n-form-many and true/false questions are objective questions, and open-ended questions are subjective questions. Objective questions have question stem and answer options where the clue to the answer is hidden. Essentially, answering objective questions is a kind of finding the best match. There are only two possibilities for an answer: either it is correct or it is wrong; there is no middle ground. While open-ended questions, especially clinical vignette-based questions, require exam taker to apply and synthesize their knowledge, and is not an easy task for humans. However, it seemed not a hard task for AI. AI tools achieved high scores in answering open-ended questions. This might be due to the grading mechanism. Even if the final decision is wrong, each key point can get a score. Since AI was trained on large scale data set, it is quite knowledgeable and easy to generate clinical vignette-related content that may contain key points, thus it performs quite well in answering open-ended questions, not necessary to have real clinical thinking and reasoning skills.

The performance differences among specialties likely stem from a combination of factors such as the types of data available for training the AI, the complexity of clinical reasoning, and the diagnostic process specific to each field. Haze et al. [39] investigated the relationship between the ChatGPT’s accuracy in different specialties and the number of related documents in the Web of Science Core Collection. They found significant positive correlation between the accuracy of ChatGPT-4 and the number of all-type documents. In specialties like psychiatric, where standardized questionnaires and diagnostic criteria are well-documented in textual form, AI can easily process the data, leading to better performance. In contrast, in fields like orthopedics, where diagnostic decisions often rely on interpreting medical imaging, current language models like ChatGPT have limitations, resulting in weaker performance. Pediatrics and obstetrics/gynecology involve more case-by-case variability, where factors like age, medical history, and developmental stage matter significantly. AI model might struggle with the complexities of clinical decision-making, thus leading to lower performance. Additionally, AI accuracy tends to decline when questions involve country- or region-specific knowledge, likely due to limited training on such localized data [67].

Prompt may also influence AI’s accuracy in answering medical questions. In Herrmann-Werner et al.’s study, detailed prompt resulted in a higher accuracy than the short prompt did but without significance [12], because the key components in detailed prompt and short component functioned the same, except that detailed one specified the answer format. When chain-of-though prompt was employed, ChatGPT could correctly answer more than half of the originally wrongly answered questions [32]. When few-shot technology was used to enhance AI models’ in-context learning, their performance were improved; and AI models’ performance were even better when few-shot technology and external knowledge were combined [40]. However, when the context information “CFPC exam” were removed from the prompt, it resulted in an improved accuracy [66], probably because AI did not understand the acronym CFPC correctly. Besides, by highlighting errors in AI’s answers through prompt engineering, the AI might arrive at the correct response. However, the studies included in this review did not address the situation of identifying AI mistakes and then re-evaluating its subsequent answers.

When AI gave an incorrect answer, a close look at it could reveal valuable insights into the limitations of AI. From the viewpoint of outcomes induced by wrong answers [28], it could remind medical users to always keep in mind the importance of human oversight and critical evaluation. From the viewpoint of thinking process to identity where and why the AI’s reasoning went wrong [12], 18], 41], it could enhance our understanding of the difference between human judgment and AI reckoning.

Using AI to generate medical exam questions could save medical educators’ time [60]. To ensure the quality of AI-generated questions, it is crucial to carefully craft the prompts as well as critically review the generated questions. Clarity of the questions is not a problem [54], 58], 60], 62], but appropriateness can be an issue. Medical educators often instructed AI to generate questions in specific field or topic [53], [54], [55, [57], [58], [59, 62], 63], rather than aligning them with intended learning objectives [56], 57], 63]. This approach can lead to questions that are correct but not suitable for assessment [58], 61], whereas focusing on learning objectives can ensure the validity of the examination [56], 57], 63]. Thus, instruction to align learning objective in the prompt is key to generate suitable exam questions.

Critically review AI-generated questions with predefined criteria before putting the questions in an exam is a good practice [54], [55], [56], [57], [58], [59], [60], [61], [62], [63]. Although these indexes that measure question quality seemed different, the key measures should focus on fact correctness and alignment with learning objectives. For MCQs, possibility of the options is also a key measure. It could ensure that the questions are not only correct and relevant but also effectively measure the intended competencies and knowledge areas. Check the AI-generated questions in an exam with difficulty level and discrimination index [57], 58], 63] could help identify questions that are either too easy or too hard, as well as those that do not effectively differentiate between high and low performers. This analysis can lead to decision on whether and how to use these questions in future assessment.

The relationship between AI’s ability to answer questions and generate them is an interesting yet under-explored area, but direct evidence on this topic is scarce right now. Previous studies have shown that students who engaged in question generation activities tended to have better academic performance [52], [74], [75], [76], suggesting that generating questions can enhance learning. This implies that strong performance in answering questions may be linked to the ability to generate high-quality questions. However, since AI models like ChatGPT have been trained on vast datasets and do not “learn” from the process of generating questions, their ability to generate and answer questions is likely correlated to the quality of the specific knowledge embedded in those datasets.

Implications of AI in medical education

Medical educators can employ AI to verify whether the questions created by humans for examination contain any ambiguities or errors [10], 53]. As mistakes are sometimes discovered after formal exams [10], 15], 24], it’s beneficial to have AI check the quality of the questions, while ensuring that the exam questions are not leaked. By using AI to answer these questions and asking it to explain its reasoning, medical educators can quickly spot potential issues in the exam questions.

AI can serve as a tool for medical students in preparing for exams. While some argue that AI tools are not yet perfect in accuracy and thus cannot be considered as learning tools [4], 54], we, along with some researchers [16], 22], 77], hold a different perspective. For medical students or residents taking licensing exams or specialty exams, the passing score is typically around 60 %–70 % [13], 17], 24], 26], 29], 35], 38], and they are not required to achieve a very high accuracy rate. They can use AI as a peer to assist in exam preparation. As they are not beginners, they should have developed medical thinking and reasoning skills, enabling them to judge the quality of AI response, especially when AI provides explanations for its answers. Although AI’s accuracy is not top-notch, this can also be an advantage, as it forces users to maintain critical thinking rather than relying on AI blindly. If a medical student detects an AI error, pointing out its mistakes might sometimes lead to the correct answer. This kind of human-AI collaboration is happening in the real workplace. The studies included in this review did not mention the scenario of pointing out AI errors and then looking at its answers again. However, for beginners who are just learning the new knowledge, AI is not an ideal authoritative source for learning [6], as they lack comprehensive judgment capabilities.

With well-crafted prompts, AI can efficiently produce high-quality examination questions [78]. Clear requirements, providing context, alignment with learning objectives, describing clinical scenarios and the provision of examples [79], [80], [81] are all good practices for ensuring the quality of the questions. Additionally, specifying output formats can significantly reduce the workload of editing. Medical educators should learn about prompt engineering or follow guidelines for crafting prompts to create excellent ones [78], [79], [80], [81]. Of course, due to the risk of hallucination, human review of AI-generated questions is always essential [61].

The knowledge-based questions generated by AI can serve as an effective tool for formative assessment in the classroom [57], 59], 63]. Medical teachers can use the questions to gain real-time insights into students’ mastery of previous knowledge and progress in learning new concepts, thereby adjust teaching content and pace if necessary. AI-generated medical cases can be used as material for classroom discussions [59], fostering students’ clinical judgment and decision-making skills. Additionally, educators can also teach students on how to utilize AI for creating questions, thus, students can use AI for self-assessment to check their understanding of the knowledge, thus support personalized learning [82].

The strong capabilities of AI in answering and creating medical exam questions undoubtedly challenge the traditional modes of examination [17], 22], 26], 38], 83]. In the near future, medical exams may more closely mirror real-life medical practice. For instance, it could involve simulating scenarios where patients describe their physical discomfort to doctors, with these descriptions potentially being ambiguous or conflicting. Doctors need to make preliminary judgments and gather key information for decision-making through questioning, laboratory tests, and other methods, continuously adjusting and refining their decisions based on new information. Accordingly, exam questions could be presented in a step-by-step adaptive manner to simulate the actual diagnostic and treatment process. Current clinical case-based questions, though seemingly complex, essentially provide necessary and consistent information in advance, subtly offering exam-takers clues to find the answers.

As AI technology continues to break new ground, the capabilities of AI are becoming even powerful. It is transforming the way we teach and learn. Educators should always maintain a vigilant and cautious approach when utilizing AI in teaching, to ensure that AI tools are used responsibly and ethically to enhance student learning experiences without compromising the integrity of the educational process [17], 25], 58], 78].