AI agent as a simulated patient for history-taking training in clinical clerkship: an example in stomatology

Introduction

The process of taking a comprehensive patient history remains an indispensable initial step within clinical diagnosis, guiding subsequent investigations. Despite the development of abundant sophisticated diagnostic technologies, the fundamental role of detailed history-taking in ensuring patient welfare is indisputable. Globally, the education of healthcare professionals emphasizes the mastery of medical interviewing skills, which are essential not only for diagnostic accuracy but also for fostering a therapeutic relationship between the clinician and patient [1]. Effective communication is a vital determinant of positive patient outcomes, resulting in its widespread integration into medical education mostly through student-patient or standardized patient interactions in their clinical clerkship.

However, conventional history-taking practices in training programs are with their challenges. First, the reliance on real patients or standardized patients may limit the diversity of clinical scenarios encountered by students, potentially compromising their preparedness to handle complex, real-world situations. The cost of recruiting and maintaining standardized patients makes this problem worse. Second, the pressure to perform well in these interactions may shift the focus toward technical proficiency at the expense of genuine empathy and understanding, thereby affecting the quality of patient care. A balanced educational approach is required, one that combines both simulated and real-world experiences to ensure students develop not only clinical skills but also humanistic qualities in medicine. In simpler terms, there is a need for cost-effective training methods that expose students to a variety of complex clinical situations, while simultaneously cultivating empathy and compassion.

Recent advances in natural language processing have led to the development of large language models (LLM)-powered chatbots, such as ChatGPT, which have proved effective in enhancing patient-physician communication [2]. By simulating diverse clinical scenarios, these chatbots can provide students with realistic interactions, fine-tuning their ability to express empathy and deliver bad news with compassion [3]. Compared to traditional interaction with standardized patients, chatbots as simulated patients have distinct advantages: they eliminate potential risks, are not constrained by time or space, and boast high teaching efficiency that allows for repetition and reversibility. Additionally, their ability to engage students in an interactive manner can significantly enhance learning motivation. Nevertheless, after reviewing literature on the application of LLM-powered chatbots as simulated patients, we found several technical barriers are waiting to be overcome:

1. How can full student participation be ensured within a limited time?

2. How can the creation of a simulated patient be made easy and accessible to educators without requiring extensive technical expertise?

3. How can the effectiveness of chatbot-mediated conversations be evaluated?

This manuscript proposes a practical solution by creating an artificial intelligence (AI) agent that simulates a patient with acute pulpitis for history-taking training within clinical clerkship in stomatology.

Materials and methods

Setting and participants

The course of “Stomatology” is offered in the third academic year for five-year undergraduate students majoring in clinical medicine, with theoretical lectures and clinical clerkship lessons conducted in the same semester. There are six clinical clerkship lessons (content of each lesson is listed in Supplementary Table 1), and we built our trial AI agent for the lesson “Differential Diagnosis and Treatment of Toothache” to test its functionality among students and to explore a unified method for future creation of AI agents for other clinical clerkship lessons.

A total of 126 five-year undergraduate students majoring in clinical medicine from the class of 2022 participated in the study, and were divided into two groups, pre-optimization group and post-optimization group. The AI agent, simulating patient consulting online, facilitates students to practice medical history taking and clinical reasoning skills, which constitutes 30 out of 180 min of the lesson chosen. Figure 1 illustrates how the AI agent is used in the lesson.

Figure 1:

How the AI agent is used in clinical clerkship lessons.

This study includes five steps for the creation and optimization of the AI agent:

1. Create the preliminary version of the AI agent (see Appendix 1);

2. Test the preliminary version of the AI agent in the clinical clerkship lesson, and ask students (pre-optimization group) to complete a digital copy of a Chatbot Usability Questionnaire (CUQ), a 16-question survey that assesses various aspects such as the chatbot’s personality and user experience [4], 5];

3. Analyze the outcome of the CUQ and optimize the AI agent;

4. Test the optimized version of the AI agent in the clinical clerkship lesson, and ask students (post-optimization group) to complete a CUQ;

5. Compare the result with the baseline data from the pre-optimization group to evaluate the overall improvement of the AI agent.

Results

A total of 126 questionnaires were collected from the students with 63 from the pre-optimization group and 63 from the post-optimization group. The CUQ is a 5-score Likert scale questionnaire with 16 questions, ranging from “strongly disagree” to “strongly agree.” The scores of each question from both groups are listed in mean ± standard deviation (SD) format (Table 1). The CUQ score of each group was calculated using a CUQ Calculation Tool (Appendix 2) according to Holmes et al. [6].

Comparison between the pre- and post-optimization groups

The mean score of the post-optimization group was calculated to be 79.3 with an SD value of 5.1 (Table 1), indicating much higher satisfaction with the optimized version (p<0.0001) compared to that of the pre-optimization group. After optimization, the AI agent has improved significantly in every aspect, with statistically significant difference in the mean score of at least one question belonging to the category (Table 1, Table 2, and Figure 2).

Table 2:

Comparison of the Chatbot Usability Questionnaire (CUQ) answers between the pre- and post-optimization groups.

Question no.	Group	Strongly disagree, n	Disagree, n	Neutral, n	Agree, n	Strongly agree, n
Q1a	Pre-optimization	10	18	19	13	3
	Post-optimization	0	2	1	37	23
Q2a	Pre-optimization	0	12	21	15	15
	Post-optimization	17	32	9	3	2
Q3	Pre-optimization	0	5	16	21	21
	Post-optimization	3	4	16	22	18
Q4	Pre-optimization	23	34	6	0	0
	Post-optimization	31	27	4	1	0
Q5a	Pre-optimization	0	6	11	34	12
	Post-optimization	0	0	5	28	30
Q6a	Pre-optimization	0	4	17	20	22
	Post-optimization	0	0	4	24	35
Q7	Pre-optimization	15	38	4	4	2
	Post-optimization	19	40	3	1	0
Q8a	Pre-optimization	0	5	14	23	21
	Post-optimization	0	0	0	26	37
Q9a	Pre-optimization	23	30	10	0	0
	Post-optimization	33	30	0	0	0
Q10a	Pre-optimization	8	19	17	14	5
	Post-optimization	0	18	20	15	10
Q11a	Pre-optimization	9	12	27	6	9
	Post-optimization	16	22	13	12	0
Q12a	Pre-optimization	7	13	29	13	1
	Post-optimization	0	5	16	22	20
Q13a	Pre-optimization	2	7	39	10	5
	Post-optimization	24	25	10	4	0
Q14	Pre-optimization	30	29	4	0	0
	Post-optimization	39	22	2	0	0
Q15	Pre-optimization	0	4	12	26	21
	Post-optimization	0	0	6	36	21
Q16a	Pre-optimization	22	19	12	9	1
	Post-optimization	30	29	4	0	0

1. ^aIndicates significant statistical differences.

Figure 2:

Comparison between the pre- and post-optimization groups. ^*p<0.05, ^**p<0.01, ^***p<0.001, Student’s t-test.

Notably, the chatbot’s personality, as indicated by a leap in scores from 2.7 to 4.2 (p<0.0001), became more realistic and engaging after optimization. This enhancement suggests that students found the chatbot more relatable and enjoyable to interact with after the improvements were made. In terms of user experience, the chatbot’s ease of navigation saw a notable increase, with scores rising from 4.0 to 4.5 (p=0.0009). The ease of use also improved significantly, as reflected by the scores moving from 4.0 to 4.6 (p<0.0001). These results underscore the success of the optimization in making the chatbot more intuitive and user-friendly. Error handling was another area where significant progress was observed. The scores increased from 2.8 to 3.3 (p=0.0386), indicating that the chatbot became better at coping with errors or mistakes. Conversely, the perception of the chatbot being unable to handle errors decreased significantly, with scores dropping from 2.9 to 2.3 (p=0.0086). Onboarding, which is crucial for first-time users, also saw substantial improvements. The chatbot’s ability to explain its scope and purpose well increased from a score of 2.8 to 3.9 (p<0.0001), while the perception of the chatbot giving no indication of its purpose decreased significantly, with scores dropping from 3.1 to 1.9 (p<0.0001).

Table 2 provides a comparative analysis of user agreement levels before and after the optimization. Questions marked with an asterisk (*) indicate significant statistical differences. After optimization, there was a marked increase in the number of users who “strongly agree” with positive statements about the chatbot, particularly in questions 1, 2, 5, 6, 8, 9, 10, 11, 12, and 13. For instance, in question 1, which pertains to the chatbot’s helpfulness, the number of users selected “strongly agree” increased from 3 to 23, a substantial endorsement of the optimization’s success. In contrast, the number of users who “strongly disagree” or “disagree” with negative statements about the chatbot decreased post-optimization. This is evident in questions 4, 9, and 14, where the number of users who selected “strongly disagree” dropped significantly, aligning with the improvements seen in the quantitative data of Table 1.

Discussion

With the integration of generative AI as a tool in education, it shows the potential to transform the delivery and cultivation of human learning [7], [8], [9], [10], [11]. In the context of virtual simulation, it has brought more humanized elements, making the dialogue-based virtual simulation more realistic. The development of LLMs has been rapid, which has not only enhanced the effectiveness of medical education but also facilitated the training of healthcare professionals in a more immersive and interactive environment [11], [12], [13], [14], [15], [16]. In the field of stomatology, although virtual simulation teaching has shown significant results, no open-access virtual simulation digital resources suitable for clinical clerkship training in stomatology have been found [17], 18]. Here, for the first time, we report a modification on the LLM-powered chatbot tailored to diagnostic training in stomatology.

The first question we posed in the introduction pertains to ensuring full student participation within limited time frames. Our optimized AI agent addressed this by providing a realistic and engaging interaction that could be accomplished conveniently by scanning the QR code with each student’s mobile phone, after which the user interface simulated the scenario of a patient consulting his conditions via the internet. With minimal guidance, students could easily start the history taking process. Traditionally, due to the limitations of teaching duration and the large number of students in a single clinical clerkship lesson, it is hard to ensure that each student experience the whole process of taking medical histories [19], 20]. More importantly, unlike a group of students, usually more than 10, surrounding only one real patient (or standardized patient) in traditional clinical clerkship, our solution ensures each student to converse with the AI agent simultaneously. The CUQ scores highlighted the importance of the chatbot’s personality being realistic and engaging. To this end, we modified the AI agent’s profile to adopt a more natural and responsive tone, ensuring that the simulated patient’s reactions and descriptions are expanded appropriately based on the user’s inquiries. This adjustment aligns with the significant increase in scores for the chatbot’s personality after optimization, indicating that users found the AI more engaging after the changes. To address the perception of the chatbot being too robotic, we focused on making the AI’s responses more human-like. The updated profile (see Appendix 3) includes a detailed medical history and rigorous definitions of how the agent responds, which helps in creating a more authentic patient simulation. This change corresponds with the decrease in scores for the chatbot seeming too robotic, as indicated in the CUQ, suggesting that the optimization made the AI feel less mechanical. The CUQ scores pointed out the need for the chatbot to understand users well and be easy to navigate. We optimized the AI agent’s capabilities to provide detailed symptom descriptions and medical history provisions based on the user’s questions. This enhancement ensures that the AI can keep pace with the user’s line of inquiry, mirroring the significant increase in scores for the chatbot’s understanding and ease of navigation. The original CUQ scores indicated that the chatbot was perceived as complex. In our optimization, we focused on simplifying the AI agent’s responses to be more straightforward and relevant, avoiding the revelation of too much content at once. This change resulted in the decrease in scores for the chatbot being complex, suggesting that users found the AI easier to interact with after the optimization.

The second question addresses the accessibility of creating simulated patients for educators. With the development of generative AI and the emergence of AI agent technology, the creation and application of a simulated patient has become much easier [21], 22]. In this study, we chose ChatGLM as the platform to build the AI agent and provided detailed procedure guiding educators to create their own AI agent. We believe this is the most simple and convenient way of doing so, as no coding or programming skills were required in the process. This democratization of AI use in education ensures that a broader range of educators can enrich their teaching methods, enhancing the diversity of clinical scenarios students can encounter.

The third question involves evaluating the effectiveness of chatbot-mediated conversations. By referencing the CUQ scores and aligning the optimization of our AI agent with the identified areas for improvement, we have successfully enhanced the realism, engagement, and educational value of the AI agent. The modifications to the AI agent’s profile, including its role, medical history, capabilities, and precautions, directly address the feedback from the CUQ. These changes have resulted in a more effective simulated patient that can provide a diverse range of clinical scenarios for medical students, fostering both technical proficiency and empathy in a controlled and safe environment. However, in a minority of instances, the simulated patient’s responses exceeded the bounds of the predefined medical history information, a phenomenon similarly observed in the applications of other researchers [23]. In general, CUQ is a reliable tool to evaluate the effectiveness of chatbot-mediated conversations.

While this study shows promising results, several limitations exist. First, the sample size of 126 students, though reasonable for an initial evaluation, is relatively small. To strengthen the findings, future studies should involve larger and more diverse groups of students across different institutions. Second, the AI agent was tested only for acute pulpitis. Expanding its use to include other medical conditions would make it more versatile and practical for broader clinical training. Third, the evaluation relied on student feedback through the CUQ, which, while helpful, doesn’t fully capture how well the AI agent prepares students for real-world patient interactions. Longer-term studies tracking students’ performance in actual clinical settings would provide a clearer picture of its impact.

There are also opportunities to make the AI agent more realistic and engaging. For example, adding features like voice interaction could make conversations feel more natural, while augmented reality (AR) could create immersive environments that mimic real clinical settings. These enhancements would help bridge the gap between simulation and real patient interactions. Finally, it’s important to address potential ethical concerns, such as ensuring the AI agent’s responses are unbiased, culturally sensitive, and medically accurate, so students can trust it as a reliable training tool.

Conclusions

In conclusion, the optimization of our AI agent demonstrates the potential of AI agents to revolutionize medical education. By providing a platform for full student participation, simplifying the creation of simulated patients, and offering a means to evaluate chatbot-mediated conversations, we have taken significant steps toward overcoming the challenges faced by traditional patient history-taking training methods. The AI agent’s ability to simulate diverse clinical scenarios, coupled with its interactive and evaluative capabilities, positions it as a powerful tool for fostering both clinical skills and humanistic qualities in medical students. As we continue to refine and expand the capabilities of AI in medical education, we move closer to a future where technology and empathy coalesce to enhance patient care [23].