Comparative analysis of diagnostic accuracy in endodontic assessments: dental students vs. artificial intelligence

Introduction

The 2022 release of OpenAI’s ChatGPT generated widespread interest for its human-like responses. It raised anticipation similar to transformative technologies like mobile devices and the World Wide Web. However, concerns about its implications for human capabilities, employment, ethics, and regulations emerged early on. Despite uncertainties, ChatGPT’s potential as a diagnostic tool in dental education sparked optimism and scrutiny.

Artificial intelligence (AI) refers to the ability of an integrated platform to obtain, process, and implement skills and knowledge acquired through education or experience that are usually linked to human intelligence. It is made up of a neural network architecture that mimics human thinking and has been applied in dentistry to diagnose dental diseases, plan treatments, make clinical decisions, and predict prognoses. There are two main types of AI in healthcare: machine learning and deep learning. Machine learning requires significant human intervention for identifying and selecting relevant features from data, a process crucial for the algorithm’s learning and prediction accuracy. In contrast, deep learning minimizes this need by autonomously discovering patterns and features directly from large datasets, though it still necessitates human expertise in crafting network architectures and managing extensive training data [1].

In recent years, AI has emerged as a powerful tool in healthcare, potentially revolutionizing clinical practice and improving patient outcomes. One area where it has shown significant promise is in aiding clinicians in making accurate diagnoses. AI algorithms can analyse large amounts of medical data, such as medical images, patient records, and laboratory results, and provide insights that can help clinicians make more informed decisions [2].

Developing diagnostic acumen is a crucial element of a dental student’s training, enabling them to transition into competent practitioners. This development necessitates exposure to a wide array of clinical cases, constructive feedback from faculty, and self-evaluation exercises. In recent years, the healthcare industry, including dentistry, has been experiencing significant technological advancements. AI, which has demonstrated its efficacy in improving accuracy, reducing costs, and minimizing medical errors, is emerging as a transformative force within contemporary dentistry. Its role extends beyond that of a simple tool; it is becoming an integral part of an advanced technological ecosystem that is shaping the future of dental practice. Notably, the trend towards integrating AI into healthcare and dentistry is supported by substantial investments from both the private and public sectors [3, 4].

In the medical field, AI has demonstrated the potential to assist clinicians in making more accurate diagnoses. For example, one study found that AI-based algorithms could accurately detect and diagnose diabetic retinopathy, a common cause of blindness, with sensitivity and specificity comparable to or better than human experts [5]. In medical imaging, AI algorithms have been shown to accurately identify and diagnose various medical conditions, including skin cancer [6], lung cancer [7], and breast cancer [8], with high sensitivity and specificity. Additionally, AI can enhance clinical workflows, reduce diagnostic errors, and improve patient outcomes by managing medical data, identifying high-risk patients for personalized care, and supporting student training in history-taking skills [9], [10], [11]. It also advances medico-dental diagnostics through the detection and classification of pathologies, rating attractiveness, and predicting age and gender; deep learning models specifically excel in identifying abnormalities in medical images and have significantly improved dentists’ diagnostic accuracy, sensitivity, and specificity [12], [13], [14].

Dental radiology benefits significantly from AI, automating image reading and enabling early diagnosis across various areas, such as dental emergencies and prosthetic planning. Through learning from experience and adapting to new data, AI improves the standardization process quality [4, 15]. Deep learning models diagnose conditions like caries, bone loss, periapical lesions, malocclusion, and anomalies radiographically [16], [17], [18], [19], [20], [21], [22]. Studies highlight AI’s efficacy in enhancing dental records by accurately assigning tooth numbers on radiographs, reducing processing time [13], identifying cysts and tumours from panoramic radiographs [23], [24], [25], and detecting proximal caries with reliability, though without depth differentiation [20].

AI has shown potential in revolutionizing dentistry, providing early and accurate detection of dental caries and addressing the limitations of traditional diagnostic methods [23, 26]. These AI-based models have been found to be highly accurate, cost-effective, and reduce false negatives, aiding dentists in clinical practice and assisting non-dental professionals in detecting dental caries [3, 27].

In dental education, virtual patient simulations play a critical role, particularly during times of shortage of real patients. The integration of AI in virtual patient simulations can enhance the realism of the experience and provide an interactive platform for students to practice on a range of clinical cases. AI-powered chatbots have been shown to be beneficial in dental education, providing students with personalized learning experiences while improving patient safety [28].

AI has also shown potential in diagnosing orofacial pain and Temporomandibular Joint Disorders (TMDs). Research has demonstrated the possibility of creating an AI-based system to support non-specialists in early TMD detection, potentially leading to faster referrals to specialized medical centres [29, 30]. The proposed system is intended to assist general dentists in making a primary diagnosis, avoiding suboptimal recognition and management in interdisciplinary treatment.

AI has been proposed for evaluating cellular and structural atypia of oral squamous cell carcinoma (SCC), offering a more advanced diagnosis and making it a suitable method for developing AI in this field [31]. In implantology, researchers have developed a method for the accurate detection of dental implants and peri-implant tissue using a deep learning model [32]. AI utilizing Convolutional Neural Network (CNN) has demonstrated remarkable accuracy in the computer-assisted classification of odontogenic cysts in digital images of histological tissue sections [33].

In endodontics, various applications of AI have been demonstrated, including the diagnosis of root canal pathology, the determination of working length measurements, and the prediction of retreatment success with very high accuracy [34, 35]. The use of AI in this field has the potential to perform diagnostic and prognostic predictions with decision-making ability, improving dental care. This can be particularly valuable in areas with a lack of highly qualified personnel.

Furthermore, AI allows for the integration of multiple and heterogeneous information domains, such as dental history, clinical data, and sociodemographic information. AI has been used to diagnose periapical diseases, discover root fractures, evaluate root morphology, determine working length, and trace the apical opening, as well as predict retreatment outcomes [35]. A systematic review found that AI models can be a valuable asset for dentists with less experience and non-specialists, providing them with a more accurate and reliable diagnostic tool to support their decision-making [34].

In a qualitative study, several enablers were identified to encourage the implementation of AI in dental diagnostics [36]. One of the main enablers was the ability to generate an automated diagnostic report after interaction with AI findings, saving time for reporting. Another enabler was the expectation that AI would increase diagnostic accuracy. The use of AI as a second independent opinion was also seen as a way to increase trust in the diagnosis made by the dentist.

While dentistry still lacks a standard level of quality and current diagnostic techniques can be dependent on the operator’s subjective perception and prone to errors, AI can help provide a unique treatment plan for each patient, from planning to treatment and follow-up. AI can learn automatically and improve its performance over time, helping clinicians improve their diagnoses, save time, and increase profitability. The development of AI-based diagnostic techniques can help overcome the limitations of manual methods. However, it is important to balance the use of AI with intelligence, objectivity, and common sense and have an appropriate learning curve.

The objective of this study is to compare the diagnostic accuracy of dental students with that of artificial intelligence, specifically a modified ChatGPT 4, in endodontic assessments, including pulpal and apical diagnoses. By doing so, this research aims to shed light on the potential of AI to serve as a reference tool that may enhance the diagnostic process within dental education. The findings are expected to contribute valuable insights into the role of AI technologies in augmenting educational outcomes and guiding future investigations in this evolving field.

Materials and methods

The study involved two distinct groups of dental students. The first group consisted of 54 junior students, defined as third-year students who had just begun their clinical exposure by treating patients. The second group comprised 55 senior students, defined as fifth-year students who are nearing graduation. To aid in the diagnosis, junior students were provided with a sheet listing terminology definitions according to the American Association of Endodontists (AAE) [37], while senior students did not receive this aid and had to rely on their accumulated knowledge.

The AI that was used is a modified ChatGPT 4 (15 Mar 2023 version) that was trained with the pulpal and apical terminology definitions according to the AEE. This modification involved fine-tuning the model with a curated dataset comprising pulpal and apical terminology definitions, case scenarios and diagnostic criteria as defined by the AEE. The fine-tuning process utilized a supervised learning approach, where the model was trained to identify and apply these specialized dental terms in the context of diagnosing pulpal and apical conditions accurately. This tailored training aimed to enhance the model’s proficiency in dental diagnostics beyond its general capabilities.

Each student was tasked with diagnosing seven distinct clinical scenarios that emulated real-life patient cases. These scenarios were presented in a uniform manner. Students made pulpal and apical diagnoses for each case without the assistance of the AI tool. The investigators then used ChatGPT to diagnose the same seven clinical scenarios 60 times. The students’ diagnoses were compared to the ChatGPT diagnoses and a gold standard diagnosis determined by a panel of experienced endodontists.

Furthermore, to explore students’ self-awareness of their diagnostic performance, both senior and junior students were asked to predict the number of mistakes they believed they made prior to receiving feedback on their actual performance. To facilitate this, a predefined scale was introduced for the prediction task, where ‘0 mistakes’ corresponded to a score of 1, ‘1–3 mistakes’ to a score of 2, ‘4–6 mistakes’ to a score of 3, ‘7–9 mistakes’ to a score of 4, and ‘more than 10 mistakes’ to a score of 5.

The primary outcome measure was the accuracy of the pulpal and apical diagnoses made by the students. This was calculated as the number of correctly diagnosed cases divided by the total number of cases. The secondary outcome measure was the confidence level of the diagnosis with and without the use of ChatGPT.

The data collected were analysed using descriptive statistics and inferential statistics (SPSS software, Version 26, IBM Corp., Armonk, NY, USA), including non-parametric one-way ANOVA (Kruskal-Wallis test) to compare the medians of the three groups (senior students, junior students, and ChatGPT). In cases where the Kruskal-Wallis test revealed significant differences, Dwass-Steel-Critchlow-Fligner pairwise comparisons were performed as a post-hoc test to identify which specific groups had significant differences in their medians.

Results

In this study, we compared the diagnostic accuracy of pulpal and apical diagnoses between senior students, junior students, and the ChatGPT tool. The results showed that ChatGPT had a higher mean accuracy (99.0 %) compared to senior students (79.7 %) and junior students (77.0 %). Similarly, ChatGPT had a higher median accuracy (100.0 %) than senior students (85.7 %) and junior students (82.1 %). When looking at the spread of the data, ChatGPT had a much lower standard deviation (0.0254) than senior students (0.122) and junior students (0.155), indicating greater consistency in ChatGPT’s performance (Table 1). The interquartile range (IQR) showed that ChatGPT had the narrowest spread (0.00) compared to senior students (0.0714) and junior students (0.214) (Figure 1). The Shapiro-Wilk test revealed that the data for all three groups were not normally distributed (p<0.001).

Figure 1:

Box plot of diagnostic accuracy for senior students, junior students, and ChatGPT. The boxes represent the interquartile range with the median line dividing the boxes. Whiskers extend to the most extreme non-outlier values, and points labelled with numbers indicate outliers within the data. The y-axis represents diagnostic accuracy as a proportion of correctly diagnosed cases, with values ranging from 0 to 1.0, where 1.0 signifies 100 % accuracy. ChatGPT demonstrates higher median accuracy and a narrower IQR compared to both senior and junior students.

The Kruskal-Wallis test reveals a statistically significant difference among the medians of the three groups being compared (χ²=107, df=2, p<0.001), with a large effect size (ε²) of 0.655. the Dwass-Steel-Critchlow-Fligner pairwise comparisons reveal no significant difference between the senior and junior groups (p=0.829), but both the senior and junior groups have statistically significant differences in their medians when compared to the ChatGPT group (p<0.001). Overall, these findings suggest that ChatGPT outperformed both senior and junior students in terms of diagnostic accuracy, consistency, and data spread.

The perceived difficulty level of the clinical scenarios, as reported by both junior and senior students, was moderately challenging on average. Senior students reported a mean difficulty level of 2.76 out of 5 (SD 0.60), while junior students reported a slightly higher mean difficulty level of 2.83 out of 5 (SD 0.94).

Additionally, both groups predicted their number of mistakes using a predefined scale (0 mistakes=1, 1–3 mistakes=2, 4–6 mistakes=3, 7–9 mistakes=4, and more than 10 mistakes=5). Senior students’ mean prediction was 2.49 (SD 0.71), indicating they expected to make between one and three mistakes on average. Similarly, junior students’ mean prediction was 2.44 (SD 1.17), also suggesting they anticipated making between one and three mistakes on average.

These findings imply that both junior and senior students perceived the clinical scenarios as moderately challenging and anticipated making a relatively small number of mistakes in their diagnoses. However, the junior students’ responses exhibited a higher degree of variability, as shown by the larger standard deviations for both perceived difficulty level and predicted number of mistakes.

Discussion

Our investigation reveals that the application of AI in dental education, particularly through a specifically adapted version of ChatGPT 4, notably surpasses the diagnostic accuracy of both senior and junior dental students in endodontic assessments. This outcome suggests that integrating AI into dental curricula may have the potential to enhance students’ diagnostic capabilities. More than just boosting diagnostic precision, this integration promises to reinforce students’ confidence and consistency in clinical evaluations, which are essential components of clinical decision-making.

Senior students’ slightly higher accuracy compared to junior students can be attributed to their educational and clinical experience. Senior students typically spend more time in dental school and are exposed to more clinical cases, which may have given them a better understanding of pulpal and apical diagnoses. However, the small difference in accuracy between the two groups suggests that, at least in the context of the clinical scenarios presented in this study, both junior and senior students have a comparable level of competence in diagnosing pulpal and apical conditions. The small difference in accuracy may also indicate that junior students’ foundational knowledge and skills are sufficient for them to perform adequately in this particular diagnostic task.

The higher degree of variability in junior students’ perceived difficulty level and predicted number of mistakes, as evidenced by larger standard deviations, is worth noting. This variation may reflect differences in individual junior students’ exposure to clinical cases, confidence level, or mastery of relevant diagnostic terminology and concepts. While senior students performed slightly better in this study, the small difference between the two groups suggests that both junior and senior students have a comparable level of diagnostic competence. The higher variability among junior students, on the other hand, emphasises the importance of continuing education and clinical exposure to improve diagnostic skills and confidence.

ChatGPT performed significantly better in pulpal and apical diagnoses than both junior and senior students. This outstanding performance demonstrates ChatGPT’s potential value as an aid for dental students seeking more accurate diagnoses. Integrating ChatGPT into educational and clinical settings could provide several benefits, including serving as a supportive resource for students, encouraging active learning and critical thinking, and aiding in the standardisation of the diagnostic process across various levels of education and experience.

Harvard Business Review’s article cited Nobel Prize Winner Daniel Kahneman, who pointed out deep flaws within human reasoning, betting on the superiority of AI’s “reasoning” with the human capability to adjust being at the core of this problem [38]. Concerns have been raised, however, about the potential overreliance on AI tools such as ChatGPT, which could impede the development of critical analytical and decision-making skills and inadvertently lead to a false sense of confidence among dental students and recent graduates [39]. To alleviate these concerns, it is critical to emphasise the importance of using AI tools in a balanced manner, positioning them as supplementary resources that supplement, rather than replace, critical thinking and clinical judgement.

The current study’s findings contrast with those of Huh’s (2023) study [40], which compared ChatGPT’s performance in a parasitology examination to that of medical students. The disparities in results can be attributed to a variety of factors, including differences in subject matter, examination formats, and the specific versions or modifications of ChatGPT used in each study.

The findings of our study are consistent with broader trends and conclusions highlighted in the literature over the last two decades, which investigate clinical and educational aspects of dentistry in relation to practical applications of AI [34], [35], [36]. There is a pressing need for dental curricula to be updated to include AI applications, ensuring that both educators and students are adequately prepared to use these technologies in their practice [41, 42].

However, it is critical to highlight the ethical and legal implications of AI integration in dental education, emphasising the importance of reaching an agreement on the safe and responsible use of these tools. While our study demonstrates the potential benefits of using ChatGPT as an assistive tool in dental diagnosis, it also emphasises the importance of taking a cautious and balanced approach to incorporating AI in dental education and practice [43], ensuring that its potential is fully realised without jeopardising students’ clinical reasoning abilities or any ethical or legal considerations.

While the absence of dental X-rays in the clinical scenarios is a limitation, it is important to note that the improvement in diagnostic accuracy observed with ChatGPT remains meaningful and clinically relevant. Despite the lack of dental radiographs, ChatGPT demonstrated a significant improvement in diagnostic accuracy when compared to both senior and junior students. This suggests that AI-based tools like ChatGPT have the potential to improve clinical decision-making and patient outcomes even when diagnostic information is limited. The incorporation of dental radiographs into future research could provide more robust evidence supporting the efficacy of AI tools in improving diagnostic precision in dentistry. This will be especially apparent with the upcoming release of the multimodal ChatGPT, which will be able to interpret and analyse images submitted to it [44]. Moreover, the research was conducted in a single dental school, which restricts the generalizability of the results. Further studies involving multiple dental schools and diverse populations are necessary to validate and expand upon our findings in order to confirm the broader applicability of ChatGPT as an effective diagnostic aid in dental education.

Conclusions

Our investigation contributes to the growing body of knowledge regarding the application of AI in dental education, emphasizing diagnostic accuracy. The study demonstrates that AI, through a specifically adapted version of ChatGPT 4, surpasses both senior and junior dental students in endodontic diagnosis accuracy. The standout diagnostic performance of ChatGPT highlights its value as a significant educational tool. By incorporating ChatGPT into dental curricula, students could greatly improve their diagnostic skills, leading to increased accuracy, confidence, and consistency in clinical evaluations, which, in turn, may positively influence patient care outcomes.

However, it is crucial to consider ChatGPT and similar AI technologies as supplements to traditional educational content. They should serve to enrich the critical thinking and clinical judgement skills developed through conventional training. Embracing AI in this supportive capacity is expected to elevate diagnostic proficiency and contribute to reducing diagnostic errors, ultimately improving healthcare delivery. Future research should delve into the practical, ethical, and legal aspects of weaving AI tools like ChatGPT into dental education and clinical practice. Identifying optimal ways to integrate these technologies for the benefit of students and patients alike will be key as the healthcare education landscape continues to evolve.