Large language models for dermatological image interpretation – a comparative study

Introduction

Artificial Intelligence (AI) has become a transformative force in medicine, offering innovative solutions to enhance diagnostic accuracy, streamline workflows, and improve patient outcomes [1], [2], [3]. In contrast to traditional models, multimodal large language models (LLMs) can process and integrate a range of data types, including images, text, and structured information. The promise of multimodal LLMs is their ability to address a range of diverse diagnostic challenges across specialties [4], 5] using nearly any given data source.

Dermatology offers a unique testing ground for multimodal LLMs, as visual features are central to diagnosis. Even experienced dermatologists are experimenting to incorporate LLMs into their diagnostic processes [6]. Concurrently, the field of dermatology presents a challenging environment for AI due to the presence of ambiguous presentations, overlapping features, and diverse skin tones. The application of advanced machine learning algorithms and deep learning models has yielded impressive results in tasks such as melanoma detection and lesion classification [7], [8], [9]. Nevertheless, a first comparative LLM study analyzing dermoscopic images for melanoma detection yielded an accuracy of only roughly 50 % [10]. Despite the great potential and increasing usage, there is still a paucity of knowledge regarding the performance of general purpose multimodal LLMs in dermatology. This study aimed to compare the diagnostic performance of seven multimodal LLMs for four common dermatological conditions.

Materials and methods

This study focused on four common dermatological diagnoses, namely, psoriasis, vitiligo, erysipelas and rosacea. Images from two publicly available verified dermatology datasets were used in this study. Atlas Dermatológico, is a comprehensive online database containing over 12,000 freely available images representing a wide range of skin diseases as of December 2024 [11]. The second dataset, DermIS.net, is described as the most comprehensive dermatology information service on the internet [12]. These datasets contain images labeled and categorized by dermatology experts and have been widely referenced in dermatological research, providing a diverse set of real-world skin disease images. While we did not conduct an independent review of image representativeness with board-certified dermatologists, we relied on the expert-curated nature of these databases to ensure data quality. All images associated with the respective skin diseases available on 01 December 2024 were used. No images were manually excluded or reviewed prior to analysis to ensure an unbiased sample. The final dataset contained a total of 500 images covering the four different skin diseases. The distribution of images per disease is shown in Table 1.

Experimental procedure

For five of the models (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 1.5 Flash and Claude 3.5 Sonnet), each image was sequentially input using their respective API interfaces. For Meta’s Llama3.2 models, which were deployed locally, images were sequentially processed using Ollama (Version v0.5.1) as the interface. A new session was created for each image to ensure that the models did not retain information from previous interactions that could influence their responses. To ensure comparability and reduce bias the same standardized prompt was used for all models Figure 1. LLMs were prompted to generate the most likely diagnosis based on the image provided. Cases where models refused to provide a diagnosis or where API requests were blocked were rated as incorrect classifications. No API failures were noted during the study period. LLM requests were made between 07 December 2024 and 14 December 2024.

Figure 1:

Standardized system and user prompts.

Results

Overall model accuracies

The evaluation revealed notable differences in model performance, see Table 2 and Figure 2. The highest overall accuracy was achieved by GPT-4o, with a score of 67.8 %, followed by GPT-4o mini (63.8 %), and Llama3.2 11B (61.4 %). The larger Llama3.2 90B model exhibited an inferior performance compared to its smaller counterpart, achieving only 50.8 %. This can be attributed, at least in part, to its tendency to frequently refuse to diagnose certain images (91, 18.2 %). Gemini 1.5 Flash demonstrated the lowest overall accuracy with 37.0 %.

Table 2:

Accuracy of LLMs in % across multiple diseases.

Disease, n	GPT-4o	GPT-4o Mini	Gemini 1.5 pro	Gemini 1.5 Flash	Claude 3.5 Sonnet	Llama 3.2 90B	Llama 3.2 11B
Psoriasis (254)	69.69	80.31	45.67	39.37	53.15	48.43	77.56
Vitiligo (108)	78.70	62.96	55.55	26.85	57.41	61.11	63.89
Erysipelasa (85)	49.41	12.94	31.76	36.47	22.35	44.71	36.47
Rosacea (53)	66.04	67.92	56.60	47.17	54.72	50.94	18.87
Overall accuracyb	67.80	63.80	46.60	37.00	49.00	50.80	61.40

1. Bold values indicate the highest accuracy achieved for each dermatological condition among the tested models. ^aCellulitis was also rated as correct. ^bWeighted accuracy.

Figure 2:

Diagnostic accuracy of respective large language models according to respective dermatological diseases.

Patterns in misdiagnosed images

Analysis of the misdiagnosed images revealed notable patterns (Table 3). A total of 11.0 % (55/500) images were misclassified by all models, highlighting the challenges of interpreting ambiguous or visually overlapping features. In contrast, 11.6 % (58/500) images were correctly classified by all models, characterized by clear and distinctive disease-specific features, such as the sharply demarcated erythematous plaques in psoriasis. Visual examples of three vitiligo images correctly classified by all models and three images misclassified by all models are provided in Figure 3 to illustrate these patterns.

Table 3:

Classification results by disease and overall.

Disease	Total images, n (%)	All LLMs are false, n (%)	≥5 LLMs are false, n (%)	All LLMs are right, n (%)	≥5 LLMs are right, n (%)
Psoriasis	254 (100.0)	15 (5.9)	66 (26.0)	39 (15.4)	123 (48.4)
Vitiligo	108 (100.0)	10 (9.3)	27 (25.0)	14 (13.0)	52 (48.1)
Erysipelas	85 (100.0)	20 (23.5)	48 (56.5)	3 (3.5)	14 (16.5)
Rosacea	53 (100.0)	10 (18.9)	23 (43.4)	2 (3.8)	13 (24.5)
Overall	500 (100.0)	55 (11.0)	164 (32.8)	58 (11.6)	202 (40.4)

Figure 3:

Examples of correctly and incorrectly classified images. The top row shows three images (A–C) that were correctly classified as vitiligo by all seven LLMs, whereas the bottom row shows three images (D–F) that could not be correctly diagnosed as vitiligo by any of the LLMs. Image sources: www.dermis.net and www.atlasdermatologico.com.br.

Discussion

This study assessed the performance of LLMs in interpreting images of common dermatological conditions. The results of this study demonstrate a relatively high diagnostic accuracy of certain models even without domain-specific further training. The strong performance of GPT-4o, which achieved the highest overall accuracy, highlights the potential of general-purpose multimodal models for visually distinct conditions such as vitiligo and psoriasis. The competitive performance of Llama3.2 11B, despite its smaller size, free availability, and local deployment capabilities, highlights the potential of medium-sized, privacy-preserving models for diagnostic decision support. However, the limitations observed highlight the need for significant improvements before such systems can be reliably integrated into clinical workflows. SkinGPT-4 exemplifies a privacy-preserving LLM trained using a two-step strategy [14]. The model was rigorously tested on data from 150 real-world patients, achieving a diagnostic accuracy or relevance rate of 80.6 % as evaluated by board-certified dermatologists. The variability in accuracy between conditions highlights the inherent complexity of dermatological diagnosis. Conditions with high-quality images and clear and well-defined visual features, such as vitiligo, were generally well-recognized by most models. In contrast, conditions such as erysipelas, which are characterized by overlapping and more subtle features, posed a greater challenge.

This discrepancy highlights a key limitation of image-only diagnosis, where contextual clinical information often plays a critical role in accurate decision-making. Incorporating multimodal data streams, including demographic variables, medical history and laboratory findings, could enable these models to align visual patterns with contextual cues, moving beyond pattern recognition towards more nuanced and clinically relevant analysis. Future research should prioritize the integration of such multimodal approaches to bridge the observed performance gaps. In addition, direct comparisons with board-certified dermatologists could provide valuable context for assessing LLM performance. Although beyond the scope of the current study, expert benchmarks would help determine whether models truly add diagnostic value or simply reflect patterns already present in publicly available datasets. Establishing such comparisons in future research could clarify the strengths and limitations of LLM-based dermatological diagnosis.

The interpretability of LLM-based diagnostics remains a pressing concern. While models such as GPT-4o and Llama3.2 11B have demonstrated substantial diagnostic accuracy, their decision-making processes remain inherently opaque. This lack of transparency may hinder clinical uptake and buy-in, as understanding the rationale behind a diagnosis is crucial for trust and validation in medical practice. The provision of interpretable output, such as visual explanations highlighting pathologies could address this issue [15]. Improved interpretability would not only promote confidence among clinicians but also support regulatory approval processes by ensuring that diagnostic decisions are explainable, reproducible and ethically sound. Data-related limitations also require careful consideration [16]. The use of publicly available datasets carries the risk of geographical bias and under-representation of different skin colors and conditions. In addition, the lack of histopathological confirmation and unclear labelling criteria may compromise the validity of the ground truth, making it difficult to distinguish between model error and dataset imperfections.

The integration of LLMs into diagnostic workflows not only poses significant regulatory challenges but also necessitates rigorous clinical validation. Under the European Union’s Medical Device Regulation, AI systems intended for diagnostic purposes are classified as medical devices, requiring stringent validation to ensure safety and efficacy. Additionally, the recently passed EU Artificial Intelligence Act categorizes healthcare AI systems as high-risk, mandating compliance with strict performance, transparency, and ethical standards [17]. Beyond regulatory requirements, these systems must undergo robust clinical evaluation through preclinical and early clinical validation studies, including clinical simulation settings and/or Early Feasibility Studies [18], 19]. These studies provide critical insights into the safety, effectiveness, and real-world applicability of LLMs, identifying potential risks and limitations early in the development process. Combining regulatory compliance with phased clinical validation will be essential for achieving both trust and widespread adoption in clinical practice.

A main limitation of this study is that tested LLM models likely had access to parts of the publicly available image datasets. This familiarity could artificially inflate diagnostic accuracy, as the models may have been trained or fine-tuned on these datasets or closely related ones. Consequently, the results may not fully reflect the models’ true capabilities when applied to unseen, real-world clinical data. To address this, future research should prioritize the use of private, independently curated datasets that are not accessible during model training. Incorporating real-world clinical images from diverse patient populations would enhance the robustness and generalizability of findings. This approach would also mitigate concerns of data leakage, ensuring that model performance more accurately reflects its diagnostic potential in practice.

Another limitation of this study is its focus on only four dermatological conditions. While these conditions represent a spectrum of diagnostic complexity, they do not encompass the full range of dermatological presentations. Consequently, the findings provide limited insight into the models’ capabilities across the broader spectrum of skin diseases, including rarer or more nuanced conditions. Future studies should aim to include a wider variety of dermatological diagnoses to ensure a more comprehensive evaluation of LLM performance. This broader analysis would help establish whether the observed diagnostic trends hold across diverse and less well-defined conditions.

Nevertheless, we believe comparative benchmarking studies are crucial and can guide decisions on which models to choose for which purposes. Given the rapid iteration of large language models, newer versions with potentially improved capabilities are frequently released. While this study assessed models available at the time of testing, future research should continue to apply systematic benchmarking to newer models to track progress and evaluate whether performance improvements extend to dermatological image interpretation. To ensure meaningful comparisons over time, it is equally important that these evaluations are conducted on high-quality, diverse datasets. Therefore, future studies should incorporate private, independently curated datasets that include a variety of skin tones, conditions, and clinical contexts to ensure robust and generalizable model evaluations. Additionally, it seems crucial to evaluate the actual clinical impact of model usage.

Conclusions

This study demonstrates that multimodal LLMs can effectively identify key diagnostic features in dermatological images, even in the absence of domain-specific training. Their ability to recognize conditions with distinctive visual patterns, such as vitiligo and psoriasis, highlights their potential for wider clinical applications. In particular, the comparable performance of locally deployed models such as Llama3.2 11B to large, cloud-based solutions highlights important implications for privacy and scalability in medical settings. Further studies are warranted investigating how to best integrate multimodal LLMs into clinical workflows.