LLM-guided multimodal attention network for robust multiclass Parkinson’s disease diagnosis

Machine learning and deep learning in PD diagnosis

Early computational approaches to PD diagnosis relied on traditional machine learning techniques applied to handcrafted features from various modalities. Studies utilizing gait kinematics, voice recordings, handwriting dynamics, and clinical scores demonstrated moderate success in binary classification tasks [9]. However, these methods were constrained by feature selection bias and limited scalability. The shift to deep learning, particularly CNNs, enabled end-to-end learning from raw neuroimaging data, significantly improving performance in detecting PD-related structural changes in MRI and DaT-SPECT scans [10]. Architectures such as ResNet, VGG, and EfficientNet have been adapted for medical imaging, though they primarily operate in a unimodal setting.

Multimodal fusion for neurodegenerative diseases

Recognizing the limitations of unimodal analysis, recent research has focused on multimodal integration for enhanced diagnostic precision. For PD, studies have combined structural and functional MRI with clinical variables, genetic data, and cerebrospinal fluid biomarkers [11]. Fusion strategies range from early (input-level) and late (decision-level) fusion to intermediate (feature-level) fusion using concatenation or tensor-based methods. While these approaches show improved accuracy over single-modality baselines, they often lack mechanisms for modeling deep, semantically aligned interactions between heterogeneous data types.

Attention mechanisms in medical AI

Attention mechanisms have become central to modern deep learning architectures, enabling models to focus on diagnostically relevant regions while suppressing noise. In medical imaging, attention modules such as CBAM have been integrated into CNNs to refine spatial and channel-wise features, improving localization of pathological signatures [12]. Similarly, transformer-based models, including Vision Transformers (ViTs) and their medical adaptations, use self-attention to capture long-range dependencies in image data [13]. However, most attention-based medical models operate within a single modality, leaving cross-modal attention underexplored in clinical applications.

Language models in clinical multimodal systems

The success of large language models in natural language processing has inspired their adoption in multimodal medical AI. Vision-language pretraining (VLP) models, such as CLIP and BiomedCLIP, align image and text representations using contrastive learning, showing promise in tasks like radiology report generation and medical visual question answering [14]. In neurology, LLMs have been used to encode clinical notes and assessment reports, but their integration with neuroimaging for PD staging remains limited. Recent works propose LLM-guided attention or prompting to steer visual feature extraction, yet few designs explicitly couple LLM-based clinical reasoning with spatially aware cross-attention for neurodegenerative disease classification.

Problem statement

Let X i , Y i i = 1 N represent a dataset sampled from a distribution D , where X denotes the image space and Y the corresponding class labels. The objective of supervised deep learning is to learn a mapping function F θ X → Y ˆ such that the predictions Y ˆ approximate the true labels Y. While conventional models have primarily leveraged visual data to optimize classification performance, complementary clinical metadata – such as age, weight, and cognitive scores from the Mini-Mental State Examination (MMSE) [15], 16], Functional Activities Questionnaire (FAQ), and Neuropsychiatric Inventory Questionnaire (NPIQ) – carry substantial diagnostic and prognostic value [17]. Consequently, the effective integration of this metadata with visual features is critical for enhanced classification. In this work, we extend the learning paradigm to a multimodal function F θ X , X t → Y ˆ , where X is the imaging data and X _t is the clinical metadata, with the objective of aligning these heterogeneous modalities for more accurate and robust predictions [18].

Attention-enhanced visual encoder

The visual pathway processes axial slices extracted from 3D structural MRI volumes using a ResNet-18 backbone, chosen for its balance between representational capacity and parameter efficiency [19]. To enhance the model’s ability to focus on neuroanatomically relevant regions, each residual block is augmented with a Convolutional Block Attention Module (CBAM). The CBAM operates sequentially through channel attention and spatial attention submodules.

Given an input feature map X _l at layer l, let F_l,0 be the output of the first convolutional block within the residual layer [20]. The channel attention mechanism first computes inter-channel dependencies using global average pooling and max pooling, followed by a shared multi-layer perceptron (MLP) and sigmoid activation:

The spatially refined feature map is then obtained via a spatial attention module that concatenates channel-pooled features and applies a convolutional layer:

The final refined feature map after CBAM is:

where ⊗ denotes element-wise multiplication. This attention-enhanced encoding ensures that the model prioritizes discriminative regions such as the substantia nigra, basal ganglia, and cortical thinning patterns associated with PD progression.

LLM-based clinical encoder

Clinical metadata are inherently heterogeneous, covering numerical scores, categorical attributes, and free-form clinical notes [21]. We therefore convert each patient’s metadata into a standardized textual prompt using a predefined template, and feed the resulting prompt into a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to obtain a unified, semantically informative clinical representation. For example:

As illustrated in Figure 1, structural MRI slices are first processed by a ResNet-18 visual backbone augmented with Convolutional Block Attention Modules (CBAM) [22] to strengthen channel- and spatial-wise discriminative representations. In parallel, heterogeneous clinical metadata are serialized into structured textual prompts and encoded by an LLM-based clinical encoder (e.g., pre-trained BERT), after which the resulting embeddings are linearly projected into the same latent space as the visual features. The proposed Meta-Guided Cross-Attention (MGCA) module then performs multi-head cross-attention by treating visual features as queries and LLM-derived clinical embeddings as keys and values, thereby injecting clinically meaningful semantics into image representation learning. The fused multimodal representation is finally fed into a classification head to predict the disease stage.

LLM-based clinical encoder

Clinical metadata are inherently diverse, including continuous measurements, categorical descriptors, and free-text clinical notes [23]. To map these heterogeneous inputs into a unified and semantically informative space, we use a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model as the clinical encoder.

where Pool(·) denotes mean pooling over the sequence dimension, and T∈R^D is the final clinical feature vector with dimension D=256, matching the visual feature dimension.

Ensemble feature selection

Prior to encoding, an ensemble feature selection step is applied to identify the most discriminative clinical variables, reducing noise and computational overhead. Five tree-based models – Random Forest, XGBoost, LightGBM, ExtraTrees, and AdaBoost – are trained to rank features by importance [24]. A weighted majority voting scheme aggregates rankings across models, yielding a consensus importance score Sf for each feature f. The top-k features (e.g., UPDRS, MoCA, Age, Hoehn & Yahr, Weight) are retained for subsequent encoding, ensuring that the clinical encoder focuses on variables with the highest predictive relevance.

Meta-guided cross-attention (MGCA) module

The MGCA module [22]serves as the core fusion mechanism, enabling fine-grained interaction between visual features F_img∈R ^{B  × C × H ×  W} and clinical embeddings T∈R ^{B  × D}. To facilitate cross-modal alignment, visual features from the third residual block are spatially averaged to produce a compact visual token V∈R ^{B  × D}. The MGCA employs a multi-head attention mechanism with four heads. For each head h, query (Q _h ), key (K _h ), and value (V _h ) matrices are derived via linear projections:

where W _h Q, W _h K, W _h V∈R^D × d _k are learnable weights and d _k =D/4. Cross-attention is computed as:

Outputs from all heads are concatenated and projected to produce an attention-refined feature vector. A residual connection preserves original visual information:

This design allows the model to dynamically weigh clinical metadata in relation to specific imaging features, enabling semantically grounded fusion.

Classification layer and optimization

The fused feature vector F_fused is passed through a fully connected classification layer with softmax activation to produce probability distributions over the three classes. The model is trained end-to-end using categorical cross-entropy loss:

where y ^c is the ground-truth one-hot label and y ^c is the predicted probability for class c. Optimization is performed using the Adam optimizer with an initial learning rate of 1 × 10⁻³, weight decay of 1 × 10⁻⁵, and dropout (p=0.5) applied to fully connected layers to prevent overfitting. Early stopping is employed based on validation accuracy with a patience of 20 epochs.

Implementation details

The framework is implemented in PyTorch. MRI slices are preprocessed with intensity normalization and augmented via random rotations (±10°), horizontal flips, and mild intensity scaling [25]. Training follows a subject-level 5-fold cross-validation protocol, with 20 % of subjects held out for independent testing. All experiments are conducted on NVIDIA RTX 3090 GPUs, with batch size set to 32. The BERT model used is bert-base-uncased, fine-tuned jointly with the visual backbone. The complete code and data splits are made publicly available to ensure reproducibility.

Dataset and preprocessing

Data & Preprocessing (summary): Structural MRI volumes were skull-stripped and intensity-normalized, then resampled to a consistent voxel spacing and reoriented to a common anatomical convention before axial-slice extraction. For 2D processing, each subject contributes 10 representative axial slices that are resized to a fixed spatial resolution and normalized to zero mean/unit variance per slice. Clinical variables are cleaned and standardized (z-score for continuous measures; one-hot encoding for categorical variables), and then serialized into a natural-language prompt for the clinical encoder.

Label definition: We formulate a 3-class diagnosis task with (i) NC/HC: neurologically normal controls (ii) Prodromal PD: prodromal or at-risk subjects (e.g., RBD/hyposmia); without a formal PD diagnosis; and (iii) PD: clinically diagnosed Parkinson’s disease. These labels are used consistently across all tables, figures (confusion matrices/ROC), and narrative descriptions.

Missing-modality handling: Because LLM-MAN relies on paired MRI and clinical metadata, subjects with missing required inputs are excluded from model training/testing to avoid introducing imputation bias. In practical deployment, the framework can be extended with modality-dropout training or imputation-aware encoders; we discuss this in Limitations.

Performance on primary PPMI dataset

The proposed LLM-Guided Multimodal Attention Network (LLM-MAN) was evaluated on the Parkinson’s Progression Markers Initiative (PPMI) dataset using a rigorous subject-level 5-fold cross-validation protocol followed by independent testing on a held-out set. The model achieved a multiclass classification accuracy of 95.68 % in distinguishing between Normal Control (NC), Prodromal PD, and Diagnosed PD categories. This performance represents a significant improvement over unimodal baselines and establishes a new state-of-the-art for multimodal PD staging. Detailed performance metrics including precision, recall, and F1-scores across all three classes are presented in Table 3.

The overall discriminative performance is further illustrated in Figure 2(a–c), which reports the multiclass ROC and Precision–Recall curves as well as the normalized confusion matrix. Training dynamics and early-stopping behavior are summarized in Figure 2(d), showing stable convergence with minimal overfitting.

Figure 2:

Performance analysis on the PPMI cohort under subject-level 5-fold cross-validation with a held-out test set. (a) One-vs-rest ROC curves for NC, prodromal PD, and PD. (b) Precision–Recall curves. (c) Normalized confusion matrix at the argmax operating point. (d) Training/validation dynamics with early stopping. (e) Ensemble-selected clinical feature importance. (f) Benchmark comparison to prior methods. Overall, LLM-MAN shows strong multiclass discrimination and balanced errors across adjacent stages.

Notably, LLM-MAN demonstrated exceptional sensitivity for the PD class (Recall=95.42 %), a clinically critical attribute given that false negatives in PD diagnosis can lead to delayed intervention and poorer patient outcomes. The model also showed strong performance on the challenging Prodromal PD category (F1-score=94.82 %), highlighting its potential for early detection during the clinically ambiguous prodromal phase.

Comparison with state-of-the-art models

A comprehensive comparison with contemporary multimodal approaches for PD diagnosis is presented in Table 4. LLM-MAN outperforms recent vision-language models like MedBLIP-PD (85.3 % accuracy) and multimodal fusion approaches such as MADDi-PD (96.88 % accuracy on a different multiclass setup). While some hybrid models report higher accuracy on binary classification tasks (e.g., 98.4 % for NC vs. PD), LLM-MAN addresses the more clinically challenging ternary classification problem that includes the subtle prodromal stage – a capability essential for early intervention strategies. A visual benchmark summary against representative baselines is provided in Figure 2(f).

The superior performance of LLM-MAN is particularly notable given the complexity of the ternary classification task. The integration of BERT-based clinical reasoning with attention-enhanced visual features through the novel MGCA module enables more nuanced differentiation between disease stages compared to conventional fusion approaches.

External validation on OASIS-3

To rigorously assess cross-dataset generalization, LLM-MAN was evaluated on the OASIS-3 cohort under identical preprocessing and evaluation protocols. Despite significant heterogeneity in scanner hardware, demographic profiles, and disease prevalence, the model maintained strong predictive performance, achieving an accuracy of 94.10 % with balanced precision, recall, and F1-scores (see Table 4). This result confirms the framework’s robustness beyond its original training distribution and underscores its potential adaptability to variability encountered across diverse clinical environments.

Ablation studies

Qualitative visualization with class activation mapping

To qualitatively assess feature refinement, we employed Class Activation Mapping (CAM). As shown in Figure 3, attention maps demonstrate that the CBAM-enhanced model focuses on anatomically plausible brain regions implicated in PD pathology – particularly the substantia nigra, basal ganglia, and thalamic regions. This interpretable attention mechanism not only supports classification accuracy but enhances clinical trustworthiness by providing neurologically grounded explanations for model decisions.

Figure 3:

Qualitative interpretability examples (PPMI): Class activation maps (CAMs) from the final convolutional block illustrate that adding CBAM yields more localized and anatomically plausible focus patterns compared with the non-CBAM variant. These visualizations support clinician-friendly inspection by highlighting regions that contribute most to the model decision.