Method comparison of diabetes mellitus associated autoantibodies in serum specimens

Study design

This was a prospective study based on data obtained during a test verification; sample collection and testing were performed parallel to routine diagnostics.

Ethics

Human leftover serum samples from routine work at the MVZ Labor Dr. Limbach & Colleagues (Heidelberg, Germany) were anonymized and stored until further use. According to applicable law in Germany, the use of anonymized leftover material for method validation does not require specific donor consent or approval of an Ethics Committee.

Sample collection

Leftover serum specimens were collected, anonymized, aliquoted, and promptly retested on the CLIA devices iFlash and Maglumi. Re-testing of specimens between CLIA devices was randomized. Sample stability as specified by the manufacturer was adhered to and additionally confirmed by retesting the samples on the same device after several days (data not shown).

Eligible for testing were anonymized leftover serum samples from routine diagnostics at the Limbach Laboratory in Heidelberg, Germany, as well as calibrators and standard samples of the respective tests. Patient sample collection was carried out over 2 months after routine testing, anonymized, aliquoted and stored at −20 °C until further use. The collection was carried out in a two-stage process. First stratified random sampling was done for initial representativeness, followed by targeted random sampling within strata for practical efficiency in the second stage. The number of tests varied depending on the parameter and device but generally met the minimum requirements of method comparison guidelines proposed by the Clinical & Laboratory Standards Institute. Measurements of the ZnT8 parameter on the iFlash did not take place, as this parameter had been extensively evaluated on the iFlash the previous year and was found a viable option for adoption.

Measurements

As a comparative method for the CLIA devices (iFlash 1800 from Yhlo (China), Maglumi 800 from Snibe (China)), the ELISA method (EUROLabWorkstation analyzer from EUROIMMUN (Germany)) was utilized as it is the standard routine operation in our laboratory. The CLIA tests were standardized against the ELISA kit produced by RSR by the manufacturer. The maximum coefficient of variation (VK %, Inter-Assay) for all parameters (GADA, IA-2A, ZnT8A) on the EIA provided by the manufacturer was between 4.2 % and 9.3 % for serum samples and 30.7–44.36 % for quality controls. Routine VK % of controls over the complete testing period was 6.25 % (GADA), 11.37 % (IA-2A), and 8.71 % (ZNT8A) on the ELISA platform and therefore well within the limits of agreement.

All measurements and calibrations of assays were performed according to the manufacturers’ recommendations and specifications. Briefly, the iFlash used a three-point calibration in double determination as well as a master curve from the manufacturer as a reference for categorizing the measurement signals. The calibration of the Maglumi was carried out using two standards as well as the manufacturer’s stored master curve. Quality controls (low and high) were used to verify inter-assay precision. Low and high controls of GADA and IA-2A were tested over a period of 10–13 days. ZnT8A control measurements were only conducted on the Maglumi for two days, corresponding to the period in which the comparison tests for this parameter were conducted.

Data analysis

The methods chosen for comparing measurements of diagnostic accuracy were comparative measurements on all three devices via intra- and inter-assay precision. In this manuscript, overestimation/underestimation denote the direction of analytical bias (positive/negative) relative to the comparator. Bias estimation was conducted along with statistical analysis of mean, coefficient of variation, Passing-Bablok regression analysis, and Bland-Altman analysis. All measurements were performed as single measurements. All measurements above 250 U/mL were set to 250 U/mL before statistical analysis. Values in the range of 10–250 U/mL are presented unchanged. All measurements below 5 U/mL were set to 5 U/mL before statistical analysis. Thus, 250 U/mL and 5 U/mL define the measurement range for method comparison. This adapted method accounts for the maximum permissible deviation of 20 %. Cut-off values for semiquantitative and qualitative method comparison were chosen according to the manufacturers’ specifications (Table S1).

The statistical analyses and graphical displays were carried out using the Excel add-in Abacus 3.0, Version 1.40.36.10 (LABanalytics GmbH, 07,743 Jena, Germany). For the detailed analysis, we calculated relative differences between the averaged measured values of two devices and derived statistical key figures including range, mean value, median and Kendall’s tau. Non-parametric regression analysis was done via a Passing-Bablok regression analysis. Method agreement and comparison were done via Bland-Altman analysis including mean deviation, deviation range (both with 95 % CI) and maximum deviation and linearity testing (slope and intercept each with a 95 % CI). Semiquantitative analysis was performed with Cohen’s kappa (weighted) of 0.8 and a 95 % CI.

Method comparison (quantitative)

All parameters displayed notable deviations between platforms, with GADA exhibiting the largest discrepancies. Regression and Bland-Altman analyses indicated a significant mean deviation of −108.8 %, with iFlash values mostly being significantly lower than those of the EUROIMMUN Analyzer (Figure 1, Supplemental Figure S1B). Furthermore, significant measurement discrepancies in strongly positive EIA samples with inconsistent levels of deviations were measured on the iFlash (Figure 1). The Maglumi 800 from Snibe correlated better with the EUROIMMUN Analyzer, displaying a moderate mean deviation of −35.8 % for GADA. Accordingly, only the Maglumi showed a moderately good correlation for GADA in the Passing-Bablok regression analyses. This was evident only for values above approximately 40 U/mL. IA-2A had a mean deviation of −32.8 % with a very broad dispersion of iFlash values around those of the EIA. The discrepancy with Maglumi was greater due to a significant underestimation (negative bias) in results which resulted in a deviation of −50.3 %. For ZnT8A, the iFlash showed a very good correlation of values (Figure 2, Supplemental Figure S1H), while the Maglumi demonstrated poor correlation to the EIA, with systematically lower values. Especially for values below 200 U/mL, the Maglumi device tended to yield negative results (Supplemental Figures S1 and S2). No significant linear correlation between the two CLIA devices (iFlash and Maglumi) was observed. Results from these devices demonstrated wide dispersion, reflecting a lack of comparability with inconsistent underestimation of results and loss of agreement (Supplemental Figures S1 and S2).

Figure 1:

Bland-Altman analysis of method agreement for GADA between iFlash and EI-Analyzer.

Figure 2:

Bland-Altman analysis of method agreement for ZnT8A between iFlash and EI-Analyzer.

Method comparison (semiquantitative)

The semiquantitative method comparison was performed via Bowker test and Cohen’s kappa (weighted). It confirms the results of the quantitative testing with a general underestimation in the CLIA devices for all comparisons excluding ZnT8A on the iFlash (Table 1). GADA results highlighted the most significant underestimation (Table 1). Excluding ZnT8A on the iFlash both CLIA devices consistently showed an underestimation in direct comparison (Table 1, Figure 3, Supplemental Figure S3).

Figure 3:

Comparative measurements (green=positive, red=negative, yellow=borderline, white=not tested) of patients tested positive for multiple autoantibodies on the ELISA-platform. All listed antibodies were tested positive on the ELISA-platform.

In previous testing, the iFlash demonstrated excellent performance for ZnT8A with a non-significant Cohen’s kappa and Bowker test (Table 1).

Disagreement in the semiquantitative graph representing an underestimation of results is visualized as yellow and red cumulative circles below the bisector.

Method comparison (qualitative)

As shown in the semiquantitative method agreement (Supplemental Figure S3) significant discrepancies regarding the results fundamentally alter the agreement of GADA, IA-2A as well as ZnT8A. For qualitative comparison we used cut-off values used to discriminate between positive, borderline and negative. In a qualitative approach, 12 out of 13 patients tested positive for multiple autoantibodies on the EIA did not test positive for at least one antibody on at least one CLIA platform (Figure 3).

Because a loss of agreement is most relevant in a semiquantitative or qualitative approach with patients testing false-negative or false-borderline, we did an additional targeted analysis for patients tested positive for any one antibody on the EIA-platform. Assuming the correctness of the EIA-platform we were able to demonstrate a cumulative false-non-positive rate of 65 % for GADA with 31 out of 48 patients testing non-positive on at least one of the two CLIA-platforms. The corresponding cumulative non-positive rate for IA-2A was 43 % with 16 out of 37 patients testing non-positive on at least one of the two CLIA-platforms. Testing for ZnT8A was carried out in two independent batches. Therefore, no patient was tested in parallel. The non-positive rate for ZnT8A on the Maglumi alone was 48 % with 10 out of 21 patients testing non-positive (Figure 4). In previous testing of ZnT8A on the iFlash 42 out of 43 results matched qualitatively, representing the best agreement of all tested parameters. However, the qualitative agreement rate between iFlash and EIA was only 51.5 % for GADA and 68 % for IA-2A (Figure 4).

Figure 4:

Antibody specific measurements of patients with positive results on the EIA-platform (green=positive, red=negative, yellow=borderline, white=not tested).

Qualitative agreements between the EUROIMMUN Analyzer and the Maglumi 800 from Snibe were significantly better, at 75.4 % for GADA and 71.7 % for IA-2A. This rate further improved when borderline values were considered. In all three parameters, the CLIA devices did not show consistent agreement among themselves even on a qualitative approach (Figures 3 and 4, Supplemental Figure S3).

As expected from the quantitative and semiquantitative method comparison the highest result agreement could be observed for ZnT8A between EIA and iFlash. The best qualitative agreement for GADA was between EIA and Maglumi, although EIA measurements in the weakly positive range were undetectable by Maglumi.

Additional measurements

Due to the lack of certified reference material, the cross-measurement of control and standard material of the individual devices on other devices was conducted as a means to still utilize defined target values.

On the two CLIA devices, using multiple patient samples from different concentration ranges for GADA and IA-2A, geometric dilution series with ratios of 1:2, 1:4, 1:8 were performed (data not shown). This aimed to confirm the linearity of the tests within the intended linear measuring range and to examine whether reliable results could be obtained with a dilution of the patient material in case of insufficient starting material. Additionally, the dilution series included an examination for possible high-dose effects in the particularly high concentration range.

An underestimation of results on the CLIA devices was identified in the low to middle range, where the EIA rendered positive signals, whereas the two CLIA devices measured negative results. With the Maglumi, this measurement gap occurred for GADA up to approximately 40 U/mL and for IA-2A up to 70 U/mL, while the iFlash showed lower agreement beyond this range. Repeat measurements of the discrepant results on all devices confirmed these values. Subsequently, EIA measurements were confirmed by retesting doubtful samples with a dilution of 1:2 on this device.

During the initial comparative measurements on the ELISA, a precipitation phenomenon was noticed with strongly positive samples for IA-2A and GADA. High concentrations, identifiable by an optical density of 2 or higher, led to black precipitates in addition to the typical yellow coloration due to the enzymatic reaction with TMB. These precipitates negatively influenced the measurement results by reducing the optical density and causing falsely low values. We performed repeated measurements and thorough investigation into the causes and were able to identify the root cause. In consultation with the manufacturers, solutions are being sought. Besides strongly positive patients, the precipitates affected either the highest standard or the two highest standards depending on the test. Since the medically validated range ends at 250 U/mL for GADA and at 400 U/mL for IA-2A and ZnT8A, for comparative measurements all values over 250 U/mL across the three devices are recorded as 250 U/mL in the subsequent analyses. This minimized the significance of the problem for the statistical evaluation to a minimum.

Lack of a gold standard

Due to the lack of a true gold standard for this method, we chose the ELISA as the reference method, and multiple efforts were made to verify test integrity. First and foremost, the ELISA method has been established for years and thoroughly officially verified via interlaboratory testing multiple times a year. Furthermore, as described, we performed series of geometric dilution with certified reference material which did confirm linearity for the ELISA method. Additionally, the ELISA method has been used and established as reference method in multiple previous publications mentioned above as de facto industry standard and therefore allows for direct comparison. While there is no indication of bias from the ELISA method, clinical correlation could further validate its use as a reference method.

Study limitations

Our clinical correlation was limited by incomplete clinical patient data, preventing ethnogenetic stratification. Therefore, we used our routine diagnostic ELISA and its provided controls as the reference. To avoid issues with known precipitation at high concentrations, we capped statistical comparisons at 250 U/mL. Because we explicitly aimed to compare agreement in a stratified lower positive range of measurements agreements are likely to not reflect the distribution of results in the clinical setting. Although we aimed to cover the full diagnostic spectrum, larger randomized studies are needed to reduce bias, strengthen statistics, and enhance generalizability.

Implications for practice

Because autoimmune antibodies are primarily used in the diagnosis with only limited clinical benefit for longitudinal intrapatient monitoring, we primarily focused on interchangeability of methods rather than longitudinal test integrity of the mentioned CLIA devices. However, our findings revealed that only ZnT8A on the iFlash met the high agreement expectations for CLIAs. GADA and IA-2A showed underestimations of up to 49.5 % and mean deviations up to −108.8 % compared to ELISA on the same device. This is in line with recent publications where gross deviations especially in the lower range have been shown. Interchangeability of GADA in CSF on the Maglumi has been proposed recently for strongly positive or negative samples. Individual sample data of positive samples showed major deviations in the ratio of measured value to the cutoff in the method comparison. When measurements are compared the quotient (ELISA/CLIA) of measurement to cutoff between both systems has been shown to range between 2.54 and 100.67 (ELISA: 10.6 times above cutoff; CLIA: 4.2 times above cutoff and ELISA: 150.4 times above cutoff; CLIA: 1.5 times above cutoff) [26]. These findings were confirmed in a subsequent study where serum samples in the range below 60 IU/mL showed a % difference deviation in the Bland-Altman plot of up to over 150 % in both directions. The results in serum as shown in the Bland-Altman plot show similar distribution to the results measured with the same platform and the same method in CSF which may point to a general discrepancy of the assay regardless of the matrix. The authors reported an underestimation by the Maglumi assay with respect to the EUROIMMUN ELISA. They were able to show a poor linearity of the Maglumi assay below 60 U/mL with dilution series which was suspected due to a flat calibration curve for concentrations below 60 U/mL [27]. Using the same reference method previously described in both major publications we were able to corroborate these findings of a poor linearity in the range below 60 U/mL for serum samples. Additionally, we were able to confirm previously published linearity issues of GADA between ELISA and CLIA. Both previously described limitations in method comparability did prove to be highly impactful in the stratified comparison in the range of cutoff values.

Even among CLIA devices, no consistent agreement for any of the three measured parameters emerged. Given the diagnostic and prognostic importance of autoantibodies in T1DM and LADA, a drop of about 50 % in method agreement could markedly diminish the quality in patient care.

We gross deviations in agreement for GADA and IA-2A between the Maglumi and the ELISA in low-positive ranges, possibly stemming from suboptimal antigen composition or matrix effects. Further complicating these results is that we could not show a reproducible overarching bias when generally comparing CLIA and ELISA. These issues could not be clarified by communication with the manufacturer and persisted even at higher concentrations, suggesting an unpredictable false-negative risk. Ethnogenetic differences likely contribute to discrepancies, since assays validated primarily on Asian cohorts may not translate well to European populations. For instance, known polymorphisms in ZnT8 can alter epitope conformation and affect detection [25]. Furthermore, manufacturer validations often rely on spiked samples, which may not accurately replicate patient sample matrices. This could explain why the discrepancy in agreement is less pronounced on the Maglumi when testing non-patient samples (e.g. standards or controls) or diluted patient samples. Consistent with our suspected matrix effect, previous studies also proposed the use of spiked samples by manufacturers in validation studies to be the reason for increased CV and respective violation of precision thresholds for both Maglumi and iFlash [28]. Previous studies did show deviating reference ranges from the reference ranges provided by the manufacturer. For measurements of GADA and IA-2A on the Maglumi, ethnic and local differences in expression of antigen and corresponding autoantibodies have been proposed as a main factor for reference range deviations [29]. Comparable results have been shown for other tests like procollagen-type-III and procalcitonin [30], [31]. Considering these findings, it is to be expected that ethnogenetic differences in epitope expression and corresponding antibody affinity and avidity play a major role in the discrepancies published in the primarily European population. Genetic polymorphisms like amino acid variations at position 325 of the ZnT8 are known to cause antigen diversity through changes in dimeric conformation and thus antibody recognition [32]. As specificity of molecular targets is increasing alongside the growing scientific knowledge of disease etiology, an evaluation of global characteristics of the relevant molecular targets is critical to ensure test validity in a clinical setting. This is especially warranted in the case of epitope distribution with immunoassays continuing to be a backbone of clinical laboratory assays. The definition of a cut-off value is a pivotal point in semi-quantitative immunoassays directly translating a method’s sensitivity to clinical consequence. Because variations in antigenic epitope distribution are mostly independent phenomena, transferability of validation data should not be expected even if comparable assays from the same manufacturer testing for different analytes are within specifications.

Outlook

Our findings reveal major inconsistencies between ELISA and CLIA methods in detecting diabetes-related autoantibodies, highlighting the need to investigate underlying causes such as ethnogenetic variations in epitope expression and antibody affinity. Moreover, the current shift from ELISA to CLIA highlights insufficient standardization and overall heterogeneity, complicating cross-study comparisons. Larger, more diverse studies are needed to clarify genetic factors and establish broader reference ranges. Further technical evaluations of CLIA devices are also necessary to understand underestimations of test results and matrix effects. Finally, international collaboration is crucial for standardizing protocols and ensuring transparency in test validation, thereby improving diagnostic reliability, comparability and patient care in a globalized healthcare environment.