Commutability of the certified reference materials for the standardization of β-amyloid 1-42 assay in human cerebrospinal fluid: lessons for tau and β-amyloid 1-40 measurements

Introduction

Biomarkers are valuable tools that can be used in the clinical diagnostic process and as an inclusion criterion for clinical trials with the aim of finding disease modifying drugs [1]. In the field of Alzheimer’s disease (AD) there are presently four core cerebrospinal fluid (CSF) biomarkers described in the literature with a potential clinical value: the 40 and 42 amino acid-long amyloid-β (Aβ40 and Aβ42), the microtubule-associated protein tau (T-tau), and its hyper phosphorylated variant (P-tau) [2]. In a large meta-analysis study, the concentrations of these core biomarkers in CSF were shown to be consistently altered in patients with AD compared to cognitively unimpaired individuals of a similar age [3]. Today these core biomarkers are only included in the research criteria for AD [4], [5], but ideally uniform global cut-off levels will be established to facilitate a more general use, also in routine clinical diagnostics. However, this is partially hindered by the lack of harmonization between different methods and laboratories. Starting in 2009, efforts have been made to investigate this discord in detail when the Alzheimer’s Association quality control (QC) program [6], [7] for CSF biomarkers was launched (www.Neurochem.gu.se/TheAlzAssQCProgram). A QC program by itself can do little to improve the situation but is a valuable tool in assessing the progress of method optimizations and to compare the quality in terms of variability and concentration differences between different assays. It may also increase the awareness of problems and engage laboratories that produce erroneous concentrations to revise their internal procedures for sample analysis. Partly driven by the rather discouraging results from the first rounds of the QC program [6], [7], manufacturers have worked on improving their assays and new fully automated methods with superior precision are emerging [8].

The lack of harmonization between different assays can be overcome by standardization through the development of reference measurement procedures (RMPs), the use of certified reference materials (CRMs) and the establishment of a traceability chain. The concept to use CSF for assay harmonization was described previously for the xMAP – enzyme-linked immunosorbent assay technology comparison, in which CSF Aβ42 concentrations were compared between assays using a CSF-calibrator series [9]. Therefore, the International Federation for Clinical Chemistry and Laboratory Medicine set up a working group with the aim to develop CRMs for Alzheimer’s biomarkers in CSF [10]. As a part of this process there is a need to show that reference materials are commutable for the different assays. The Vocabulary of Metrology [11] defines commutability as the “property of a reference material, demonstrated by the closeness of agreement between the relation among the measurement results for a stated quantity in this material, obtained according to two given measurement procedures, and the relation obtained among the measurement results for other specified materials”. In other words, results obtained using different assays must show a high degree of correlation, and results from CRMs must be indistinguishable from representative individual samples for which the methods are intended to be used. Based on the results from a previous commutability study for Aβ42, it was obvious that artificial matrices (e.g. phosphate buffered saline, or artificial CSF), to which synthetic Aβ42 was spiked, were not commutable [12]. As a consequence, CRMs were prepared from CSF pools with different endogenous concentrations of Aβ42. The first aim for the current study was to examine if the CRMs are commutable with respect to Aβ42. Samples of the three CRMs and 34 individual CSF samples were sent to five immunoassay manufacturers with commercially available assays for Aβ42. In addition, commutability data for the other core biomarkers, i.e. T-tau and P-tau and Aβ40 (the two amino acid shorter and less aggregation-prone Aβ form that may be used to normalize individuals in regards to their constitutive Aβ production rate [13]), were generated on the same sample set. The latter can be considered as a feasibility study for the development of CRMs for these other CSF biomarkers.

Materials and methods

Results

The different assays that were used in this study are listed in Table 1. For the assay from IBL one result for T-tau was not reported and this sample was excluded in the subsequent analysis for all other T-tau assays as well. For T-tau measured on the Lumipulse platform only one result per sample was reported due to shortage of sample volume. Over all the repeatability, based on duplicate determinations, was very good for all assays and analytes with an average coefficient of variation per method of 0.4%–5.9%.

The correlations, using data only from individual samples, for all pair-wise method comparisons for Aβ42 were strong with a median correlation coefficient of 0.98 (min–max: 0.94–0.99) (Figure 1, Supplementary Figure 1 and Supplementary Table 1). The slopes of the regression lines were in the interval 0.31–2.99 (Supplementary Table 1) and the normalized distance to the regression line for the individual samples and the CRMs were not statistically significant at the 95% confidence level (Supplementary Figure 2A) with an AUROC of 0.557 (95% CI: 0.495–0.619).

Figure 1:

Scatter plots for pair-wise comparisons for Aβ42 with the LC-MS method against (A) ROCHE, (B) IBL, (C) Euroimmun, (D) MesoScale, (E) INNOTEST, (F) Modified INNOTEST and (G) LUMIPULSE.

The solid black line represents the Passing-Bablok regression and the upper blue dotted lines are constructed from the upper limit for the 95% confidence interval of the slope and intercept while the lower are based on the lower limits. The red dashed line denotes the unity line (y=x). Symbols: individual CSF samples (gray circles); candidate CRMs (ERM-DA480/IFCC [green square], ERM-DA481/IFCC [yellow diamond], and ERM-DA482/IFCC [red triangle]).

For Aβ40 measurements the correlations were strong as well with a median correlation coefficient of 0.97 (min–max: 0.93–0.98) (Figure 2, Supplementary Figure 3 and Supplementary Table 2). The distance to the regression line for the individual samples and the CRMs were also not statistically significant at the 95% confidence level (Supplementary Figure 2B) with an AUROC of 0.598 (95% CI: 0.496–0.701). The distribution range of the CRMs was smaller for Aβ40 as compared to Aβ42, attributed to the fact that only the Aβ42 concentrations were considered when preparing CSF pools used for the production of the CRMs. Furthermore, the individual samples were selected solely based on the clinically relevant range for Aβ42 but nevertheless these samples still cover an adequate range for Aβ40.

Figure 2:

Scatter plots for pair-wise comparisons for Aβ40 with the LC-MS method against (A) IBL, (B) Euroimmun, (C) MesoScale and (D) INNOTEST.

The solid black line represents the Passing-Bablok regression and the upper blue dotted lines are constructed from the upper limit for the 95% confidence interval of the slope and intercept while the lower are based on the lower limits. The red dashed line denotes the unity line (y=x). Symbols: individual CSF samples (gray circles); candidate CRMs (ERM-DA480/IFCC [green square], ERM-DA481/IFCC [yellow diamond], and ERM-DA482/IFCC [red triangle]).

Even though the correlation was strong between different assays for both Aβ42 and Aβ40 this did not generally hold true for the Aβ42/Aβ40 ratio (Figure 3, Supplementary Figure 4 and Supplementary Table 3). The correlations had a median of 0.85 (min–max: 0.58–0.97). Comparing the correlations for the components of the ratio, the main contribution for the lower correlations seemed to come from Aβ40 and the variability increased for the ratio as a consequence of the propagation of uncertainty. The CRMs essentially covered the range defined by the individual samples and it can be noted that in all method comparisons for the ratio, the three CRMs fell closely on a straight line even when the variability for the individual samples was high (Figure 3 and Supplementary Figure 4). The effect size indicated that there was no statistical difference between the individual samples and the CRMs at the 95% confidence level with an AUROC of 0.576 (95% CI: 0.481–0.671; Supplementary Figure 2C).

Figure 3:

Scatter plots for pair-wise comparisons for 10xAβ42/Aβ40 with the LC-MS method against (A) IBL, (B) Euroimmun, (C) MesoScale, (D) INNOTEST and (E) Modified INNOTEST.

The solid black line represents the Passing-Bablok regression and the upper blue dotted lines are constructed from the upper limit for the 95% confidence interval of the slope and intercept while the lower are based on the lower limits. The red dashed line denotes the unity line (y=x). Symbols: individual CSF samples (gray circles); candidate CRMs (ERM-DA480/IFCC [green square], ERM-DA481/IFCC [yellow diamond] and ERM-DA482/IFCC [red triangle]).

For T-tau the correlations were strong for all assays used (Supplementary Figure 5 and Supplementary Table 4) with a median of 0.97 (min–max: 0.92–0.99). Although there was a small difference between the CRMs, with regard to the measured concentration, none of them deviate substantially from the regression line (Supplementary Figure 5) and they behave very much like the individual samples (Supplementary Figure 2D) with an AUROC of 0.564 (95% CI: 0.471–0.656). The same is true for P-tau for the only pair of assays that were included in the study, where the correlation was r_s=0.975 and the slope of the regression line was 2.55 (Supplementary Figure 6). As there were only three results for the CRMs (Supplementary Figure 2E) the uncertainty of the effect size was large with an AUROC of 0.657 (95% CI: 0.392–0.922).

The F-test for goodness of fit comparing the nested linear or second order polynomial fit revealed that both INNOTEST and Lumipulse had a p<0.05 for all pairwise comparisons of Aβ42 measurements. This indicates that the associations were non-linear. The F-test results were not statistically significant at the 95% confidence level for any of the other Aβ42 method comparisons. For Aβ40 measurements, two out of four comparisons with MSD (i.e. vs. INNOTEST and IBL) indicated non-linearity. Of the ten possible combinations for T-tau assays, three had a p<0.05 for the F-test (Roche vs. Euroimmun, Euroimmun vs. INNOTEST, and INNOTEST vs. MSD).

Discussion

The presented results demonstrate that the approach to prepare CRMs for Aβ42 from pooled CSF samples can be considered valid. All three CRMs are commutable for all combinations of methods and all immunoassay methods for Aβ42 show a strong correlation of results with the results obtained with the LC-MS method. This is a prerequisite for the use of the RMP, which is also a LC-MS method, for the value-assignment of the CRMs.

One of the potential confounding factors for the LC-MS method is the methionine residue on position 35 in the Aβ42 sequence which can be subjected to oxidation forming a methionine sulfoxide [21]. This type of post translational modification has a biological function in other proteins [22] and it can also form in vitro on Aβ42 [23]. The LC-MS only measures non-oxidized Aβ42, as opposed to the antibody-based methods that are expected to not discriminate between the two forms. However, strong correlations between LC-MS and the other methods suggest that the fraction of oxidized Aβ42 is very low and/or constant. Thus, oxidation of methionine in Aβ42 is insignificant in this system. The same reasoning and conclusion holds true also for Aβ40.

By itself, CSF Aβ40 is not a biomarker with a high clinical utility in the field. However, the ratio of Aβ42/Aβ40 seems to improve the diagnostic performance [24], [25] and it also shows better concordance with amyloid positron emission tomography [16], [26], which can be considered the gold standard for imaging the Aβ pathology in the brain. A candidate RMP for Aβ40 has been submitted to Joint Committee for Traceability in Laboratory Medicine for approval as there is a need for a CRM for this marker as well to establish a global cutoff of the Aβ42/Aβ40 ratio. However, the three CRMs for Aβ42 do not cover an appropriate clinical range of Aβ40 concentrations, which means that separate CRMs for Aβ40 are needed. Nevertheless, the presented results serve as a successful feasibility study for developing CRMs for Aβ40. The feasibility also applies to T-tau and P-tau, even though the latter only involved two different methods in this study. Also for T-tau and P-tau, the CRMs investigated do not span the clinically relevant range of concentrations and therefore new CRMs will have to be prepared to verify the concept of using the same type of materials. The protocol to prepare CSF and impact of confounding factors during storage and collection is much more robust for tau, P-Tau and other proteins as compared to Aβ1-42 [27]. Work is ongoing to establish a LC-MS method for T-tau, which is within reach, while the sensitivity needed for P-tau makes a MS based method for this analyte a bigger challenge. Compared to Aβ42 the other analytes showed less systematic deviation from linearity for any of the assays.

Non-linear association between different methods might pose a problem for the harmonization of global diagnostic cutoffs and manufacturers should take measures to ensure that their methods have linear associations with the RMP. It can be that the association is in fact linear and that a few erroneous results drive the non-linear appearance highlighting the need of replication studies. While this might be true, for example, for Aβ42 measurements on Lumipulse, the INNOTEST assay appears to be non-linear when compared to other methods. This has already been noticed in a previous commutability study although the linearity issue was not assessed in detail [12]. In this regard the modified INNOTEST [15] is an improvement as the test did not indicate non-linearity for any comparisons. It is important to keep in mind that the concentration of the calibrator used with the LC-MS is not fully characterized, and deviations from the unity line might be attributed to the uncertainty of the concentration of the calibrator used.

Harmonization of methods used to measure biomarkers is important for the establishment of global cutoff levels but pre-analytical confounding factors also contribute to the variability of the measured concentrations of the biomarkers [28], [29], [30], [31], [32], [33]. One aim with the BIOMARKAPD project in the framework of the Joint Programming Neurodegenerative Disease was to address this source of error resulting in recommendations for pre-analytical handling of CSF samples [34]. Within the framework for the Alzheimer’s Association QC program for CSF biomarkers an attempt was made to find methodological factors that contributed to the variability in the reported results. A questionnaire was sent out in which the participants were asked to answer questions about instrumentation, equipment, training of staff, storage of samples and kits, etc. From the replies, no conclusion could be made regarding specific factors that potentially contributed more to the variability than others (unpublished results). The human factor is possibly a great contributor to the variability and fully automated assays such as the Cobas (Roche) and the Lumipulse (Fujirebio) systems, included in this commutability study, have the potential to minimize human interference on the results. Other random-access analyzers will be available in the future, such as qualified by Euroimmun [30]. Although these platforms are not new to the market they are not commonly used for AD biomarkers. It remains to be seen if they surpass the more manual methods with regard to precision in the future. Results from the initial rounds in the QC program in which these platforms were included indicate that this might be the case (www.Neurochem.gu.se/TheAlzAssQCProgram).

The materials both in the external QC program and in the CRMs are pooled leftover CSF samples which mimic patient samples well, as show in this study. This property is requested for internal QC samples according to ISO 15189 but the CRMs are primarily to be used by kit manufacturers and not by individual medical laboratories. If pooled CSF is not available in practice, internal artificial QC samples can be prepared that fulfill the criterion of similarity with patient samples [35]. However, not all artificial matrices commute [12] and this must be taken into account before an internal QC program is established.

In summary, the results presented supported the work for releasing the first CRMs for CSF measurements of Aβ42. For the other biomarkers (i.e. Aβ40, T-tau, and P-tau) more work is needed, but the present results of the feasibility are encouraging. The value assignment of the CRMs is completed and the CRMs have been released and available for purchase from the Joint Research Centre (crm.jrc.ec.europa.eu). They can now be used by the immunoassay manufacturers to recalibrate their assays as mandated in the EU Directive on In Vitro Diagnostics Medical Devices (Directive 98/79/EC). The release of the materials should improve the concordance of measurement results between different analytical platforms as well as laboratories and will allow the establishment of a global diagnostic cutoff value for Aβ42 in CSF that can be used for the diagnosis of AD. The arrival of a global diagnostic cutoff is positive but the pedagogical hurdle of informing the physicians, and make them accept the new reality, should not be neglected.