Performance criteria for combined uncertainty budget in the implementation of metrological traceability

Introduction

The laboratory measurement paradigm is based on the assumption that assays claiming to measure the same analyte should provide equivalent results for long-term and within clinically meaningful limits. Time, laboratory location and the assay system used to produce them should not influence patient results. There is now an international agreement on the fact that, to become equivalent for long-term, results must be traceable to higher order references [1]. In this regard, it is essential to build an unbroken metrological traceability chain whereby in vitro diagnostics (IVD) manufacturers can implement a reliable transfer of the measurement trueness from the highest level of the metrological hierarchy to calibrators of commercial analytical systems used in clinical laboratories and, finally, to the patient’s results. Only in this way results obtained by the calibrated commercial procedure are expressed in terms of the values obtained at the highest available level of the calibration hierarchy [1]. The European Union (EU) Directive 98/78 on IVD medical devices, supported by a specific standard of the International Standardization Organization (ISO) [2], requires manufacturers to ensure traceability of their analytical systems to recognized higher order references and the compliance with the IVD Directive is indicated through the CE (“Communautés Européennes”) marking of conformity on diagnostic products [3].

For establishing traceability some basic requirements should be fulfilled. Firstly, it is essential to establish a calibration hierarchy starting from the unequivocal definition of the measurand as the quantity subject to measurement [4]. The assay selectivity for the measurand at each level of the traceability chain is a crucial aspect and in a standardization project correlation studies should preliminarily be performed to test the relationship among methods. By definition, results on different measurands are indeed not comparable and cannot be standardized [5]. Another basic concept behind the theory of metrological traceability is that the measurement bias along all the traceability steps, if any, should be appropriately eliminated [6]. The applied implementation strategy should reliably transfer the measurement trueness from the highest level of the hierarchy to values of commercial calibrators, permitting to obtain unbiased results on clinical samples. Finally, an adequate estimation of measurement uncertainties should be performed [7]. Even when measurement procedures operate under unbiased conditions, results produced on biological samples have an associated uncertainty that derives both from uncertainties accumulated along the previous steps of the metrological chain and from random effects [8].

The temple of laboratory standardization

Classical key elements of the reference measurement system are higher order reference materials, reference measurement procedures and accredited reference laboratory services performing them [9]. Over time, it was however realized that three additional components are essential to correctly implement the metrological traceability concept in clinical practice [7]. First, only the availability of traceable reference intervals/decision limits will permit the definitive implementation of standardization projects, filling the gap between the production of traceable results and their correct use for patient care [10, 11]. Second, an analytical (internal and external) quality control program meeting metrological criteria has to be organized, becoming the fifth pillar of laboratory standardization [1, 8]. Finally, the sixth pillar sustaining the “temple” of laboratory standardization is the setting of targets for uncertainty and error of measurement that fit for purpose [7, 8].

Defining analytical performance goals for each analyte is essential to make their determinations clinically usable and to ensure that the measurement error does not prevail on the result [7, 12]. In the conference held in 1999 in Stockholm, a hierarchy of sources for deriving the analytical goals of a laboratory measurement and defining quality specifications in laboratory medicine was established [13]. After 15 years, a new conference recently held in Milan has revisited the Stockholm consensus, investigating to what extent the advocated hierarchy is still valid or if it has to be changed or expanded [14]. Although the essence of the hierarchy established in Stockholm was supported, new perspectives have been forwarded prompting simplification and explanatory additions. According to the new consensus statement, the recommended approaches for defining analytical performance goals should rely on the effect of analytical performance on clinical outcomes or on the biological variation of the measurand [14]. The most innovative aspect of this consensus is that it is recognized that some models are better suited for certain measurands than for others; the attention is therefore primarily directed towards the measurand and its biological and clinical characteristics. For instance, the model based on biological variation is probably not appropriate for analytes showing high intra-individual CV (CV_I) (i.e., >33%) [15]: for those analytes, outcome data or, in their absence, the state-of-art of the measurement should be the models to derive analytical goals.

Definition of the measurement uncertainty budget

Once the analytical goals are established, it is essential to understand how they can be fulfilled in practice. Focusing on the measurement uncertainty, the relevant goal that should be considered is that classically related to the allowable analytical variability (“imprecision”) [16–18], as the correct trueness transfer along the entire metrological traceability chain should allow the achievement of unbiased (or negligibly biased) results. For instance, using the biological variation model the analytical quality specification for measurement uncertainty is 0.25, 0.50 or 0.75 CV_I, depending on the quality level one want to reach [18]. The (expanded) uncertainty goal (G_U) represents the budget that should be fulfilled when combining the uncertainty of the analytical system employed in the individual laboratory (i.e., the random effects) to that accumulated along all the steps of metrological traceability chain, and then expanding it by a coverage factor. It is, therefore, essential that contributions to measurement uncertainty are defined across the entire traceability chain, starting with the reference materials, extending through the IVD manufacturers and their processes for assignment of calibrator values and, ultimately, to the final results reported to clinicians by clinical laboratories [1, 19].

The reference provider, the IVD manufacturer and the clinical laboratory have different roles along the metrological traceability chain. Although independent in their tasks, their activities contribute together to the total measurement uncertainty budget. Basically, the value assigned to the measurement standard(s) at each level in the calibration hierarchy shall have an uncertainty of measurement that includes the uncertainty contributions from each higher calibration step in the hierarchy. Accordingly, the combined uncertainty must be calculated as the square root of the sum of the squares of each uncertainty component [i.e., uncertainty of references, uncertainty of commercial calibrator and random uncertainty (analytical system imprecision plus individual laboratory performance)] and then converted to expanded uncertainty by multiplying the value by a coverage factor k (k=2 is recommended for a confidence level of approx. 95% for a normal distribution). Once the employed traceability chain and the corresponding (expanded) combined uncertainty are drawn, it will be possible to compare the latter with the appropriate G_U selected for that specific measurand. We previously reported some examples of this approach, which is useful for establishing if the current status of the uncertainty budget of an analyte measurement is suitable for clinical application of the test [1, 7, 19, 20]. Hopefully, it should be applied to every analyte measured in the clinical laboratory.

Allowable limits for measurement uncertainty across the traceability chain

To obtain that a given measurement is able to fulfil the established G_U when patient samples are determined, it is necessary to define what part of the G_U should be used across the different steps of metrological traceability chain, by establishing combined uncertainty limits at different chain levels as a proportion of G_U.

Uncertainty margins for clinical laboratories

With regard to the contribution to measurement uncertainty, the role of clinical laboratory is to monitor the reliability of the employed commercial system through an internal quality control (IQC) program estimating the uncertainty due to the random effects, which includes the analytical system imprecision together with the individual laboratory performance in terms of variability [8]. This program should provide, through mechanisms of retrospective evaluation, data useful to the knowledge of variability of analytical system (e.g., the reagent lot-to-lot variation) and of its daily use by the individual laboratory. We already emphasized how this IQC component II should be appropriately designed and keep completely separated by the other IQC component I addressed to check the alignment of analytical system [7, 8]. Table 1 reports the most important characteristics for a control material to be used in the IQC component II program.

Table 1

Main characteristics for a control material to be used in the internal quality control component II program in order to derive the uncertainty of the analytical system due to the random effects.

Characteristic	Remarks
Matrixed material from a third-party independent source should be used (e.g., fresh-frozen pool)	Material must be different from the system control material used for checking its alignment
Material should closely resemble to authentic patient samples (fulfil commutability)	Commercial non-commutable controls may provide a different impression of imprecision performance
Material concentrations should be appropriate to the clinical application of the analyte	When clinical decision cut-points are employed for a given analyte, samples around these concentrations should preferentially be selected

As reported above, if for being acceptable, the degree of uncertainty (expanded) of a measurand results in the clinical setting (including the accumulated uncertainty of the corresponding traceability chain) should stay within G_U, decisions need to be made on what proportion of this budget can be used up in the traceability chain to ensure enough budget is left for use in clinical analysis. In principle, to allow that diagnostic laboratory provides clinically reliable patient results fulfilling the performance analytical goals it is necessary that uncertainties accumulated at level of reference providers and IVD manufacturers be sufficiently confined. Figure 1 summarizes the recommended limits for combined uncertainty budget (expressed as percentage of total budget G_U) in traceability implementation.

Figure 1:

Recommended limits for sources of combined uncertainty budget (expressed as percentage of total budget uncertainty goal) in traceability implementation.

IQC, internal quality control.

Why uncertainty limits matter

The CE marking on IVD products by itself does not guarantee that manufacturer has transferred trueness successfully as at present no normative verification or validation of the manufacturer’s statement by a third party is provided [1]. However, even when the traceability of calibration is successfully implemented, this does not ensure acceptable trueness and/or uncertainty for individual patient results. Using the example of glucose, we recently showed that the selection of different types of traceability chains (all officially recognized in the JCTLM database) may lead to different combined uncertainties at the level of commercial calibrators, not always permitting to fulfil the G_U [7]. Unfortunately, the information regarding uncertainty associated with commercial calibrators is currently very poor when consulting calibrator package inserts. Manufacturers only provide the name of higher order reference material and/or procedure to which the assay calibration is traceable, without any description of implementation steps and their corresponding uncertainty or, if any, the use of acceptable limits for calibrator uncertainty assignment. Table 2 lists the information that, in our opinion, laboratory users should be able to access, ideally in the assay or calibrator package inserts. In this regard, it is expected that the ISO Technical Committee 212 working on the revision of the ISO 17511 standard will consider these requirements, asking manufacturers to make them easily available, in line with the concept that the traceability implementation by IVD manufacturers should necessarily be associated to the demonstration that the commercial system meets the analytical goals established for its clinical use.

Another aspect that may compromise the patient’s result accuracy regardless of successful implementation of calibration traceability is the too large uncertainty (including assay imprecision) of the available traceability chain that does not permit to meet clinical needs. By analyzing the uncertainty of the current traceability chain for HbA_1c, it was clear that the (expanded) combined uncertainty associated with the measurement of a human sample in even optimal conditions (approx. 4.0%) is still >2 times the minimum acceptable target for the clinical application of the test [19]. Our conclusion was that further advances are needed to reduce the uncertainty associated with higher order metrological references and to improve the precision of commercial HbA_1c assays [19].

The last important aspect that may influence the accuracy of individual patient’s results is the selectivity (“specificity”) of commercial assays for the measurand as the quantity subject to measurement. A classic example is that of creatinine, whose determination in daily laboratory work can be performed by alkaline picrate-based and enzymatic methods. It is well known that the alkaline picrate-based assays are unable to measure solely creatinine and some endogenous and exogenous substances in serum, particularly proteins, may significantly interfere causing significant overestimation [26]. Similarly to the previously reported example of glucose [7], by analyzing IVD manufacturer statements we were able to show that at least four different types of metrological traceability chain are currently employed to assign values to commercial calibrators of marketed creatinine assays (Figure 2). The (expanded) combined uncertainty associated with the selected chain ranged from 1.5% to 2.8% showing, like for glucose, that IVD manufacturers may spend substantially different amounts of the total uncertainty budget just by selecting one or the other of the available traceability chains (Table 3). Differently from glucose, however, the same calibrator on the same platform (e.g., Roche Cobas, Modular or Siemens Advia) shows different uncertainty depending on which method is employed, with alkaline picrate assays displaying markedly higher calibrator uncertainties than enzymatic assays (Table 3). It is well known that traceability implementation is unable to correct for analytical non-specificity issues [26, 28] and the manufacturers’ approach trying to mitigate this problem inevitably leads to higher uncertainties of commercial calibrators when used for alkaline picrate methods.

Figure 2:

Types of metrological chains that can be used to implement the traceability of blood creatinine results.

GC-IDMS, isotope dilution-mass spectrometry coupled to gas chromatography; LC-IDMS, isotope dilution-mass spectrometry coupled to liquid chromatography; NIST, National Institute of Standards and Technology; SRM, standard reference material.

Conclusions

In addition to the correct implementation of calibration traceability, the definition of the total expanded uncertainty budget is essential to make sure that laboratory measurements are clinically usable. In particular, it is necessary to define combined uncertainty budget limits across the entire metrological traceability chain. We recommend that no more than one third of the total uncertainty budget, established by appropriate analytical performance specifications, is consumed by the uncertainty of references and approximately 50% of the total budget consumed by the manufacturer’s calibration and value transfer protocol. The remaining 50% should be available for the commercial system imprecision and individual laboratory performance as a safety margin in order to fulfil the G_U.

Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

Financial support: None declared.

Employment or leadership: None declared.

Honorarium: None declared.

Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.