The quest for equivalence of test results: the pilgrimage of the Dutch Calibration 2.000 program for metrological traceability

Pre-requisites for Calibration 2.000

The approaches used were comparable ‘avant la lettre’ to the integrated harmonization protocol described in the toolbox of technical procedures for harmonization at http://www.harmonization.net/media/1004/tool_box_2013.pdf.

Standardization and harmonization projects were prioritized based on unmet clinical needs and feasibility. The ultimate goal of Calibration 2.000 is to contribute to better patient care by delivering the right test result for the right patient at the right time. Accurate results are key for adequate patient care in which laboratory data play a major role, which is the case in the majority of medical decisions (70% rule). To that end, metrological traceability of laboratory data according to ISO 17511:2003 is pertinent.

For correct interpretation of test results, matching clinical reference intervals and decision limits are key in combination with SI-units whenever feasible. Absolute concentrations are often used in clinical decision making (e.g. cholesterol, creatinine or derived parameters, thyroid stimulating hormone, glucose) whereas interlaboratory and intralaboratory differences, especially differences in analytical set-points, can markedly affect the clinical interpretation of tests [10]. This also holds for reference intervals. The use of standardized reference intervals, decision limits and SI-units is still debated at the international level but currently gets attention from European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) working groups. However, standardization of units should not be controversial as the use of different units for results of the same analyte can lead to clinical misinterpretation and patient harm [11]. In the Netherlands, the discussion on the necessity of molar standardization started in the 1960s and led at that time to a controversy between internal medicine doctors and specialists in laboratory medicine, described in the Blind Mole [12]. Since then, Dutch medical laboratories are nationwide using SI-units.

Definition of preset analytical performance criteria for the needed level of equivalence were obtained at first from the Stockholm conference on strategies to set global quality specifications in laboratory medicine [13], and more recently from the 1st Strategic Conference of the EFLM on defining analytical performance specifications [7]. The Milan conference criteria based on clinical outcome studies (Model 1) and biological variation (Model 2) were the preferred models. State of the art based criteria were considered second best.

Tools developed for the Calibration 2.000 program were commutable trueness verifiers or harmonizers and a targeting system. Standardization or harmonization can only be reached when targeted commutable materials are available for trueness verification. The development of such materials was considered as the ‘Holy Grail’ of the Calibration 2.000 program. As well, a toolbox of technical procedures to be considered for developing a process to achieve harmonization or standardization of measurands formed the basis.

In the program, participating reference laboratories for enzymes, lipids, PT/INR and HbA_1c are encouraged to apply for ISO 15195 and ISO 17025 accreditation in order to get their reference measurement procedures internationally recognized and Joint Committee for Traceability in Laboratory Medicine (JCTLM)-listed. The development of new candidate reference methods is also recommended.

Calibration 2.000 is a national endeavor and is progressed by joint effort and collaboration among different laboratory specialisms after creating national support. Dutch laboratory specialists are excellent organizers, firm in their principles, consensus oriented and scientifically engaged. The EQA-sections involved in standardization/harmonization are all occupied by volunteering laboratory professionals who want to push standardization and harmonization forward, for the sake of better patient care with the focus being on quality. Also, medical laboratories are mostly led by professionals who are specialists in laboratory medicine and carry end-responsibility for the services of medical laboratories.

Education is a major aspect of Calibration 2.000 focused on specialists in laboratory medicine and medical laboratories. All SKML sections have closed quarterly meetings but share expertise during yearly conferences attended by specialists in laboratory medicine and also by technicians and quality officers involved with EQA. The Calibration 2.000 steering committee has biannual meetings which provide a podium for aligning standardization efforts and sharing expertise. Education consists of lectures, courses, and symposia on topics such as standardization, harmonization, method standardization, validation procedures and explanation of EQA results. Also the lay out and parameterization of the EQA system contributes to education, e.g. the method definition for enzymes is restricted to two method groups, for methods that are either IFCC traceable or not. This gives laboratory professionals insight and triggers them to opt for IFCC-traceability.

Toolbox

Tools to reach equivalence are (1) commutable trueness verifiers and harmonizers, (2) value assignment and (3) checks.

1. Commutable materials were produced from fresh frozen patient samples. Commutability was assessed originally in a twin study design [14] and more recently according to CLSI EP 30A (formerly C53-A) [15].

2. Value assignment of the materials was done according to the traceability chain using internationally recognized reference methods, reference materials and reference laboratories. Bias and imprecision criteria are derived from the Milan conference models. In addition the Six Sigma metric was introduced as a measure of allowable error.

3. Checks to analyze conformity to the preset criteria were constituted in two systems, (1) an ongoing national EQA scheme using multi sample evaluation, and (2) provision of trueness verification sample sets for individual laboratories for local analytical performance checks.

With the above described approach and toolbox for standardization/harmonization of measurands, the Dutch EQA scheme organization SKML established a Category 1 EQA system [8], [16], according to the criteria described in the ‘Roadmap to Harmonization’ paper [17], [18]. According to the definition of a Category 1 EQA scheme [18] the SKML scheme uses commutable native sera with value assignment by reference methods and allows for verification of trueness and precision of IVD tests based on predefined tolerance limits of measurement errors to evaluate whether medical tests are fit-for-purpose [19]. It incorporates the following:

1. Trueness verification materials are offered to laboratories for validating/verifying new or reset methods or instruments.

2. Analytical performance is assessed by the MUSE scoring system [8]. The system assesses both the performance of participating laboratories and their methods. Poor (and adequate) performance can result from poor (and adequate) laboratory performance or from the method performance. For example, the Jaffe methods for creatinine perform poorly and should be replaced for more adequate performance of the laboratory [20], [21]. Also the suitability of the analytical performance of current tests in medical laboratories, in the light of guidelines for diabetes and ischemic heart disease, was questioned [22].

3. Improvement of trueness and imprecision was and is driven by successive standardization projects, where attainable, or otherwise harmonization projects both nationally and internationally.

The Calibration 2.000 activities catalyzed the transformation of the Dutch SKML from a traditional EQA-organization with a peer group method approach to an EQA-organization that embedded the metrological traceability and commutability concepts, in line with upcoming IVD regulations, ISO-guidelines and international standardization efforts such as those coordinated by JCTLM and the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) in the last 20 years. Based on this national and holistic approach, in combination with a well thought out toolbox and EQA-design for harmonization of measurands, the current SKML-EQAS gives specialists in laboratory medicine actionable information about test performance. The toolbox and Calibration 2.000 approach to achieve equivalence of test results are summarized in Figure 1. Most items were set up in the program itself. In Figure 1 focus is on analytes for which reference methods and/or reference materials are available. For analytes where these are not in place steps towards harmonization are being set in line with ISO 17511 (under development) in which a special harmonization chain will be described. Presently in the EQA schemes expert values are being used as target values rather than consensus values. Where no such target value is available scoring is omitted in the scheme.

General chemistry

Most initiatives for harmonization and standardization have been in the field of general chemistry. The development of commutable EQA materials [33], [42], [54], [55], the first ‘Holy Grail’ materials, made evaluation of interlaboratory differences possible. In the studies it was shown that harmonization was achievable for several analytes, at first for lipids and also for serum enzymes [42], [43], [54], [55]. For lipids in national studies interlaboratory variation was shown in normal lipemic commutable samples [42]. The introduction of biological variation based tolerance limits made objective judgment of the performance of laboratories and methods possible. Wide variation with both acceptable and unacceptable performance was shown in international studies for lipids, enzymes, creatinine, electrolytes and other analytes [16], [33], [45], [56]. The joint project by the national EQA organizers in Italy, the Netherlands, Portugal, UK and Spain (INPUtS) [45] showed that the analytical performance of 11 of 17 general chemistry analytes measured in European medical laboratories met the minimum performance specifications. However, only one analyte (creatine kinase) met the desirable specifications in all countries and for all manufacturers. There were however, major differences between other analytes. There were six analytes for which the minimum quality specifications are not met and manufacturers should strive to improve their performance for these analytes. Standardization of enzyme methods towards IFCC reference measurement procedures (RMPs) has not been fully achieved as shown in the INPUtS study. Forty-four percent of results of seven enzymes on six platforms failed the desirable criterion based on biological variation. It requires ongoing efforts. Performance of laboratories and tests showed under-performance in several studies, and the necessity of better standardization of both methodology and calibration materials was stressed [20], [21], [22], [23].

HbA_1c is another test that needed harmonization [57]. Also for HbA_1c important results were achieved. An international HbA_1c network has been established for global standardization of HbA_1c. In the Netherlands, two laboratories operate the international reference method for HbA_1c and are recognized as certified reference laboratories. Practically all Dutch laboratories implemented in a coordinated action with clinical societies and patient organizations the new units of mmol/mol. The improvement in HbA_1c test performance is huge as interlaboratory CVs reduced from 30% in the early 1990s to about 3% in 2017. Stressing necessity is one thing, achieving the objectives is another. Figure 2 shows improvements reached in analytical performance in the Netherlands for enzymes, creatinine and other analytes after running the type 1 EQA-program for 5 years [44], [58]. Additional challenges were demonstrated when lipemic sera were used for testing lipid methodology [59] or high glucose sera in creatinine analysis [60]. The demonstration of poor performance of non-IFCC traceable methods for enzymes and of Jaffe methods for creatinine led to better standardization of methodology in the Netherlands [44].

Figure 2:

Analytical performance trends for 17 general clinical chemistry parameters between 2005 and 2010.

The analytes are divided in four color coded groups: three color coded groups refer to minerals, substrates and enzymes of which the EQA-materials have been value assigned with recognized reference methods (n=13); the red colored group encompasses the parameters that rely on consensus means and have not been value assigned with recognized reference methods (n=4). (A) Evolution of the degree of equivalence of test results as indicated by the between-laboratory CV (CV in 2010 is smaller than in 2005 for most analytes). (B) Evolution of the trueness component between 2005 and 2010. Criterion for bias is based on deviation from reference value in relation to the biological variation desirable criterion [1], [2]. The percentage laboratories that pass the bias criterion is larger in 2010 than in 2005. (C) Evolution of the precision component between 2005 and 2010 criterion for imprecision (within-laboratory CV) is based on biological variation desirable criterion [1], [2]. The percentage laboratories that pass the CV criterion is about equal in 2010 and 2005. (D) Evolution of total allowable error (TEa) between 2005 and 2010 Criterion for total error is based on biological variation desirable criterion [1], [2]. The percentage laboratories that pass the TE criterion is larger in 2010 than in 2005. For several parameters only 20% pass the TE criterion because of the rather strict biological variation criteria particularly for sodium, chloride, calcium and magnesium. (Reprinted from [44], with permission from the publisher Elsevier.)

Reference laboratories operating internationally recognized reference methods exist in the Netherlands for serum lipids (the Lipid Reference Laboratory of Erasmus University Hospital which is a member of the CRMLN operated by CDC), serum enzymes (Dr. Paul F.H. Franck, LabWest, The Hague) and HbA_1c (Dr. Cas Weykamp, Beatrix Hospital Winterswijk, Dr. Erna Lenters, Isala Hospital Zwolle).

Lipid and apolipoprotein standardization

Notwithstanding adequate standardization of direct high-density lipoprotein cholesterol (HDL-C) and low-density lipoprotein cholesterol (LDL-C) tests, it was demonstrated at the national level that direct HDL-C and LDL-C tests suffer from non-selectivity in hypertriglyceridemic sera, leading to significant misclassification rates [59]. As the residual cardiovascular risk after reaching the treatment goals for LDL-C, blood pressure and diabetes remains high (around 70%), alternative (apolipo)protein tests should help to unravel cardiovascular disease pathophysiology and to select better treatment targets. To that end, an IFCC working group on apolipoprotein (APO) standardization has been established which should develop a complete mass spectrometry based Reference Measurement System for multiplex apolipoprotein quantitation and phenotyping (http://www.ifcc.org/ifcc-scientific-division/sd-working-groups/wg-apo-ms/). Investigations have been started to metrological traceability in mass spectrometry-based targeted protein quantitation of the apolipoproteins A-I and B100 [61].

Endocrinology, binding assays, vitamins

The issue of harmonization and standardization was also recognized in endocrinology and binding assays as tumor markers and vitamins. Although reference methods exist in the JCTLM database for several steroid hormones, mostly based on mass-spectrometry methods, standardization is still a major problem. There is a lack of commutable materials in EQA surveys to check routine tests which are often based on immunological methods.

Peptide hormones are even a bigger challenge. A special problem in this respect are growth hormone (hGH) and insulin like growth factor (IGF-1). The diagnosis growth hormone deficiency largely depends on the hGH test result. Because of the relevance of growth hormone treatment in children with short statue, both clinically and financially, between-laboratory harmonization of results is paramount. A practical approach in the Netherlands led to such national harmonization using a national harmonizer [34], [46], [62]. This initiative is now taken to the international level in an international IFCC working group on hGH standardization, led by Eef Lentjes from the Netherlands (http://www.ifcc.org/ifcc-scientific-division/sd-working-groups/growth-hormone-(wg-gh)/).

Today carbohydrate deficient transferrin (CDT) is considered an important objective indicator of sustained high alcohol consumption and is used to support the diagnosis of alcohol abuse and dependence in medical and forensic settings, including driver’s license withdrawal and reinstatement [63]. Harmonization of the assay obviously is important. The Calibration 2.000 project and co-workers of it contributed both to harmonization [64], [65] and to the development of a reference method [63].

Another example of an analyte in need of harmonization is vitamin B6 (pyridoxal/pyridoxal-phosphate). van Zelst et al. [35] showed large within-laboratory variation for some methods, and in addition between-laboratory variation. The authors suggested that the lack of a reference method or suitable certified reference material for the measurement of vitamin B6 in whole blood is impeding the standardization or harmonization of this assay.

Protein chemistry

In various fields of protein chemistry studies were done towards harmonization. Initial studies to find commutable material for serum proteins such as albumin, α₁-antitrypsin, immunoglobulins and other proteins were done in the Netherlands by Klasen et al. [40], [66]. Successful follow-up of the initiative has found application with commutable materials, value assigned in the SKML EQA system for plasma proteins, but this has not yet been published. For other more specific proteins research focuses on method robustness and harmonization. For example, first steps are underway for hepcidin [67].

The use of NIST SRM 2921 and recombinant cTnI-based serum pools for harmonization was investigated by Cobbaert et al. [50], [68] who found the material unstable.

Coagulation

Harmonization of coagulation data is important in particular for situations including monitoring anticoagulation therapy such as prothrombin time INR, diagnosis and monitoring of factor VIII-C, for decision levels of fibrinogen, antithrombin and activated partial thromboplastin time. The SKML section on coagulation participated actively in the Calibration 2.000 project from the start and harmonization initiatives were taken for fibrinogen, Factor VIIIc and antithrombin [36], [41], [47], [48], [69]. Extensive scientific work on PT/INR standardization was done at the Leiden reference laboratory (Dr. Ton van den Besselaar), which runs the reference method for INR. PT/INR standardization activities are currently embedded in the Coagulation Reference Laboratory of the Department of Clinical Chemistry and Laboratory Medicine, at LUMC, which has initiated global standardization of the Manual Tilt Tube method for PT/INR standardization. Also, a collaboration between the Coagulation Reference Lab in Leiden, the international EQA organization for coagulation (ECAT) and IVD-industry has been set up for developing a mass spectrometry based reference measurement system for antithrombin [51].

Therapeutic drug monitoring

Recently the Calibration 2.000 tools and toolbox were also applied for standardization of TDM tests. Commutable materials were developed for carbamazepine, valproic acid and tobramycin [37], [49]. Between-laboratory studies showed that some methods were inaccurate. Future studies will indicate whether the use of the commutable materials could lead to harmonization between laboratories and methods.

Hemocytometry and flow cytometry

Initial studies indicated that commercially available materials cannot be used as a calibrator for all parameters on all of the common hematology analyzers. Most EQA samples, however, as prepared and used in the Netherlands, were commutable for all parameters and analyzers [70]. Most routine parameters including RBC, WBC, platelets and Hb are harmonized in the Netherlands. Future studies are necessary to show harmonization of the parameters RBC-distribution width and platelet-distribution width and also of WBC subclasses.

For flow cytometry of the parameters HLA-B27, CD-45 and CD-34+ lymphocytes, studies were performed and showed inter-laboratory variation [38], [71], [72], [73], [74]. The competition with PCR methodology hampered further efforts.

Microbiology

Cut-off values in the sense of positive or negative for serology parameters could have a clinical impact if there is lack of harmonization. Studies were started in this respect in line with the early onset of Calibration 2.000 [39], [75]. Also in this field further studies are needed to develop commutable materials and promote calibration.