A new integrated tool for assessing and monitoring test comparability and stability

Performing accurate measurements, comparable over time and location and across assays, is essential to ensure appropriate clinical practice [1]. One step towards achieving this goal is using assays that are metrologically traceable to a higher order reference measurement system or harmonized using internationally recognized procedures [2, 3]. In Europe, diagnostic manufacturers are required by the European Union Directive on in vitro diagnostic medical devices to demonstrate the metrological traceability of their analytical systems [4]. Thus, in principle, laboratories using CE-marked homogeneous systems (calibrator, reagent and instrument from the same manufacturer) can assume accuracy and interchangeability of their measurements results. However, it is the task of clinical laboratory profession to verify that the alignment process has been correctly implemented and that the performances of marketed systems are really appropriate for their clinical use. In particular, the laboratory has to verify the consistency of the manufacturer’s declared performance during routine operations performed strictly in accordance with their instruction, and to participate in external quality assessment (EQA)/proficiency testing (PT) schemes structured so that they provide objective information on the analytical quality of measurements performed by clinical laboratories and on the performances of the peer group assays [5–7]. EQA programs using commutable materials and reference method target values are becoming more widespread, but until now they are applied only to few constituents (i.e., hemoglobin A_1c) [8, 9]. Generally, an EQA program for clinical biochemistry on serum, performed by an independent national or regional center, with a sufficient number of participants so that the peer groups can be well defined, consists in a panel of about 25–35 measurands. Even if controls materials are made ad hoc, namely to evaluate clinically important concentrations, the commutability for all measurands and diagnostic systems is difficult to be achieved. Moreover, being target values by reference methods available for a few measurands and extremely expensive, it is very difficult to get a reference target values for all measurands. Finally, just for the intrinsic design of an EQA program, the data are not available in real time. In this issue of Clinical Chemistry and Laboratory Medicine, the team of Thienpont and colleagues, whose remarkable dedication to the improvement of quality, standardization and harmonization of clinical laboratory testing is well appreciated at an international level and by the readers of the journal [10–14], describes a new project aimed to establish a bottom-up cooperation between laboratories and manufacturers, so that they can pursue the common objective of assessing test comparability and stability of laboratory results [15]. The “Empower” project, proposed as an independently operated “online” tool that should monitor comparability and long-term stability between peer groups and laboratories, comprises four pillars: 1) Master comparison with panels of frozen single-donation samples; 2) Monitoring of patients percentiles; 3) Monitoring of IQC data, both across laboratories and manufacturers; 4) Conceptual and statistical education about analytical quality in the medical laboratory and elaboration of actionable experiments for analytical quality management and assurance. Laboratories are free to participate in all pillars of the project or to select the most appropriate one(s) for their purpose and the paper reports the status of the project with respect to the first two points. The focus of the master comparisons, which are conducted across assays and laboratories, is on how well the intrinsic analytical quality of assays released by the respective manufacturers is reproduced by the end users under “field” conditions all over the world. Results about master comparison with a panel of 20 frozen single donation samples for calcium, magnesium, albumin, total protein [16, 17], sodium [18], creatinine, glucose, phosphate, uric acid, total cholesterol, HDL cholesterol, LDL cholesterol and triglycerides [19], have been previously described. In a recent survey AST, ALT, LDH, GGT and ALP, sodium, potassium and chloride have been added. A limitation of these studies was that for the most measurands the target values were the “all method trimmed mean” rather than obtained using reference methods (with the exception of total cholesterol, creatinine and uric acid). However, it should be highlighted that the primary focus of the study was to evaluate the differences between methods on real patient materials and avoiding any commutability issue, rather than the accuracy of the assays. Another limitation of these studies was the narrow concentrations range covered by the single donation samples, leaving open the question about the analytical performances at other concentrations that, in fact, should be observed in real patient samples. However, these specially designed surveys (which distribute a relatively small number of single donor samples to a relatively small number of laboratories representing the major homogeneous systems), showed that, even for “simple” clinical chemistry measurands, the standardization status of certain assays is still a matter of concern and there is much room for improvement. The data further showed that assays, for some tightly regulated measurands, do not meet the optimal bias limits necessary for clinical use and this also applied for cholesterol and creatinine, in spite of dedicated standardization programs, such as the National Cholesterol Education Program and the National Kidney Disease Education Program [20, 21]. It is very important to emphasize that these surveys must be complementary to those of independent national or regional EQA schemes, which distribute commutable controls materials with a range of concentration for each measurand covering the main clinical situations, among a large number of participants. The paper published in this issue focuses on the second pillar of the “Empower” project: the monitoring of patients percentiles to which currently 124 laboratories participate with 250 devices. The laboratories calculate instrument-specific daily medians from outpatient results and send the data (by e-mail or electronic transfer) to the author’s database. Via a user interface the participating laboratory can plot for each measurand the course of the moving median. Each plot also shows the long-term median of the concerned individual laboratory, as well as the peer group or all devices median [22]. Additional numerical information are provided on the long-term imprecision (robust CV, %) and bias calculated in comparison to a “desirable” target, such as the medians of the reference intervals determined in the trueness-based “Nordic Reference Interval Project (NORIP)” [23]. Laboratories can use their medians as a tool to monitor the mid- to long-term stability of their own calibration status, in comparison to their peer, and/or to uncover shifts/drifts and the sources thereof. Another asset of percentile monitoring design is that it shows the instrument-specific stability in one plot allowing laboratories to monitor the interchangeability of results among different instruments, and detect the occurrence of instrument-specific special events. Results from patient percentile monitoring show that laboratories with high daily throughput and/or low variation in patient population typically perform with low variation and mostly good concordance between the different instruments while laboratories with a lower throughput or higher variation in patient population (typical for laboratories operating in a medium-size hospital) have a higher long-term variation in performance. Observations about drifts or shifts, or transient to long-term bias of the laboratory compared to its peer, or between different instruments used in a laboratory are very interesting. For example, shifts applied to several laboratories belonging to the same peer are an index that a major manufacturer event is occurred, i.e., a reagent or calibrator lot change. The major contribution of the “Empower” project is that it works with data generated from real patients samples and can be linked to observations in daily IQC practice. Obviously, it will not solve all of the problems that laboratory medicine face in trying to achieve interchangeability and accuracy of clinical laboratory test results, but the Empower database can become a source for “big data mining” and contribute to the definition of common reference intervals and reliable clinical decision limits. It is very important to emphasize that monitoring the patients medians is not a substitute for daily IQC but is a complementary observation tool from patient data that can cover much longer observations time. In a pilot study [12], for measurands with season-independent concentrations (i.e., calcium, phosphate, FT4, and TSH), it was demonstrated that laboratories can improve quality assurance by mid- to long-term quality management from patient percentiles. This additional IQC practice, in fact, may help to early alert the laboratory to potential problems with a direct impact and return on clinical decision making. The online access to the same data might promote the dialogue between laboratorians and manufacturers as the authors advocate the “Empower” project as a new integrated tool for quality management intended to establish cooperation between the two main stakeholders. This additional quality tool may help in the identification of root causes for analytical instability so that adequate corrective actions can be taken at the side of both the laboratories and/or the manufacturers.