Volume 53, Issue 6 - Special issue: 1st EFLM Strategic Conference / “Defining analytical performance goals – 15 years after the Stockholm Conference”

The setting of analytical quality specifications in laboratory medicine has been a topic of discussion and debate for over 50 years: 15 years ago, as the subject matured and a profusion of recommendations appeared, many of them from expert groups, it was realised by a number of leading professionals that there was a need for a global consensus on the setting of such specifications. The Stockholm Conference held in 1999 on “Strategies to set global analytical quality specifications in laboratory medicine” achieved this and advocated the ubiquitous application of a hierarchical structure of approaches. The hierarchy has five levels, namely: 1) evaluation of the effect of analytical performance on clinical outcomes in specific clinical settings; 2) evaluation of the effect of analytical performance on clinical decisions in general using a) data based on components of biological variation, or b) analysis of clinicians’ opinions; 3) published professional recommendations from a) national and international expert bodies, or b) expert local groups or individuals; 4) performance goals set by a) regulatory bodies, or b) organisers of external quality assessment (EQA) schemes; and 5) goals based on the current state of the art as a) demonstrated by data from EQA or proficiency testing scheme, or b) found in current publications on methodology. This approach has been much used since its wide promulgation, but there have been ongoing criticisms and new developments. The time seems right for an objective reappraisal of recommended strategies to set analytical performance goals.

The 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine proposed a simplified hierarchy for setting analytical performance specifications (APS). The top two levels of the 1999 Stockholm hierarchy, i.e., evaluation of the effect of analytical performance on clinical outcomes and clinical decisions have been proposed to be replaced by one outcome-based model. This model can be supported by: (1a) direct outcome studies; and (1b) indirect outcome studies investigating the impact of analytical performance of the test on clinical classifications or decisions and thereby on the probability of patient relevant clinical outcomes. This paper reviews the need for outcome-based specifications, the most relevant types of outcomes to be considered, and the challenges and limitations faced when setting outcome-based APS. The methods of Model 1a and b are discussed and examples are provided for how outcome data can be translated to APS using the linked evidence and simulation or decision analytic techniques. Outcome-based APS should primarily reflect the clinical needs of patients; should be tailored to the purpose, role and significance of the test in a well defined clinical pathway; and should be defined at a level that achieves net health benefit for patients at reasonable costs. Whilst it is acknowledged that direct evaluations are difficult and may not be possible for all measurands, all other forms of setting APS should be weighed against that standard, and regarded as approximations. Better definition of the relationship between the analytical performance of tests and health outcomes can be used to set analytical performance criteria that aim to improve the clinical and cost-effectiveness of laboratory tests.

Background: In the general classical model for diagnoses based on a single analytic component, distributions of healthy and diseased are compared and several investigations of varying analytical performance on the percentage of misclassifications have been published. A new concept based on an alternative type of diagnosing, based on sharp decision limits has been introduced in diagnostic guidelines, but only a few publications on investigation of analytical performance have been seen. Methods: The two diagnostic models (bimodal and unimodal) based on natural logarithmic Gaussian distributions are simulated. Results: In the bimodal model it is possible to evaluate the influence of prevalence of disease in combination with varying analytical performances. In the unimodal model the prevalence is pre-decided by the chosen decision limit. In this model the influence of analytical performance is investigated for diagnosing diabetes using haemoglobin A 1c (HbA 1c ), and for patients with high and low risk for coronary heart disease defined by serum-cholesterol concentrations. Conclusions: For HbA 1c the guidelines and recommendations define a maximum inter-laboratory coefficient of variation of 3.5%, but this is in DCCT units (without a true zero-point), so after transformation to IFCC units (which are proportional) it was 5.2%, which allows for analytical bias as high as approximately ±9%. Consequently, analytical quality specifications should be separated as maximum bias and imprecision.

Analytical performance specifications can be based on three different models: the effect of analytical performance on clinical outcome, based on components of biological variation of the measurand or based on state-of-the-art. Models 1 and 3 may to some degree be combined by using case histories presented to a large number of clinicians. The Norwegian Quality Improvement of Primary Care Laboratories (Noklus) has integrated vignettes in its external quality assessment programme since 1991, focusing on typical clinical situations in primary care. Haemoglobin, erythrocyte sedimentation rate (ESR), HbA 1c , glucose, u-albumin, creatinine/estimated glomerular filtration rate (eGFR), and Internationl Normalised Ratio (INR) have been evaluated focusing on critical differences in test results, i.e., a change from a previous result that will generate an “action” such as a change in treatment or follow-up of the patient. These critical differences, stated by physicians, can translate into reference change values (RCVs) and assumed analytical performance can be calculated. In general, assessments of RCVs and therefore performance specifications vary both within and between groups of doctors, but with no or minor differences regarding specialisation, age or sex of the general practitioner. In some instances state-of-the-art analytical performance could not meet clinical demands using 95% confidence, whereas clinical demands were met using 80% confidence in nearly all instances. RCVs from vignettes should probably not be used on their own as a basis for setting analytical performance specifications, since clinicians seem “uninformed” regarding important principles. They could rather be used as a background for focus groups of “informed” physicians in discussions of performance specifications tailored to “typical” clinical situations.

The aims of this study are: 1) to use the data included in the biological variation (BV) database to address the usability of BV estimates; and 2) to use different examples from the authors’ laboratories to illustrate the use and the usefulness of BV data in laboratory medicine. The BV database is an essential tool for laboratory management. Examples of application of data derived from BV are given in this paper, such as analytical performance specifications that have been included in various quality control software designed to optimize operative rules; also they have been incorporated as acceptability limits in external quality assurance reports. BV data from pathological status are of utmost interest for monitoring patients and differences between the intra-individual coefficients of variation (CV I ) estimated from healthy and patients are shown. However, for a number of analytes there are limited data available and for many there are no data, consequently new studies should be encouraged at an international level. In addition, developing international criteria to evaluate publications dealing with the estimation of BV components would be of the utmost interest. We are ready and willing to collaborate with such worthy initiatives. The first EFLM strategic conference on analytical performance specifications is an excellent opportunity for debating these ideas.

Background: Biological variation (BV) data enable assessment of the significance of changes in serial measurements observed within a subject and are used to set analytical quality specifications. This data is available in a database held in Westgard website (http://www.westgard.com/biodatabase1.htm). Some limitations of this data, however, have been identified in recent published reviews. The aim of this paper is to show the reliability of the published BV data and to identify ongoing works to address some of its limitations. Methods: The BV data currently hosted on the Westgard website was examined. Distribution of measurands stratified by the number of cited references upon which the database entry is based and the distribution of papers stratified by publication year, are shown. Moreover, BV data available in literature for glycated hemoglobin, C-reactive protein, glycated albumin, alanine aminotransferase, aspartate aminotransferase and γ-glutamyl transferase are evaluated. Results: The results obtained show that most BV data come just from a few papers or only one paper and that a lot of publications are dated, therefore this data is too obsolete to be used. Furthermore critical review of the BV database highlights a number of factors that might impact on the reliability of the BV data entries and translation into current practice. Conclusions: A number of issues clearly undermine the value of the current database. These issues are being considered by the European Federation of Clinical Chemistry and Laboratory Medicine, biological variation working group, in collaboration with a Spanish group responsible for the database updating.

Data on biological variation are used for many purposes in laboratory medicine but concern exists over the validity of the data reported in some studies. A critical appraisal checklist has been produced by a working group established by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) to enable standardised assessment of existing and future publications of biological variation data. The checklist identifies key elements to be reported in studies to enable safe accurate and effective transport of biological variation data sets across healthcare systems. The checklist is mapped to the domains of a minimum data set required to enable this process.

The organizers of the first EFLM Strategic Conference “Defining analytical performance goals” identified three models for defining analytical performance goals in laboratory medicine. Whereas the highest level of model 1 (outcome studies) is difficult to implement, the other levels are more or less based on subjective opinions of experts, with models 2 (based on biological variation) and 3 (defined by the state-of-the-art) being more objective. A working group of the German Society of Clinical Chemistry and Laboratory Medicine (DGKL) proposes a combination of models 2 and 3 to overcome some disadvantages inherent to both models. In the new model, the permissible imprecision is not defined as a constant proportion of biological variation but by a non-linear relationship between permissible analytical and biological variation. Furthermore, the permissible imprecision is referred to the target quantity value. The biological variation is derived from the reference interval, if appropriate, after logarithmic transformation of the reference limits.

Performance criteria should be a challenge for the laboratories to improve their quality. In countries with mandatory proficiency testing, the definition of performance criteria is a particular issue. If the definition of performance criteria is mandated from the regulatory bodies to medico-scientific institutions, scientific approaches (i.e., based on biological variation), the state-of-the-art approach (i.e., based on technical feasibility) as well as medical needs can be used to set up performance criteria such as the Richtlinie der Bundesärztekammer (RiliBÄK; Guideline of the German Medical Association on Quality Assurance in Medical Laboratory Examinations) in Germany. The experiences with RiliBÄK show that these performance criteria have to be revised on an ongoing basis.

The concept of metrological traceability of measurement results to property values assigned to measurement standards of higher metrological order or to the International System of Units (SI) through sequential calibrations, using reference materials and reference measurement procedures, plays a key role in ensuring that end user measurement procedures perform at an acceptable level in the clinical context. The aim is that measurement results produced over time or by different end users or with different end user measurement procedures for the same measurand will be equivalent within their corresponding uncertainties. These goals can only be reached under certain conditions and if requirements laid down in international standards on calibration concepts, reference measurement procedures and reference materials are fulfilled. Calibration hierarchies have to be implemented correctly and parameters contributing to measurement uncertainty and systematic bias need to be controlled and eliminated, respectively, by technically improving methods and reference materials and intermediate calibrators used for effectively achieving equivalence of measurement results and for meeting analytical performance requirements for in vitro diagnostic devices.

The measurement uncertainty budget should combine the uncertainty of higher order references, the uncertainty of commercial system calibration, the system imprecision and individual laboratory performance in terms of variability. Here we recommend that no more than one third of the total uncertainty budget, established by appropriate analytical performance specifications, is consumed by the uncertainty of references and approximately 50% of the total budget consumed by the manufacturer’s calibration and value transfer protocol. The remaining 50% should be available for the commercial system imprecision (including the batch to batch variation of the reagents) and individual laboratory performance in order to fulfil the uncertainty goal. For commercial systems to work properly, in vitro diagnostics (IVD) manufacturers will need to take more responsibility and ensure the traceability of the combination of platform, reagents, calibrators and control materials for system alignment verification that only as such (as a whole) are certified (“CE marked”) by the manufacturer itself in terms of traceability to the selected reference measurement system. Particularly, IVD manufacturers should report the combined (expanded) uncertainty associated with their calibrators when used in conjunction with other components of their analytical system (platform and reagents). This is more than what they are currently providing as traceability and uncertainty information.

Background: Internal quality control (IQC) is an everyday practice described in several documents. Its planning requires the definition of quality goals and a documentation system able to provide alarms as soon as the goals are not reached. We propose the use of the uncertainty approach to develop an effective alarm system. Methods: The use of the uncertainty information to verify the conformity to specifications is described. A top-down approach to the definition of the uncertainty of the method is described. Once the uncertainty is calculated, the complete measurement result (result±expanded uncertainty) is compared with the maximum permissible error (quality goal). An alternative and more immediate presentation is obtained defining an “acceptance zone” derived from the maximum permissible error reduced on either sides by expanded uncertainty. This approach is applied to two analytes: glucose and creatinine. Results: The relationship between quality goal and expanded uncertainty defines the width of the acceptance zone; if uncertainty is equal or larger than the quality goal, the goal is not attainable. Conclusions: The proposed approach uses an information, expanded uncertainty, that each laboratory seeking ISO 15189 accreditation should already have. The data presentation is immediate and easy to interpret allowing a direct comparison between the performance of the method and the quality goals.

External Quality Assurance (EQA) is a vital tool in laboratory medicine to assess individual laboratory analytical performance and also the differences between the results from different laboratories. This information is also useful for professional bodies and manufacturers as part of post-market surveillance. The process involves the measurement of one or more samples by many laboratories and then assessment of the results. Individual results are generally assessed by how far they lie from a target, which may be established using reference methods or a median of some or all of the submitted results. The distance of a result from the target is compared with analytical performance specifications in order to assess the analytical quality. One of the uses of the Stockholm hierarchy of performance goals is to set the performance specifications for analysis of EQA results. Fifteen years after the Stockholm consensus meeting, EQA analytical performance specifications appear to still vary widely between EQA providers. This can be due to a range of factors, including the rationale for setting the criteria, the expected response to a failure to meet the specified performance, the clinical meaning behind meeting the specifications, and the possible need for further analytical improvements. There are also differences in the models chosen to set the criteria, usually either state of the art or biological variation, and then differences in how these are applied. While harmonisation of EQA performance specifications may be some time off, all EQA providers should define the nature of their specifications and the basis for their selection and make this information available to customers.

Appropriate quality of test results is fundamental to the work of the medical laboratory. How to define the level of quality needed is a question that has been subject to much debate. Quality specifications have been defined based on criteria derived from the clinical applicability, validity of reference limits and reference change values, state-of-the-art performance, and other criteria, depending on the clinical application or technical characteristics of the measurement. Quality specifications are often expressed as the total error allowable (TE A ) – the total amount of error that is medically, administratively, or legally acceptable. Following the TE A concept, bias and imprecision are combined into one number representing the “maximum allowable” error in the result. The commonly accepted method for calculation of the allowable error based on biological variation might, however, have room for improvement. In the present paper, we discuss common theories on the determination of quality specifications. A model is presented that combines the state-of-the-art with biological variation for the calculation of performance specifications. The validity of reference limits and reference change values are central to this model. The model applies to almost any test if biological variation can be defined. A pragmatic method for the design of internal quality control is presented.

The term “qualitative test” is ambiguous and should not be used for nominal (classification) or ordinal (grading) tests. Characteristics for nominal and ordinal scale test results, such as traceability and uncertainty, remains to be established before general performance criteria can be agreed upon. For ordinal binary test with a quantitative back ground scale an assay could be characterized with the three quantities “C 5 ”, “C 50 ”, and “C 95 ”. The C 50 -value, or the equivalence point, ought to be declared by the manufacturers of diagnostic products. For the correct understanding and communication of results from ordinal and nominal tests the way of expressing the results also need to be harmonized.

The definition, implementation and monitoring of valuable analytical quality specifications have played a fundamental role in improving the quality of laboratory services and reducing the rates of analytical errors. However, a body of evidence has been accumulated on the relevance of the extra-analytical phases, namely the pre-analytical steps, their vulnerability and impact on the overall quality of the laboratory information. The identification and establishment of valueable quality indicators (QIs) represents a promising strategy for collecting data on quality in the total testing process (TTP) and, particularly, for detecting any mistakes made in the individual steps of the pre-analytical phase, thus providing useful information for quality improvement projects. The consensus achieved on the developed list of harmonized QIs is a premise for the further step: the identification of achievable and realistic performance targets based on the knowledge of the state-of-the-art. Data collected by several clinical laboratories worldwide allow the classification of performances for available QIs into three levels: optimum, desirable and minimum, in agreement with the widely accepted proposal for analytical quality specifications.

Quality in healthcare is ideally at an optimal benchmark, but must be at least above the minimal standards for care. While laboratory quality is ideally judged in clinical terms, laboratory medicine has also used biological variations and state-of-the-art criteria when, as is often the case, clinical outcome studies or clinical consensus are not available. The post-analytical phase involves taking quality technical results and providing the means for clinical interpretation in the report. Reference intervals are commonly used as a basis for data interpretation; however, laboratories vary in the reference intervals they use, even when analysis is similar. Reference intervals may have greater clinical value if they are both optimised to account for physiological individuality, as well as if they are harmonised through professional consensus. Clinical decision limits are generally superior to reference intervals as a basis for interpretation because they address the specific clinical concern in any patient. As well as providing quality data and interpretation, the knowledge of laboratory experts can be used to provide targeted procedural knowledge in a patient report. Most profoundly critically abnormal results should to be acted upon to minimise the risk of mortality. The three steps in quality report interpretation, (i) describing the abnormal data, (ii) interpreting the clinical information within that data and (iii) providing knowledge for clinical follow-up, highlight that the quality of all laboratory testing is reflected in its impact on clinical management and improving patient outcomes.