Outcome-based analytical performance specifications: current status and future challenges

Introduction

In 2014 in Milan, the 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) was dedicated to a single topic on how analytical performance specifications (APS) should be defined to guide various stakeholders involved in the delivery of laboratory service. As a result of the meeting three non-hierarchical models were proposed to replace the Stockholm criteria released in 1999: Model 1 APS based on outcomes, Model 2 APS based on biological variation and Model 3 APS based on the state of the art [1]. The Milan consensus group has also recognised the importance of the pre- and postanalytical phases of testing and encouraged users to expand quality specifications to the total testing process [1].

In this paper we focus on Model 1 APS and the related key definitions, concepts and considerations that should assist in finding the most appropriate methods for deriving APS which contribute to improved laboratory service that delivers patient benefit. We also review published methods for Model 1 APS and discuss the current status and future challenges of defining outcome-based APS for laboratory tests.

Key definitions and concepts

According to the Milan consensus, “APS are criteria that specify (in numerical terms) the quality required for analytical performance in order to deliver laboratory test information that would satisfy clinical needs for improving health outcomes” [1].

To satisfy clinical needs and improve health outcomes through laboratory testing, laboratory professionals generally tend to believe that tests that perform better analytically may lead to better patient outcomes. However, this is only true, if the better performing tests 1/improve the clinical performance of the test: i.e., they increase the rate of appropriate diagnoses and decrease the rate of missed diagnoses; and 2/this improves the clinical effectiveness of management decisions: i.e., they support appropriate treatment selection [2], [3], [4]. In the context of the above definition of APS, the primary focus is clinical performance to satisfy clinical needs; other health care related organisational or economic outcomes such as test safety, accessibility, convenience, turn-around time and costs may be important mediators for health outcomes but are not primary considerations and can be assessed separately [4].

To further dissect the actual meaning of the Milan definition, APS refer to how ‘good’ the analytical performance of a test needs to be to do more good than harm to the patient. For answering this question, a few more key concepts need clarification. How can we determine what ‘good’ analytical performance is? At this point it is important to highlight the difference between analytical performance goals and analytical performance requirements. The former is more aspirational in terms of the quality we ideally would like to achieve if we had better and more advanced technology; whilst analytical performance requirement is more of a pragmatic term to describe the analytical quality we must/should and can realistically achieve with existing technology. In the literature these two types of ‘good’ are often used interchangeably under the umbrella term of APS and that can lead to confusion by end users.

The other key concept or question is to whom ‘good’ is good enough. Analytical performance specifications are used by many stakeholders, including the IVD industry, the regulator, the health care purchaser, accrediting bodies, the external quality assurance (EQA) provider, the medical laboratory, the clinical staff including clinical guideline groups, and ultimately and indirectly the patient. The purpose of the APS for each stakeholder might be somewhat different and currently they all use different methods or criteria for defining what ‘good’ analytical performance means to them.

The relationship between the analytical and clinical performance of tests is reciprocally interdependent. By clinical performance/clinical validity we mean the ability of a test to provide information for its intended use about a health condition or state of interest in the relevant population [5]. This includes diagnostic accuracy (ability to correctly identify whether or not a condition is present; e.g. sensitivity, specificity, negative and positive predictive values) and prognostic accuracy (ability to predict whether or not an event will occur in the future; e.g. risk classification, concordance or c-statistic). Analytical performance of a measurand affects its clinical performance. However, without first setting the clinical performance requirements, it is difficult to define how good analytically the test needs to be to achieve the desired clinical performance. The clinical performance requirements may vary for the same test depending on how and where it is used and how its decision limit is established; e.g. for a triage test to rule out a condition, the test needs to have a cut-off that offers the highest possible diagnostic sensitivity; for diagnosis the test needs to have a cut-off that offers high specificity; or for prognosis a cut-off value can be set at a clinically significant or critical risk limit at which more intensive treatment or change in managing the condition is required.

According to the cyclical test evaluation framework proposed by our group in an earlier publication [6] there is an interplay between how good analytically a test is and how it can screen, diagnose, prognose or monitor a condition in a certain population. In other words, how good its clinical performance is to achieve the best outcome for patients. Crucial to all this interplay is the clinical pathway, i.e. how a test is used in making decisions about the management of the patient, which then more directly influences the patient’s health outcome. Thus, the link between testing and health outcomes is almost always indirect [7] and is dictated by the clinical pathway, and the purpose and role and the significance of the test in clinical decision making [6].

Due to this indirect link, high quality analytical performance potentially leading to higher clinical performance by itself does not lead to better clinical action or patient compliance or more effective treatment and improved health outcomes. For example, if a test with better analytical quality and better clinical performance is applied to the wrong patient or clinical scenario, or at the wrong time in the course of treatment, then there will not be higher therapeutic success or better health outcomes. On the other hand, analytically less well performing tests that play a small part in a complex clinical pathway may not necessarily lead to adverse or unfavourable outcomes [4]. Sometimes tests with relatively inferior analytical or clinical performance, but that are accessible in a rural or remote or under-resourced setting may achieve better health outcomes than better performing tests only offered in a central laboratory far removed from patient care. Point of care tests are an example, where less stringent APS are used [8]. Timely (albeit poorer quality) results may avoid irreversible patient harm due to delayed treatment, or loss to follow-up. This complexity makes setting universal APS that meet every stakeholder’s medical needs particularly hard and complicated, if not impossible, or even counterproductive [4]. This again highlights the importance of understanding how a test is used in practice and what the consequences of testing are to the patient.

How can analytical performance specifications be developed according to Milan Model 1?

Model 1a APS can be addressed by clinical studies, ideally diagnostic randomised controlled trials (RCTs), that directly look at the impact of differing analytical performance on clinical outcomes. These types of RCTs are as yet unavailable and aspirational, so most people turn to Model 1b indirect (simulation-based) approaches that assess the impact of analytical performance on the probability of clinical outcomes by investigating the impact on medical decisions for patient management (see details below; [9]) as intermediates to patient health outcomes [2].

Most Model 1b studies use simulation modelling of the impact of analytical imprecision and bias on patient classification, or how medical decisions on, for example drug dosing [10], or treatment selection or advice could be impacted. Smith et al. have systematically reviewed the literature for these indirect approaches by investigating how they assessed the impact of test measurement uncertainty (by their definition and in frequency order it included random, systematic and total analytical error) on downstream clinical performance, and operational, and economic outcomes [11]. Fifty four percent of published studies investigated tests that were used for monitoring, 42 % for diagnosis or screening, and 9 % for prognosis across 4 clinical topics of diabetes mellitus, cardiovascular disease, cancer, and metabolic or endocrine disorders. They found various indirect, mostly statistical approaches in the literature; e.g. for HbA1c that used distributional analysis [12] or regression analysis [13]; for glucose, an error model simulation [14] and decision analytic models [15]; for aminoglycoside antibiotics, error grid/contour plots [10]; and for calcium, cost curve analysis [16]. Another approach, most commonly used in health technology assessment, is the so-called linked evidence approach that uses systematically reviewed evidence on diagnostic accuracy of a medical test and investigates its impact on clinical decision making and on the effectiveness of consequent treatment options [2, 17, 18]. Several studies linked the impact of improved imprecision to downstream outcomes, via a Markov cost-effectiveness model estimating, for example, the impact of glycaemic rates on cardiac events, and subsequent mortality and quality-adjusted life-year [19].

The review of methods by Smith et al. [11] has also identified a common analytical framework underpinning the various methods. It consisted of 3 key steps: (a) assignment of “true” test values; (b) calculation of measured test values (incorporating uncertainty due to bias and imprecision); and (c) calculation of the impact of discrepancies between the true and measured value on specified outcomes. Most studies suffered from defining the ‘true’ values of a measurand from empirical data (that includes bias and imprecision) or simulated data (which are not from any real-life source). Assessing bias would be most ideal using commutable (quality assurance) materials against a target that was set by a fully traceable reference measurement procedure. However, assessing the real impact of deviation from the ‘true’ value that can lead to misclassification in real life should be done against the actual method, including its analytical and preanalytical conditions, that was used to establish the clinical decision limit around which modelling of misclassification is investigated. Therefore, appropriately conducted correlation studies on patient samples against a carefully chosen reference are more suited for such modelling as these give a more realistic assessment on the impact of bias and imprecision on classification rates around guideline driven decision limits. Another problem with data sources used in modelling studies is that due to method changes and lot-to-lot variations the modelled analytical errors also vary over time and some methods that were used when certain decision limits were established are no longer available with the advancement of technology. Therefore, local transferability of data coming from such modelling studies should be critically assessed.

The majority of these simulation models have no direct evidence on the true impact of the modelled misclassification on medical decisions and consequent patient health outcomes. The clinician always interprets the result in the context of other specific clinical and diagnostic information and weights the strength of the abnormality found. Borderline results are not considered as strongly as marked abnormalities and in the absence of borderline zones, or comments that grade the abnormality, many of the simulation studies are naive imitations of the more complex clinical decision making. Given that clinicians often use laboratory tests as adjuncts in their decision making, simulation models most probably overestimate the real-life impact of tests on patient well-being. For realistic Model 1b APS, the assumptions coming out of modelling therefore need to be tested in new studies using risk assessment or clinical audit or other forms of outcome assessment that can provide clinical evidence about the true consequences to patients.

To illustrate the complexity of deriving Model 1 APS, we have looked at the example of HbA1c [20] as one of the measurands that was allocated to Model 1 of the Milan consensus [21]. HbA1c plays an important role in and has well-defined decision limits for screening and diagnosing diabetes as well as for monitoring the progression and response of patients to treatment. We have reviewed publications that investigated how good the HbA1c test needs to be to meet medical needs [20]. We extracted the APS from 18 papers that used various Milan models. Eight of these papers used a Model 1b study of which 5 used simulation modelling, 2 statistical derivation and one used an international clinician survey that investigated at what level of change in HbA1c result clinicians would alter their management decision, and translated that to an APS. Three of the 8 Model 1b studies provided no numerical APS and the rest provided APS for analytical coefficient of variation (CVa) that ranged from 2 to 9 per cent [20]. Most papers that used statistical approaches or simulation models came up with an analytical performance specification for CVa between 2 and 5 per cent or simply provided the magnitude of misclassification rates at various levels of imprecision and bias or assumed that bias was zero. The one international physician survey from 6 countries used a case scenario to assess at what change in the HbA1c result general practitioners would change their clinical management when diagnosing or monitoring diabetes mellitus. Using their responses as the clinically significant difference the authors calculated CVa using the reference change value (RCV) concept [22] and came up with much higher APS for CVa of up to 9 per cent at 95 % probability and even a higher figure of CVa of 25 per cent at 80 % probability than the earlier mentioned approaches [9]. In that study it was also interesting to see that clinicians tolerated far less analytical error when HbA1c results were increasing, i.e. the patient was deteriorating, than when the HbA1c was decreasing, and patient was improving [9]. Given these published figures, it is truly hard for any stakeholder to decide which analytical performance is good enough for successful management of diabetic patients. Nevertheless, these findings also indicate that in some scenarios clinicians may tolerate more analytical error than what comes out of statistical models. An important recent contribution to this type of analysis comes from assessing the contribution of non-glucose effects on HbA1c, e.g. such as average red cell life-span [23]. This approach recognises the additional complexity of unknown variation in the relationship between the measured result and the clinical classification. This raises the importance of understanding the pathophysiology of the measurands and the potential to over-emphasise analytical performance when there are other unaccounted factors for variation in the results.

Limitations of indirect approaches for Model 1b APS

For the moment we do not yet have accepted methodology for deriving Model 1b APS and published studies provide heterogeneous data that are difficult to synthesize. To assess the quality of such studies, we do not have a critical appraisal and meta-analysis tool that could be used to generate estimates for APS from papers using indirect approaches. The APS provided in most papers assume that long-term bias of a method is incorporated in the MU estimate and that any bias in test results has been controlled and preferably eliminated. This assumption does not realistically reflect routine laboratory services and already known biases between methods.

An important limitation of selecting APS from indirect studies is that they model misclassification in a certain population that may not be transferable to another population or setting (see further discussion of the impact a disease prevalence and spectrum later). Furthermore, the true impact of misclassification rates presented in various modelling studies has not been assessed in practice or in a clinical setting that differs from the study population and whether the APS derived from a certain study are clinically suitable or required. Therefore, most published Model 1b APS are based on untested assumptions, and some could be too stringent, others potentially too loose.

Hyohdoh et al. conducted a study that aimed to more realistically assess when physicians action a test result in real life using a statistical approach to analyse 65 million laboratory test data from 99,000 patients and ordering patterns in medical records [30]. Using repeat test intervals as indications for considering a test result abnormal enough to warrant further monitoring or actions and linking laboratory results to therapeutic decisions of initiating the prescription of specific medications, they derived so-called ‘real-world clinical decision intervals’ [30]. These ‘real-life’ decision intervals could also be translated to clinically acceptable deviations of laboratory results from reference intervals or decision limits – in other words values that could be used similarly to the clinically significant difference mentioned above, but that triggered action in real life, rather than being based on a few experts’ opinion. This approach is linked to the concept of the clinical importance of misclassification. For example, wrongly classifying a patient with an HbA1c of 6.6 % as not having diabetes is not as important as to do so for a patient with a value of 7.2 %, where treatment benefit is more likely. Clinical outcome studies also have uncertainties about the decision points, and “staged” misclassification rates (e.g. likely minor, moderate, major clinical effects) may be more relevant than a total number or proportion of patients misclassified.

Irrespective of the approaches used so far, the current status of Model 1b APS is that most assess the theoretical impact of imprecision and bias (or sometimes bias is assumed to be zero) on clinical classification. Often the published models do not directly translate to APS; they simply inform about potential classification errors at a certain degree of imprecision and bias in a certain modelled, often hypothetical population which may not always be comparable to the local population in which the APS is supposed to be used. Therefore, two key questions remain: 1/How much analytical error is tolerable without severely affecting disease classification, management decisions and health outcomes? 2/How transferable these estimates are to various patient populations and settings with prevalence of disease or disease spectrum that differs from the studies which modelled the impact?

Defining minimum acceptable clinical performance

What can be considered a clinically acceptable misclassification rate is the central component of Model 1b APS. To answer this question, we need clinical consensus and, even more importantly, we need to understand how good a test needs to be in terms of clinical performance before we can decide how good it needs to be analytically to meet those clinical performance expectations.

Target product profiles (TPPs) for medical tests would be one good example of how clinical performance requirements could be set a priori which is more often done for new biomarkers. Clinical performance specifications are a set of criteria that quantify the clinical performance a new test must attain to allow better health outcomes than current practice [5]. These describe the necessary properties of a new test to address an unmet clinical need. They often start with considering the end, i.e. the health outcome of testing. They are usually produced by regulators or health care purchasers or public health institutions to guide manufacturers in the development of ‘fit for purpose’ tests [31].

Most TPPs for diagnostic tests were developed for infectious diseases. For example, in the UK the Medical and Healthcare products Regulatory Agency (MHRA) set TPPs for point of care tests for the detection of acute SARS-CoV-2 infection in people with or without symptoms. The published TPP was based on the consensus of what is ‘minimally acceptable’ in the opinion of the IVD industry, healthcare professionals and medical device regulators in the UK. They have set desirable and acceptable clinical performance criteria for diagnostic sensitivity at ≥95 and ≥80 % for ruling in the infection, respectively; and for diagnostic specificity at ≥99 and ≥95 % for ruling out the infection, respectively. For more details the reader is referred to Target Product Profile: Point of Care SARS-CoV-2 detection tests – GOV.UK (www.gov.uk). One may argue that these TPPs also suffer from subjectivity of expert opinion and lack of agreed and transparent methodology for setting clinical performance specifications. Often, the clinical performance specification is set arbitrarily by statistical ‘convention’ around 80 , 90, 95 and 99 % statistical probability or confidence.

The paper by Lord et al. provides further guidance on a more systematic approach for defining test performance that meets clinical need [5]. Using decision analytic principles, this paper provides a 5-step practical guide for developing minimum clinical performance requirements that consider the trade-off between the benefits and harms of a test and use net benefit as a measure [32]. Unquestionably, the trade-off between benefit and harm is a value judgement and therefore it also requires expert consensus. This judgment may also vary between health-care settings due to economic and organisational considerations. As pointed out earlier, for meaningful estimates of clinical performance, diagnostic accuracy of a test and the consequences of true positive and false positive or true negative and false negative tests need to be assessed in a population and in a well-defined clinical pathway that closely reflects how the test is intended to be used in practice [5]. One classic and widely known example for such value judgment has been described by Than et al. [33]. Emergency medicine physicians from the US, Canada, Australia and New Zealand were surveyed at various professional conferences and internally for the acceptable risk of false negative Troponin results when patients present with symptoms of chest pain in the emergency department. The consequence of a false negative Troponin result was defined as a missed major adverse cardiac event (MACE) within 30 days after discharge. One thousand twenty-nine clinicians responded to the survey and 40 % accepted 1 % or higher false negative result. Fifty five percent accepted 0.5 % or higher miss rate. The investigators used these data to conclude “clinicians may expect diagnostic strategies for the assessment of suspected ACS to achieve a sensitivity of 99 % or higher for AMI or other MACE.”[33].

This leads to the next key question: How transferable these estimates are to different patient populations and settings with prevalence of disease or disease spectrum that differs from the studies which modelled the impact of analytical performance on classification rates? This is an often-overlooked aspect of modelling studies, even though we have clear evidence from multiple studies that disease misclassification rates depend on the prevalence/pretest probability of the condition. In addition to the impact of prevalence on the positive and negative predictive value of a test, sensitivity and specificity may also vary. For example, with increasing prevalence, lower specificity and higher sensitivity have been observed due to differences in disease spectrum in studies [34], [35], [36]. From the above it follows that APS apply to the actual population and the diagnostic pathway in which the clinical performance requirements were derived, and the comparability of the local and published population and clinical pathway need to be assessed before using the APS.

Analytical performance specification is a moving target

The various aspects mentioned above should be carefully considered before adopting any APS based on Model 1b studies. As already emphasised, APS should respond to clinical needs, but the clinical performance requirements are also highly dependent on patient population, disease spectrum and prevalence and existing practice in addition to many other pragmatic organisational, geographical, economic and sometimes societal considerations that often influence medical laboratory practice. Therefore, APS cannot always be fully harmonised or rigidly applied without taking into account the priorities and preferences of the local health care system.

Analytical performance specifications also have constantly moving targets not only because of rapid technological developments in the laboratory field but also due to changes in the ways conditions are managed and treated with newer medications which alter the needs and expectations of clinicians and patients. One good example to illustrate this point is the APS for LDL-cholesterol that is still the recommended primary target of lipid lowering therapy [37]. According to the European Society of Cardiology (ESC) and European Atherosclerosis Society (EAS), the treatment goals for patients at very high risk and high risk of atherosclerotic cardiovascular disease are <1.4 mmol/L (<55 mg/dL) and <1.8 mmol/L (<70 mg/dL), respectively [38]. The last APS of CVa <4 %, bias ≤4 % and total error ≤12 % were developed by expert consensus of the National Cholesterol Education Program (NCEP) for higher LDL-cholesterol decision limits that guided on-treatment goals and initiation of treatment at 2.6 mmol/L (100 mg/dL) and 4.9 mmol/L (190 mg/dL), respectively [39]. Cole et al. have recently demonstrated by modelling that using the NCEP APS for risk stratification based on calculated LDL-C, up to 10 % of cases would be misclassified into a different risk group, potentially leading to mismanagement. These risk groups ranged from LDL-cholesterol of <1.8 mmol/L (<70 mg/dL) to >4.9 mmol/L (>190 mg/dL). To reduce the rate of inappropriate risk stratification, authors proposed the adoption of tighter APS by reducing the NCEP allowable bias to ≤3 % and the imprecision to CVa <3 %, which, according to the state-of-the-art, are currently achievable by most lipid methods [40, 41]. They modelled the misclassification rate after lowering the APS and found a modest reduction by up to 10 % and, unsurprisingly, that reduction in proportional bias had a larger impact than reduction in imprecision on the number of cases misclassified. Again, stricter APS seem to be justified if the true misclassification rate is high, but it would be good to have firmer evidence from clinical practice whether improved analytical performance would truly lead to better patient management and outcomes.

Future challenges

Given all the complexities and shortcomings of existing methods we need to recognise that we still have significant gaps in our knowledge and lack the tools for better defining Model 1b APS that are universally implementable. Quoting Wytze Oosterhuis [42]: “The practice of the clinical laboratory is such, that it is impossible to describe performance specifications in a mathematically perfect model, and all models will be based on assumptions and can only approach complex reality. The challenge is to reach consensus on a model that is both useful and as less flawed as possible.”

How should we fill the gap until we have better studies, least flawed methodologies and consensus that tell us how good we need to be? The below is mostly personal opinion and we raise these points to generate more discussion and broader international collaboration of key stakeholders, and facilitate further research in this area. As Model 1b APS should respond to clinical needs, one of the first steps is that we as laboratory professionals start translating our analytical performance to clinical performance metrics and apply risk-based approaches [43] in order to quantify the impact of potential misclassification in our local population. Such an activity should include the assessment of the combined impact of analytical, biological and preanalytical variations, by defining and reporting a grey zone around discrete decision thresholds, and facilitate repeat testing, when necessary, before a diagnostic or treatment decision is made.

Currently we already have a few published and freely available but as yet unvalidated statistical and interactive tools [44], [45], [46] and a large amount of real-life laboratory data that could help us inform our clinicians, guideline developers and the IVD industry about the impact of analytical performance on test accuracy and clinical decisions. These first steps would also offer an opportunity for more patient focused conversations between laboratorians, clinicians and the IVD industry and for a more concerted and globally better coordinated effort to improve the safety and quality of laboratory service in a way that better meets patients’ medical needs.