What the Milan conference has taught us about analytical performance specification model definition and measurand allocation

Introduction

In general terms, analytical performance specifications (APS) are the criteria that specify (in numerical terms) the quality required for laboratory test information to satisfy clinical needs. If APS are not objectively defined and fulfilled, the variation in laboratory results may overwhelm the clinical information supplied, even potentially causing a negative effect on patient’s outcome. Although 60 years are already passed since the “Tonks’ rules” were established [1], the first 35 years of the APS story were spent in a vast and interesting discussion on both sides of the Atlantic Ocean, including some important proposals to set APS that however did not reach a global consensus [2], [3], [4], [5], [6]. Only in 1999, the International Union of Pure and Applied Chemistry (IUPAC)-IFCC conference held in Stockholm was a landmark in trying to achieve a consensus on how APS should be set, with a hierarchy of models being established [7, 8]. In the 10 years following the Stockholm conference there was however a clear realization that the hierarchy of strategies to set APS was not finality as stated in a review of progress made by two of the organizers [9]. It was also considered that the proposed approaches could be modified in the future if significant new ideas were developed and used in practice [10]. Roughly in the same period, the definition and use of the reference system concept for standardization of measurements in Laboratory Medicine became to be closely associated with the setting of targets for measurement uncertainty (MU) to make laboratory results clinically acceptable and establish what degree of quality is needed and what MU can be tolerated without jeopardizing patient safety [11], [12], [13]. A debate between the total error (TE) advocates and those conversely supporting the importance of MU, which requires special attention to the bias component and its correction, was also started, including criticisms to the common model employed to derive a limit for the allowable TE using a mathematically incorrect method relying on the sum of mutual exclusive terms [11, 14]. So that, in 2014 the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) considered timely to update the topic of APS by organizing a conference in which some strategic actions and measures should be taken [15].

The Milan Strategic Conference and its heritage

In November 2014, 215 participants from 41 countries attended the conference, organized by the EFLM in cooperation with the Institute for Reference Materials and Measurements of the European Commission-Joint Research Centre and the Research Centre for Metrological Traceability in Laboratory Medicine (CIRME) of the University of Milan. All contributions given during the conference were published in June 2015 in a special issue of Clinical Chemistry and Laboratory Medicine. In particular, the Consensus Statement describing models for establishing APS has been cited more than 470 times to date (according to Google scholar – http://scholar.google.com/) and wide acceptance of the agreed principles has occurred [16], [17], [18]. Immediately after the conference, five Task Groups (TG) were established dealing with the main topics of the conference. Among them, one worked on the criteria for assigning different measurands to each of the three Milan models, contributing a fundamental step towards understanding model philosophy and proposing practical principles for how to allocate measurands to different models [19]. Here, it seems useful to employ definitions provided by this TG to discuss about the Strategic Conference heritage.

The outcome-based APS model should be applied “when the measurand has a central and well-defined role in the decision making of a specific disease or a given clinical situation and test results should be interpreted through established decision limits” [19]. Regarding this model, although the Strategic Conference identified in principle two possible types of studies (direct and indirect outcome studies), in the following experience it has become clear that direct outcome studies are unfeasible or impractical, and even not necessary when: a) the clinical decisions associated with the test results are well defined; b) evidence about the diagnostic accuracy of the test to classify patients for these clinical decisions is available and is generalizable to the patient population; c) the consequences of correct/incorrect classification (i.e., true and false positives and negatives) are established [20]. This awareness led to a re-evaluation of studies using indirect methods for assessing the impact of measurement variability on clinical outcome. In doing this, Smith and coworkers showed that a significant number of studies was already available before the Milan conference and should simply be “taken out of the drawer” [21]. Among others, the pioneering contributions of Per Hyltoft Petersen approaching the APS issue by applying distributions of both diseased/non-diseased patient data and simulating the influence of analytical variability on the disease misclassification, still have an important part [22], [23], [24], [25]. More recently, other measurands with a central diagnostic role (e.g., cardiac troponin and urine albumin) have been the object of simulation studies evaluating the effects of varying the analytical performance on the patient misclassification [26, 27]. Simulations based on the current thresholds employed in clinical decision making for the definition of the acceptable measurement error have also been done [28].

According to Ceriotti et al. [19], the biological variation (BV)-based APS model should be applied to measurands with high homoeostatic control (e.g., plasma ions) or in a “steady state” status when a subject is in good health (e.g., serum creatinine). Setting APS from BV data uses concepts proposed 25 years ago by Fraser and coworkers [29]. Before and during the Milan conference, criticisms were however raised about the general quality of available BV studies and the need for a set of standard attributes highlighted [30]. Major caveats were related to: a) differences between biology vs. pathophysiology, i.e., the use of diseased individuals for BV data derivation in which the inherent BV may be significantly amplified by the disease (and treatment) related variability [31]; b) statistical drawbacks related to an ignored non-Gaussian distribution of the data (needing mathematical transformations for calculations), resulting in too high biological CV estimates [32] or to a poor study power to deliver robust data with narrow confidence intervals [33]; c) analytical issues, such as different method selectivity or insufficient method sensitivity to detect measurands present in plasma at very low concentrations in all samples of all subjects enrolled in the BV protocols [27, 34]. This is why a dedicated TG was established by EFLM for operating a careful appraisal of available BV studies by using a detailed checklist and generate an exhaustive database on the EFLM website [35], [36], [37], [38]. It has been however noted that it is not acceptable to use the BV-based model to derive APS for all measurands just because the BV information is now more easily obtainable from the mentioned database: in particular, the model 2 should not be used for measurands having not sufficient steady state status, such as, for instance, most hormones [39].

Defined by the Milan consensus as the highest level of analytical performance technically achievable, the “state of the art” (SA) should be used when a measurand has neither central diagnostic role nor strict homoeostatic control [19]. For example, this model does apply for most urinary measurands, for which the concentrations vary to maintain the corresponding plasmatic concentrations stable, compensating for dietary provision, water supplementation or loss, etc. In addition, the model 3 can be temporarily used also for those measurands still waiting for the definition of outcome-based APS or while waiting for robust BV data [19]. On the other hand, it is important to pointing up that the myth of SA as a ‘rescue’ model when APS correctly obtained with other more appropriate models for a certain measurand appear too stringent should be on principle dismantled [40]. Even if this opinion is not shared by everyone, using APS derived from the correct allocation of measurands in different models has been shown helpful in identifying measurands that need analytical improvement for their clinical use. The case of plasma electrolyte measurements recently described showed the efficacy of using objectively derived APS in driving laboratories to improve the quality of provided results [41]. Recognized drawbacks for this model, partially already highlighted during the EFLM conference, are: a) the difficulty to define how good the ‘highest’ is; b) some dependency on industry defined quality, and, more importantly, c) the lack of direct relationship between what is analytically achievable and what is clinically needed, potentially leading to unnecessary improvements. A proposal to define SA best MU performance according to the definition provided by the Milan conference and the ISO/TS 20914 technical guidance for the estimation of MU [42], taking serum C-reactive protein as an illustrative measurand, has been published [40, 43].

Table 1 summarizes the essential points representing the EFLM Strategic Conference heritage. One can consider as a part of this heritage also the fact that, in the last few years, International Organization for Standardization (ISO) standards and guidance dealing with metrological traceability in Laboratory Medicine started to explicitly emphasize the importance of APS. While ISO 17511:2020 made a direct reference to the Milan conference in the Note 1 of clause 4.3.2 [44], ISO/TS 20914:2019 widely summarized the conference deliberations under the paragraph 5.2 dealing with the concept of maximum allowable MU [42].

Moving from theoretical principles to practice

Nine years after the Milan conference it is now time to identify the approaches that permit us to become more concrete. First, the concepts described in the previous paragraph should be applied for all measurands used in the clinical setting. We should extend the list: any single laboratory measurand should be allocated to a specific APS model. Second, based on the chosen model, APS values should be established and put into practice. This is an ongoing process that would require some kind of official endorsement from professional laboratory bodies. In doing this, we should acknowledge that not all the established desirable APS will be immediately reachable, but this approach will highlight which limitations of the current technology should be prioritized and solved. Very helpful is grading different quality levels of APS to define priorities in ameliorating actions (Figure 1) [17]. Even if the derivation approach of this grading has sometimes little scientific basis, the possibility to elaborate APS at different levels of quality to move, in case, from desirable to minimum APS when the requirements are particularly stringent may help to accept the described theory and, on the other hand, represent a stimulus for involved stakeholders to work for improving the quality of assay performance when needed.

Figure 1:

The stairway of analytical performance specifications: grading quality levels helps to define priorities in ameliorating actions by stakeholders sharing the responsibility of laboratory performance, i.e., in vitro diagnostic manufacturers and medical laboratories themselves.

Figure 2 depicts the workflow for assignment of a measurand to an APS model as defined by Ceriotti et al. following the Milan conference [19]. The recently published APERTURE study, a project for establishing Analytical Performance Specifications for Measurement Uncertainty of most common laboratory measurands, has used this flowchart to allocate different measurands to one of the three APS models [18, 27, 39, 45], [46], [47], [48]. Afterwards, both minimum and desirable quality levels of APS for standard MU of clinical samples were defined by using information obtained from available literature preliminarily checked in terms of robustness. A list of information related to the selected measurands, model allocation, and APS definition is available in Table 4 of reference [18]. The related figures were then adopted by the Task Force on Reference Measurement System Implementation of the Joint Committee on Traceability in Laboratory Medicine (JCTLM) in their ongoing work to produce a synopsis of JCTLM-listed higher-order certified reference materials and reference measurement procedures for selected measurands, including their main characteristics for implementing metrological traceability and fulfilling (or not) the APS for suitable MU [49].

Figure 2:

Workflow for assignment of a measurand to one of the analytical performance specification models as defined in the 2014 Milan conference. Adapted from Ceriotti et al. [19].

What further developments are expected in the APS field?

Drugs represent a special category of measurands that probably needs a dedicated approach for deriving APS, based on fundamental pharmacokinetic theory and average elimination half-life of the drug [50, 51]. Based on these concepts, Braga et al. have proposed a kind of hybrid model, mixing some characteristics of Milan models 1 and 2 [45]. Indeed, although concentrations of drugs do not fluctuate randomly around a homoeostatic set point, the proposed approach has a relationship with biological knowledge. On the other hand, therapeutic drug monitoring (TDM) is also linked to the patient outcome in defining the concentrations of drug which are potentially toxic or when the treatment can be ineffective. Using this conceptual framework, APS for MU in TDM of immunosuppressive drugs were recently established [52].

In the next future, computerised approaches using the framework of simulation studies will become more popular and definition of APS according to indirect outcome-based studies less difficult [21]. As an example, by using a modification of the typical error model published in 2001 by Boyd and Bruns [53], the EFLM Working Group on Accreditation and ISO/CEN Standards has quite recently made up a tool for setting APS for MU of measurands that should be allocated to Milan model 1 [54]. This process can be directly implemented by laboratory professionals using a freely available web application (https://hikmetc-apscalculator.streamlit.app/), providing that users have test value data of the population of clinical interest and the evidence that the underlying data are a reliable proxy for the truth [21]. Multiple verifications of the software by other investigators are expected in the next future. Other studies have employed contour plots to present findings from which to derive APS information according to a given rate of clinical misclassification [55, 56].

An important aspect that relates with practical application of objectively defined APS is their introduction as mandatory feature in the External Quality Assessment (EQA) programs for the evaluation of the performance of participating laboratories in terms of inter-assay harmonization of the provided results and, ultimately, of the suitability of laboratory measurements in clinical settings [11, 13, 18, 34, 57], [58], [59], [60]. Just after the Milan conference, Jones et al. highlighted the wide variation in the definition of APS used by EQA programs and the huge difference in the impact of provided information [61]. Consensus has not yet been reached on this topic. However, taking as reference the almost classic EQA program categorization proposed by Miller and coworkers [62], it is probably time to integrate the EQA category 1–2 by adding a letter that identifies which model is employed by the EQA organizers to define APS, i.e., 1–2 A when one of the Milan models is used; and 1–2 B when other models, e.g., statistic-based, are employed [58, 59]. Validation/acceptability criteria for the Internal Quality Control should also be redesigned by using appropriately derived APS [18, 63].

Conclusions

As discussed in this paper, in the last nine years a considerable work to underpin APS according to the Milan consensus has been performed. However, we should generally recognize that many aspects remain predominantly theoretical. It is still a question whether APS theory as agreed during the Milan conference will be enough to convince stakeholders that change is needed. Therefore, now we need to be more functional and make a quantum leap forward in the way of practicality. According to roles and actions described in Figure 3, stakeholders are expected to work together in the field of APS for improving the quality and suitability of contributions of Laboratory Medicine to patient care. Particularly, pressure from the laboratory professionals will be crucial to invoke change and an APS list supported by the professional bodies in the field should be welcomed for helping to identify which tests (a minority) require an actual improvement.

Figure 3:

Roles and main actions expected from each stakeholder for contributing a quantum leap forward in the way of practicality of Milan consensus about analytical performance specifications. APS, analytical performance specifications; IVD-MD, in vitro diagnostic-medical device; IQC, internal quality control; EQA, external quality assessment.