Analytical performance specifications for external quality assessment – definitions and descriptions

Background

External Quality Assurance (EQA) is one of many processes used to ensure the analytical quality of laboratory measurements. The usual process is for an EQA provider to distribute proficiency test items or samples to participating laboratories (customers or participants), which perform a range of measurements and return the measurement results to the EQA organiser. The scheme organisers then provide a report to the participants that compares the submitted result(s) with a target value (assigned value). Analytical quality is assessed by considering the difference between the result(s) and the target value(s) assigned by the EQA scheme. EQA organisers provide analytical performance specifications (APS) that indicate whether the deviation from the target value achieved by the laboratory is acceptable. An APS is generally expressed as a number of units or a percentage deviation from a specified target, creating upper and lower acceptance limits. Other terms for APS that have been used include Performance Goals, Quality Specifications, Analytical Performance Goals, Quality Standards, Allowable Limits of Performance, Acceptability Limits and Quality Goals. The term Analytical Performance Specifications is preferred in line with the terminology used in the Milan conference 2014 [1].

Different EQA providers however use a wide range of APS for the same measurand [2], [3], [4], [5]. A survey performed by the European Organisation for External Quality Assurance Providers in Laboratory Medicine (EQALM) in 2014 showed that the criteria used to set the APS also vary widely [6], which is presumably a strong contributor to the variation in APS. It was noted that there was a wide variation in the way APS were determined for EQA schemes, indicating that different information was being conveyed to laboratories by the APS in different schemes [4]. Table 1 shows the variation in processes used to establish APS from some EQA schemes involving the authors.

There has been considerable work over the last 15 years in the general field of APS to improve their application in laboratory medicine. The concepts were initially codified in the so-called Stockholm hierarchy that outlined a structured approach to setting APS [7]. There have been a number of demonstrations of the application of these principles to EQA [5], [8], [9]. In 2014, these criteria were revisited in Milan where a set of three models were proposed for establishing APS [1]. As a follow-on from the Milan meeting, the EFLM convened a number of task and finish groups (TFG) to address issues arising from the meeting [10]. The present paper is a product of the TFG on APS for EQA.

Since EQA schemes are designed differently, the EQA APS are based on different criteria, often aiming to convey different information about assay and/or participant performance. These variations increase the complexity of making a comparison between the APS provided by different EQA organisers. There are also different designs of EQA schemes that are also relevant to interpretation of results [11].

The aim of this document is to define, describe and facilitate communication of the essential components of EQA APS in the field of quantitative laboratory medicine testing. It is our intention that this information will assist EQA organisers in establishing APS and then providing descriptions of the APS to participants and other stakeholders. This also provides a background for the development of APS, which should be set by laboratory professionals using a model for each measurand selected from the Milan consensus statement [1] using concepts that have been defined by another EFLM TFG and are now available [12].

Results for qualitative, semi-quantitative or morphological examination are not dealt with in this document. The recommendations do not apply to EQA schemes scores based on combined results from multiple measurands or non-analytical aspects, such as participation rate, delayed responses, number of amendments, and so forth.

The target audience for this document includes EQA organisers, EQA participants and potential participants, accreditation bodies, competent authorities, IVD manufacturers and laboratory professional organisations and has been prepared to be compliant with the relevant ISO documents, ISO17043:2010 [13] and ISO13528:2015 [14].

Recommendations

The recommendations below are relevant to any setting supported by EQA, including laboratory, point-of-care testing and physician office. It is recognised that multiple APS may be applied to the same result(s) for the same measurand. In this case, the information must be supplied for each APS.

The EFLM TFG-APSEQA has identified six important elements to facilitate the description and communication of APS.

Discussion and conclusions

The response of laboratories to EQA reports is influenced by the APS provided by the EQA organiser. The same result(s) may be either accepted or further investigated depending on the APS in place. Given this importance, the processes used to set APS in EQA and the communication of the use and meaning of EQA APS should be clarified by the implementation of the structured approach and terminology recommended in this paper. The elements listed above to describe APS are all considered necessary to make a fully informed assessment of the analytical performance of a laboratory based assessment of EQA results using supplied APS.

The nature of the material must be known to ensure it is appropriate for the comparison being made (element 1). If a comparison is made with results derived from different measurement techniques, including reference methods, then a knowledge of the commutability is required for correct interpretation. The details of the process for value assignment of the target is also required (element 2). An assigned value with a different traceability chain to the laboratory’s method, or with a large uncertainty, will influence the interpretation of the result. Supplying information about the uncertainty of the assigned value may also be of use to program participants. A valid comparison with a higher-order reference method using a commutable sample requires the information described in elements 1 and 2.

The data to which the APS are applied must be clearly defined (element 3). Within a program report, averages of multiple results or multiple measurements, or other ways of combining results may be handled seamlessly. However when interpreting the factors which have produced an abnormal result a clear understanding of the components is required. Additionally awareness of the structure of a program, i.e. the number of samples in a survey and the frequency of the surveys, allows each result to be interpreted in the context of the other available results.

It is also important to understand the analytical quality to which an APS is applied (element 4). Information derived from measurements of multiple samples permits the assessment of laboratory bias or imprecision, as opposed to single results where total error is assessed. The use of total error limits to assess bias, for example, could lead to a misinterpretation of the assay quality to be better than it actually is because the APS would be too wide.

A knowledge of the rationale behind setting the APS is also required for correct interpretation of the EQA result (element 5). A result inside a wider limit (e.g. regulatory) may pass this criteria, but not be optimal for patient management. Alternatively a failure to meet tighter limits (e.g. aspirational) may be due to limitations in the available methods rather than individual laboratory performance.

Finally, knowledge of the model used to establish the APS can affect interpretation (element 6). As well as the three models described in the Milan consensus, grading (e.g. optimal, desirable and minimal) is highly recommended with clear definitions of the grades required. If a method meets the optimal level relative to biological variation, or meets a defined clinical need, then spending time considering further improvement is unnecessary. However, if the APS are based on state of the art, or minimal standards for biological variation, then further improvement may be of benefit to patients. It is recognised that currently EQA organisers often use models not included in the Milan consensus (Table 1). While these should be noted as such, organisers are encouraged to base decisions on APS on the Milan models, recognising that there is progress in the field of assigning measurands to the various models.

As stated in the preceding paragraphs all the elements listed in this paper are required for a considered interpretation of an EQA result against the supplied APS. An additional reason for a detailed description of required information is to allow comparison of APS between schemes from different providers. If APS from one provider are known to be based on state of the art and from another are set based in desirable biological variation, then differences can be explained and customers can be aware of the reason for apparent different performance in different schemes.

An example of summary supporting information is shown in Table 2 based on the RCPAQAP from Australia. The table provides the required information and demonstrates that APS may differ between measurands in the same EQA scheme. The table includes a reference to a detailed description of the process used to establish the APS.

In setting APS, EQA providers should take into account the required response from participants who fail to meet specifications. This aspect reflects the intention of the information conveyed by passing/failing relative to an APS. For the same reason, EQA providers should clearly indicate when the APS elements differ between schemes or between measurands so that participants are aware of any such differences. For example in Table 2, different analytes in the same scheme use APS based on total error as well as precision criteria, with some at the optimal level and some minimal.

The ISO standard for clinical laboratories, ISO 15189, [16] requires that laboratories validate or verify the performance of a measurement procedure for the ‘intended use’. Since a participant may apply a test for a different use than was envisaged by the EQA provider, the APS of a particular scheme may be not applicable to their situation. For instance, if a laboratory applies a certain glucose test only to separate hypoglycaemic from hyperglycaemic comatose patients in the intensive care unit from its hospital, wider APS can be applicable than for other applications of glucose testing (e.g. diabetes diagnosis). As EQA organisers cannot have APS for every possible intended use of a test, laboratories are recommended to document their own required response to results if their use of the assay differs from generally expected use.

The final goal of laboratory medicine is enabling high quality medical decision making. One aspect of this is understanding the effect that the quality of laboratory data has on the manner in which it can be used in patient care. EQA data can be employed to inform how laboratory measurement results are clinically suitable and should be interpreted correctly. For example, data for a measurand from laboratories with a small variation relative to the within-subject biological variation of the measurand can be safely used to monitor a patient. Data from laboratories with a large variation relative to the within-subject biological variation will conversely increase the noise seen by the clinician such that larger changes in results might be required to be certain of a significant change in a patient’s clinical status. Similarly, results from laboratories with a clinically insignificant bias may be able to share a common reference interval or decision limit. However, laboratories with a clinically significant bias should not use common reference intervals or clinical decision limits.

In conclusion, we consider it indispensable that EQA schemes advise participants and all interested stakeholders about the nature of their provided APS in sufficient detail to allow informed decisions about the meaning of the results, as well as to allow valid comparison of APS from different EQA schemes.