Behind the scenes of EQA–characteristics, capabilities, benefits and assets of external quality assessment (EQA): Part IV – Benefits for participant laboratories

Using EQA for verification of IVD-MD suitability

Verification requires sufficient objective evidence to determine that a marketed IVD-MD fulfils the requirements as specified by the manufacturer [9], and EQA programs are essential to the post-market surveillance of IVD-MDs [10]. To fulfil this task, EQA must meet certain requirements [8].

Firstly, the EQA material commutability must be known [11]. Use of non-commutable EQA materials prevents the direct comparison of laboratory EQA performance to the measurement performance of patient specimens [12]. Using non-commutable EQA materials has been justified for demonstrating that the employed IVD-MD is similar to other IVD-MDs using the same measurement procedure. However, as healthcare professionals, we should expand our horizon to know whether the quality of laboratory measurement results is suitable for clinical use, independent of the IVD-MD type and the simple fulfilment of the manufacturer’s specifications [13]. A working group of the European Federation of Laboratory Medicine (EFLM) has stressed the need that EQA providers should specify that material matrix and its commutability, because the interpretation of differences between results in an EQA program is strongly dependent on the nature of the survey material [7]. A recent publication by the WG-commutability in IFCC describes in detail how to assess the commutability of EQA material [14].

Assigning values to EQA materials with higher-order measurement procedures (MP) is the second requirement. For measurement results traceable to the International System of Units (SI) (as described in the three first calibration hierarchy models of ISO 17511:2020) [15], assigning values by the reference measurement procedure (RMP) to commutable EQA materials ensures the objective evaluation of performance through a trueness-based grading, therefore providing invaluable information about the correct implementation of metrological traceability and standardisation of results. It has been argued that this approach is not replicable for measurands where an RMP is lacking. The classic approach using the “peer group” EQA performance assessment is commonly used in this case. However, a drawback of this approach is the definition of “peer group”, which may be heterogeneous between different EQA providers [16].

Finally, the EQA performance should be evaluated against clinically suitable limits. Defining analytical performance specifications (APS) regarding metrological traceability is a relatively new science. A consensus conference in 2014 concluded that APS derivation should be based on three models: clinical outcome, biological variation, or state-of-the-art [17] criteria for allocating measurands to these models were elaborated, and recommendations for MU APS were proposed [18].

EQA that meets metrological criteria has unique benefits that add substantial value to the practice of laboratory medicine (Table 3), e.g., the standardisation of glycated haemoglobin [19].

The relationship between internal quality control and external quality assessment in laboratory medicine

Internal quality controls (IQC) are conducted at least daily for each analyte. A higher frequency is possible for measurements whose stability might otherwise pose a risk for the patients. Therefore, consumption and cost of IQC materials is considerable. Many laboratories use less expensive IQC materials supplied by the assay manufacturer or non-commutable control materials with assay specific target values [20]. In the chain of traceability, these types of IQCs are linked to the primary reference material or reference method through the same higher order measurements as the assay itself. An error at the assay manufacturer propagates to the IQC target values as well and cannot be detected. IQC is primarily used to monitor analytical stability in the laboratory only. EQAs with values assigned with a RMP can provide an independent link to the SI units and the primary reference material or reference method and guarantee traceable measurements over a longer time-period.

IQC results should be interpreted together with EQA performance of programs using commutable materials to detect lot-to-lot variation without any sophisticated statistical and mathematical evaluation [16], 21]. While commutability of EQA materials is required in category 1, 2, 3 and 4 EQAs, it is not a prerequisite to evaluate the stability of an examination procedure. However, IQC materials non-commutability should not result in a biased estimate of imprecision and therefore of MU [22]. The providers of reference and control materials for the next generation of in-vitro diagnostics should assess the commutability of those materials before their use. Origin of materials may influence the responsibility of assessing commutability. Certified reference material (CRM) suppliers are expected to assess the commutability of their products if they intended to serve as viable calibrators in a calibration hierarchy [15]. There are different responsibilities for control materials: The commutability of EQA materials should be assessed by the EQA provider, while the commutability of IQC materials (if the material is used for uncertainty estimation) is, at least nowadays, the responsibility of the end user [23], [24], [25].

Some data-driven quality control programs shift the boundaries between internal and external quality control. An example presented in part I of this manuscript series uses patient medians to monitor analytical quality [1].

Use of EQA results for the calculation of measurement uncertainty

Specification of MU of a laboratory examination method allows clinical laboratories and clinicians to evaluate the quality of test results based on an identifiable performance characteristic. High levels of trueness of laboratory results are required when sharp guideline-driven clinical decision limits are applied, as e.g. HbA1c in the diagnosis of diabetes and cholesterol in classification of low or high risk for coronary heart disease. It was shown that an analytical bias of 2 % resulted in a doubling of false positives in HbA1c and cholesterol screening [26]. To counteract this, recommendations have been made regarding the maximum tolerable measurement uncertainties [27].

Different approaches have been proposed to determine the MU mainly using imprecision (note: different coverage factors are in use), but some formulas also take into account the bias as a component of the MU. An essential requirement to properly evaluate bias is to use commutable EQA materials value assigned with a RMP. As both conditions are difficult to meet, bias estimation can be flawed. Whether bias is included or not in calculation of MU, bias needs to be taken into account in the overall communication of uncertainty of result to a clinician. This is where EQA plays a significant role in equipping the laboratory with such information. EQA schemes involve testing of identical samples with many laboratories using different established test systems. It is important to note that the observed variance of interlaboratory comparison results is relevant only when commutable EQA materials are used. However, this only allows conclusions to be drawn about the MU of a laboratory or an analysis method under certain conditions and with restrictions [28]. Although ISO 15189:2022 requires laboratories to consider the MU when verifying or validating a method and that the MU is regularly reviewed, it does not explicitly state how the MU should be calculated. However, guidance on this can be found in ISO/TS 20914:2019 [29].

Use of EQA results to monitor laboratory performance over time

EQA organisers accumulate large amounts of data, which have been used to demonstrate the improvement of laboratory analytical performance over time for specific parameters [30], [31], [32], [33]. Several reasons could contribute to improvement: (i) the awareness of the importance of providing high quality results, which is promoted by the accreditation of the laboratory according to ISO 15189:2022 [34], (ii) the approach of using state-of-the-art examination methods, (iii) the harmonisation of examination procedures [30], (iv) possible recognition that other laboratories have achieved better results with the same method, or (v) following comments or advice of the experts at the EQA provider. EQA data also show a correlation between the laboratory’s good performance and regular participation in EQA programs [33], [35], [36], [37]. To assist participants in their long-term retrospective performance evaluation, some EQA providers plot the history of relative deviations of their results from the assigned value of past cycles so that any patterns (e.g. systematic or fluctuant biases) can be recognised and evaluated (Figure 2).

Figure 2:

Development of z-scores of selected individual measurement results as reported by participants over several consecutive EQA cycles. EQA programs provide information on the deviation of a laboratory’s results from the respective target value over time. The representation of the deviations as a z-score enables their evaluation independently of the respective level of the measurand in the EQA sample.

Use of EQA samples and results to develop and monitor staff competence

Participation in EQA, from sample preparation, examination and results submission to reviewing the report, can be used for continuous professional development (CPD) for laboratory professionals [1]. To monitor staff competence, EQA samples can be analysed by several laboratory employees. Each employee’s results are collected by their supervisor and one person’s results are selected to be reported to the EQA provider. Results obtained by other employees are compared with the targets once they are published at the end of the EQA cycle. Discussing the results and analysing any deviations and their causes is valuable for staff training and professional development. EQA programs are currently being developed that allow or even encourage group registration by an employer.

Bacteriology and mycology

Reliable and accurate characterization and interpretation of microbiological specimens are impacted by many variables in a laboratory setting and the field of microbiology continues to evolve rapidly each year. EQA providers can start by supporting microbiology laboratory quality by including a patient case history with each EQA sample which should guide the laboratory’s processing and interpretation of results. Evaluation of results then depends on the specific case and microorganism combination. This usually requires an interpretation of participant results and consensus of opinion on grading by expert committee review (composed by experienced and actively practising microbiologists) (Table 4). Samples and challenges are designed to ensure laboratories report normal flora or contamination as such, and that the final results report is clinically relevant. New organisms are frequently recognized and associated with new disease states, and new methods are developed for their detection or screening (e.g. Candida auris).

Challenges should also be designed to check laboratories’ adherence to current guidelines and the appropriateness of participant antimicrobial susceptibility results reporting. New interpretation and reporting guidelines are published frequently by professional and standards organisations such as CLSI, EUCAST, and ISO and so it can be challenging for working laboratories to keep up to date with the flow of new information. For example, the Guidelines for the Detection and Identification of Group B Streptococcus were first published in 1996, updated in 2002, 2010, 2020 and 2021 directly impacting the role of the laboratory in the reporting of results [44]. By participating in EQA challenges that are designed to test the interpretation of results beyond the analytical aspect, laboratories make sure their interpretation of results are aligned with current guidelines, local regulations, and latest taxonomy changes.

Although historically microbiology lagged behind other laboratory disciplines in the implementation of new technologies, this has changed dramatically with the implementation and saturation of MALDI-TOF instruments in medical laboratories and also the emergence of molecular technologies for genetic identification. Although these technologies are now used routinely in the laboratory, there are still pitfalls, and a thorough validation needs to be performed with each assay. Importantly in the field of microbiology, participation in EQA may be the only chance laboratories have to become familiar with a rare organism and to test their ability to isolate and identify organisms that are rarely isolated and may never reach a laboratory if not for a EQA sample. Microbiology laboratories and their personnel should not blindly trust results without EQA participation.

Virology

The focus in microbiological-bacteriological diagnostics is on pathogen and resistance determination, and serological diagnostics are of lesser importance. In virus diagnostics, however, pathogen detection is of equal importance as determination of specific immune responses. Accordingly, EQA programs for virological purposes are designed so that participants receive samples in which they can detect either one (or more) viral pathogens or specific antibodies against a particular virus. In addition to assessment of reported results, EQA for virus detection can also provide participants and manufacturers with information on sensitivity, specificity and linearity of their test systems [45]. Unpublished data indicate clinically relevant discrepancies between the results obtained by infection diagnostic IgG and IgM assays from different manufacturers. Even if EQA providers cannot offer a solution to this, they can still repeatedly draw the attention of participants (and manufacturers) to this fact in summary reports written in connection with completed EQA cycles and thus sensitise them to any strange constellations of results that inevitably occur in routine patient diagnostics. Several other features offered by virology EQA are described in Part V of this manuscript series [4].

Parasitology

Although DNA amplification techniques are increasingly used, manual microscopic examination is often used as an all-round method for the direct detection of parasites in clinical specimens because parasites are relatively large microorganisms and often difficult to culture. In addition, some serological assays are used to indirectly detect invasive parasitic infections. As discussed above, EQA is beneficial for all medical laboratories and for those that perform examinations to detect parasitic infections for the following two specific reasons. (i) Although parasitic infections are worldwide very abundant, the large number of species and the differences in their endemic areas, pose a challenge for diagnostic laboratories as most parasite species will only be detected on rare occasions, and therefore, exotic EQA specimens are not only crucial for assessment of the competence of the laboratory staff but also to detect the knowledges gap that can subsequently be addressed by education [46]. The recent developments in virtual microscopy and its use in EQA provides new opportunities for detailed and personal feedback, which is of high educational value. (ii) Because the parasite load in clinical specimens is low compared to other pathogenic microorganisms, both the pre-analytical concentration methods and the subsequent DNA extraction methods are critical for optimal parasite detection by DNA amplification techniques. EQA schemes for the detection gastro-intestinal protozoa and helminths in stool as well as for Acanthamoeba spp. in corneal scrapings, have demonstrated an enormous difference in detection efficiency between laboratories and not only notifies medical laboratories on their poor efficiency performances but can also provide information on which part of their examination process should be improved (DNA isolation or DNA amplification) [47], 48]. (iii) Due to the immune regulatory properties of parasites and their complex antigenicity, most serological assays to detect invasive parasites cannot use a single or small set of recombinantly expressed proteins as antigen(s) as this will result in a poor sensitivity. Instead, serological assays to detect invasive parasites are often based on antigens prepared from purified fractions of parasites. Therefore, serological assays to detect parasitic infections not only differ in technological set up (e.g. agglutination, ELISA, western-blot, etc.), but also on the type of antigens used and the procedure by which these fractions are produced. Since EQA schemes are in fact a longitudinal inter-laboratory comparison study, EQA is the most powerful tool to evaluate the performance of distinct serological assays.

National metrology institutes

The “metre convention”, first signed in 1875, is an international treaty which created an international organisation called the Bureau International des Poids et Mesures (BIPM). Since its inception the BIPM has been tasked with facilitating the standardisation of measurements worldwide by coordinating the activities of member states on activities related to measurement science. On its inception, the treaty was focused on the unification and improvement of the metric system. This mainly focused on physical measurements vital for the industrial revolution and intercontinental communications and trade. Therefore, historically many of the national institutes that engaged with the BIPM were devoted to physics (e.g. national physics laboratories such as NPL (National Physical Laboratory) in the UK or PTB (Physikalisch-Technische Bundesanstalt) in Germany). However, as the measurement needs for trade and other areas of where the international comparability of measurement results using the international system of units (SI) expanded such as environmental monitoring and measurements in the heath sector, the activities of the BIPM and its members expanded to include these activities. In 1995 the first meeting of the Consultative Committee for the amount of substance (CCQM) took place. The major focus of this committee was to improve the comparability of chemical measurement across the different application areas. In 1999 the mutual recognition agreement (MRA) of national measurement standards and calibration and measurement certificates issued by national metrology institutes was drawn up. The objectives of this agreement are to: (i) establish the degrees of equivalence of national measurement standards maintained by NMIs; (ii) provide for the mutual recognition of calibration and measurement certificates of NMIs; (iii) provide governments and other parties with a sound technical basis to conduct wider agreement related to trade and regulatory affairs. This agreement stipulates the nomination of one NMI per member state. In many countries the national physical laboratory was nominated to fulfil this role. However, for many countries this laboratory did not possess the range of measurement activities covered by the MRA. Therefore, many countries have designated institutes (DI), such as LGC in the UK or BAM in Germany, where measurements for specific sectors or activities are delegated by the NMI to these institutes. The role of the NMI is to coordinate the actives within their country to assure their national and international needs are met. Under the MRA the consultative committees or sometimes regional metrology organisations organise office comparisons of the services and/or measurement services offered by the NMIs/DI. These comparisons are referred to as key or supplementary comparisons. They have very strict rules and all methods and their associated uncertainty estimates are rigorously peer reviewed. These comparisons lead to calibration measurement capability claims which are listed on an online BIPM database [72]. In the clinical sector these normally take place via two different mechanisms. Where NMIs have CRMs for the same measurand, all participants can send their CRM to a single laboratory where a comparison of the materials and their assigned values is undertaken. However, more commonly materials are sent from one coordinating laboratory to participating NMIs who provide estimates and their associated uncertainties for the measurands in the materials received. The NMIs normally use a primary method or primary ratio method of analysis, which for pure materials normally incorporates quantitative NMR supported by mass balance approaches, while for matrix materials this normally involves isotope dilution mass spectrometry-based methods. While the measurement and calibration approaches remain the same, no two materials are samples are identical, therefore the NMIs will often have to alter the extraction, chromatographic or mass spectrometry conditions for the sample received. Effectively this requires the method to be revalidated on the sample as received. This approach enables the NMIs to assess and quantify the individual uncertainty contributors for the material received. This is one of the reasons why the procedures and approaches are not suitable for high throughput analysis and why these procedures may not readily identify commutability issues if the materials are to be used as CRMs.

On the one hand, while the key comparison could be considered a sort of EQA, NMIs benefit from participating in broader based EQA from a number of factors. (i) If an NMI participates in key studies and maintains Calibration and Measurement Capabilities (CMCs) by participating in the same EQA as its national reference or calibration laboratories it can assess the agreement of reference values between reference laboratories and assure their link to SI; (ii) cooperation with expert laboratories allows the exchange of knowledge and experience of the NMIs regarding the establishment of higher metrological order examination procedures and the expertise of the expert laboratories on specific properties of certain measurands and matrices [73], 74]; and (iii) as unknown bias can be introduced by change of methods and technology, participation in EQA enables an unbiased assessment of the current state of the art. On the other hand, NMIs can use EQA challenges to (iv) test suitability of newly developed CRMs; this provides extra confidence in the NMI assigned value and may quickly identify commutability issues with the material (this should never substitute a complete and thorough assessment of commutability). Finally, (v) NMIs can assist in obtaining comparable measurement results for a particular sector; this can be done by engaging with a whole community (e.g. infection diagnostics) where the specific measurands need both harmonisation and standardisation activities. In these and other areas it is essential that NMIs engage with stakeholders to help provide solutions. This is best done in a collaborative manner and with early intervention to prevent the development of SI traceable measurement with small uncertainties for a different measurand than is intended by the community.

Calibration laboratories

Laboratories may be listed as so-called calibration laboratories in the Joint Committee on Traceability in Laboratory Medicine [75] (JCTLM) database as reference measurement services providers qualified to perform JCTLM-listed RMPs [76]. Inclusion of reference measurement procedures and calibration services in the JCTLM database is directly linked to participation in interlaboratory comparisons like RELA-IFCC, the IFCC EQA program for reference laboratories in laboratory medicine [77]. The applicable international standard ISO 15195 also requires accredited calibration laboratories to demonstrate their competence to carry out calibration work correctly through interlaboratory comparisons, and ISO 17025 even requires them to ensure the validity of results [78], 79]. The RELA interlaboratory comparison process uses target value assignment by RMPs, and transparency of the program and information to third parties are particularly important. Results to be reported by participants for two different samples include measured values and associated MU estimate according to the Guide to the Expression of Uncertainty in Metrology (GUM) [80]. A draft of the final report, in which the participants are anonymised, is distributed to the participants at the end of the submission period. Participants may request that their results be removed from the report before the results and the disclosed identities of the other laboratories are made freely available on the IFCC-RELA website [77]. In addition to the results, limits of equivalence (LoE) as set by the RELA advisory board and the IFCC Committee for Traceability in Laboratory Medicine (C-TLM), are disclosed in the report. LoE are not considered as a “grading” system and have no regulatory impact, but they may be used for educational purposes to compare and monitor the performance of the procedures at the highest metrological order. Since the introduction of RELA in 2003, the number of participants has increased continuously, which underlines the utility for reference laboratories to compare their respective results.

Interlaboratory comparisons are also an important tool for validation during the development of new calibration procedures. Target values assigned by calibration laboratories are used for the evaluation of EQA schemes for routine laboratories or for calibrators to establish the metrological traceability of measurement results in the framework of IVDR. This places calibration results at a high level in the metrological traceability chain. Therefore, the direct link from the calibration results to the patient results helps to continuously increase measuring accuracy and therefore also patient safety. Also, other networks, such as the IFCC network for HbA1c, or intercomparison measurements organised by NMIs, support this concept of quality assurance for calibration laboratories as part of the metrological traceability of measurement results.

Reference laboratories

Several competent organizations have designated reference laboratories for their interests, like the European Union (EU) and national governments [71]. Tasks, required competences and authorizations assigned to these reference laboratories are defined in regulations or contracts and may include the requirements for participation in EQA. Even if the client does not explicitly request this, the objective proof of passed EQA challenges can serve the contractor as objective proof of his analytical competence and good laboratory performance. It is also quite possible that the standard according to which the reference laboratory is accredited requires participation in interlaboratory comparisons anyway. Reference laboratories may have a different analytic portfolio than routine medical laboratories, but they also may employ different examination procedures, e.g. for confirmatory tests. For these two reasons, as described below, they participate in customized EQA programs that either include the required measurands and/or allow comparison with other laboratories using the same test systems.