Harmonization of External Quality Assessment Schemes and their role – clinical chemistry and beyond

Introduction

External Quality Assessment Schemes (EQAS) play a central role in laboratory medicine [1]. In Europe, successful participation in EQA programs is set as a mandatory requisite by country-specific accreditation bodies such as the Italian Accreditation Body (Accredia) and the Board of Accreditation (RvA) in the Netherlands, to have access to the ISO 15189 accreditation according to the flexible scope approach [2]. ISO 15189:2012 dedicates a full paragraph 5.6.3 to interlaboratory comparisons [3]. EQAS are (or should be) the final step to confirm the success of the traceability chain [4] and are also seen as part of the surveillance of the performances of the In Vitro Diagnostic medical tests [5]. Thus, although there is no doubt about their importance, the questions are as follows: Are EQAS really fit for purpose? Are all EQAS equivalent and sufficiently harmonized? Is their role similar and necessary in all branches of laboratory medicine?

Present situation

Miller already in 2009 clearly indicated advantages and limitations of participation in an EQAS [6] and 2 years later, together with some coworkers designed a table classifying the EQAS in six categories according to their evaluation capabilities [7]. Category 1 schemes, based on commutable materials with target values assigned by reference methods, are the schemes that have full evaluation capabilities, and the authors conclude that, ideally, all EQAS should be category 1 schemes but are rare because of a number of constraints, including costs, technical difficulties and lack of awareness of the importance of these characteristics. Infusino et al. [8] proposed a further subdivision of EQAS categories 1 and 2 in A and B: A, when using the higher order models for performance specifications identified by the EFLM Milan Strategic Conference [9], B if using lower levels.

Unfortunately, with the exception of a few examples [10], [11], [12], [13], [14], category 1 schemes remain scarce and so the real benefit of participation in EQAS remains modest. Other areas of laboratory medicine, where the experience of EQAS is more recent and/or deals with different type of measures, seem to benefit more from the participation in EQAS as reported for immunophenotyping [15] or microbiology [16]. In the Netherlands, the Calibration 2.000 initiative is a nationwide program, established in 1998, which strived for either standardization or harmonization of tests in domains beyond clinical chemistry such as hematology, endocrinology, coagulation, immunology, TDM, parasitology, etc. [11]. The output of its research was fed into the EQAS of the Dutch SKML and is reviewed in this special issue [11]. Notwithstanding the achievements of Calibration 2.000 in the past 20 years, especially in the clinical chemistry domain, EQAS for the other domains are not yet at the category 1 level.

Open questions

Are EQAS really fit for purpose? The concept of traceability and trueness-based grading is typical for clinical chemistry and all other laboratory disciplines producing numerical results on a continuous scale. In the case that a reference measurement system is in place, category 1 EQAS should be implemented for an effective improvement of the analytical quality. On the contrary, participation in Miller’s category 5 schemes only allows to make peer group comparisons [6]. As briefly indicated above, for real help in improving clinical laboratories’ accuracy, verifying the calibration-related and/or specificity bias as well as imprecision of specific tests or analytical systems, the vast majority of the presently available EQAS are not fit for purpose. Exception to the rule is the EQAS and MUSE scoring system from the Dutch SKML [17], which allows to systematically distinguish bias and imprecision of tests. In other EQAS, the core of the problem is the control materials in the schemes that are not commutable or whose commutability is not verified or demonstrated. The verification of commutability is extremely demanding both in terms of costs and time for realization [18], [19], [20] and thus not affordable for small schemes. Moreover, it is quite clear that manipulation of control materials to stabilize them (addition of exogenous substances, lyophilization, etc.) and to obtain predefined measurand concentrations often brings along non-commutability [21], [22]. The production of suitable control materials requires research and investments and their distribution in frozen form is also expensive. Another important cost is the value assignment using reference methods. However, assigning target values with reference methods to non-commutable control materials is not only useless but also dangerous because it may cause biases that do not exist but are only caused by non-commutability, thus inducing undue corrective actions. All these reasons explain why the development of category 1 or 2 EQAS is a significant challenge that therefore progresses slowly, notwithstanding all the literature evidence regarding the absolute necessity of commutability of EQA materials in accuracy-based EQAS. Another relevant drawback that impairs the efficacy of some schemes is the use of inappropriate analytical quality specifications (APS) to evaluate the performances of the laboratories. Only when applying adequate APS, according to the model proposed by the first EFLM strategic conference [9], there is an effective relationship between the test performances and the level of quality needed. EQAS should help laboratorians to make informed decisions about test accuracy by being enabled to verify the adequate implementation of the metrology concept through IVD manufacturers. Unfortunately, most EQAS are not equipped to do this and do not allow a proper verification of the correct implementation of the traceability concept, within allowable measurement uncertainties.

The situation is different when looking at EQAS devoted to assessing performance in pre- or postanalytical phases by circulating questionnaires or clinical cases or when distributing images or slides to assess the competence of the professionals [23], [24], [25], [26]. In these cases, usually the improvement obtained is defined, even if it is difficult to quantify it and to demonstrate an objective efficacy.

Are all schemes equivalent and sufficiently harmonized? Also this question has a negative reply. Apart from the different evaluation capabilities depending on the type of EQA materials used, there are several other reasons that create a very large heterogeneity in the EQA programs. Jones and coworkers [27] well identify these reasons that are summarized in six groups: (a) the nature of the EQAS material, including its commutability, which may affect the result interpretation; (b) the procedure used to assign the target value; (c) the data set to which performance specifications are applied (i.e. single result, series of data); (d) the analytical property being assessed (i.e. total error, bias, imprecision); (e) the rationale for the selection of the performance specification; and (f) type(s) of model used to set performance specifications. The problem of the evaluation criteria is very relevant, and their setting depends on the main scope of the program: if it has a regulatory impact, the criteria are looser because the scope is to identify the very poor performer; if it has an educational scope, the criteria are tighter because failing to meet them does not automatically imply sanctions but only a remedial action by the laboratory. This problem exists also outside clinical chemistry as indicated by Olson and coworkers for coagulation [28]. The situation may improve with the implementation on a larger scale of accreditation for EQA providers based on ISO 17043:2010 [29]. Unfortunately, this ISO standard defines no requirements regarding commutability or performance specifications. In this situation of great heterogeneity of EQAS, just the participation in any type of EQAS is not sufficient in terms of verifying precision and metrological traceability of test results as well as compliance with ISO 15189:2012. Therefore, inspectors need to verify also this aspect [30].

Is the role of EQAS similar and necessary in all branches of laboratory medicine? The reply to this last question is absolutely positive; no diagnostic discipline of the clinical laboratory can operate its test performance without EQAS from independent third parties. EQAS are essential tools for evaluating test result equivalence by comparing test results to value-assigned trueness verifiers in case of commutable EQAS (i.e. category 1 EQAS) or to peer group means in case of category 5 EQAS, and for detecting problematic, non-traceable tests, which can be traced back to specific IVD manufacturers. EQAS are very important tools for standardization and harmonization of laboratory practices. Paradoxically in some laboratory areas, apparently more complex than clinical chemistry, the problems of materials and accuracy evaluation seem less critical, e.g. for microbiology where microorganism identification is the gold standard and both traditional and molecular biology-based techniques can be successfully tested, with very good results [31]. Yet Schuurs et al. [31] demonstrate that harmonization of PCR-based detection of intestinal pathogens is still in its infancy.

We conclude that harmonization of EQAS has still a long way to go, and much technical and organizational work has to be done, but important milestones indicating the way to follow have been defined [1], [7], [9], [11], [18]. Intensive collaborations or alliances between country-specific EQA organizations under the umbrella of the European Organization for External Quality Assurance in Laboratory Medicine are urgently needed, as well as efforts to merge EQAS in countries where different schemes for the same measurands are in use. These efforts should allow to develop jointly affordable and sustainable category 1 EQA schemes by sharing expertise and enlarging market share in medical laboratories.