Spanish society of laboratory medicine external quality assurance programmes: evolution of the analytical performance of clinical laboratories over 30 years and comparison with other programmes

Introduction

External quality assurance (EQA) programmes are a key component of laboratory quality management. Their involvement enables the laboratory to evaluate and monitor its analytical performance, compare the reliability of measurement procedures and understand the degree of harmonisation of analytical results.

The Spanish Society of Laboratory Medicine (SEQC^ML) launched its first EQA programme in 1981, including 20 biochemical serum magnitudes and with 147 participants [1]. Since then, we have incorporated our own programmes and established collaborations with national and European programmes (Spanish Society of Haematology and Haemotherapy for haematological magnitudes, UK NEQAS for immunology and allergy, and OELM for trace elements). The SEQC^ML currently distributes 29 programmes, for all laboratory areas, covering 189 biological magnitudes and involving around 700 laboratories.

The information to be provided by an EQA programme according to the Task Group on Performance Specifications for EQAS of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM-TG-EQA) created at the 1st Strategic Conference in Milan in 2014 [2], in order to allow participants to correctly interpret their results [3] involves: the control material distributed, the target value to which the laboratory result is compared, how the analytical performance of the participant is expressed and the evaluation criterion used.

The control material for most SEQC^ML EQA programmes is stabilised serum except for the commutable serum programme (SCR) with reference method values assignment (SCR) which uses frozen human serum.

The target value of the control samples is the average value obtained by each group of users of the same analytical method and instrument (homogeneous group or peer group), and that of the SCR programme is the certified reference value.

The analytical performance of the laboratory is expressed by the percentage deviation (PD%) of each result from target value, indicative of total analytical error (ET). The analytical performance of the participating methods is expressed by the coefficient of variation (CV) between laboratories, which indicates the imprecision of each homogeneous group, and by the PD% of the average for each homogeneous group, indicating the bias of that method.

The assessment criterion has changed over the years: until 1993 the statistical criterion of the standard score with respect to the homogeneous group was used [4]. From 1994 onwards, the analytical quality specification (AQS) derived from the biological variability (BV) was also applied, where data were available [5] and otherwise, after 2011, the 90th percentile of the results of the individual deviations (PD%) obtained by the participants, after exclusion of extreme values, was applied.

In the case of using BV, three “levels” of demand are used, called minimum, desirable and optimum [6], according to the percentage of individual deviations of the laboratories complying with the specification:

1. Minimum: less than 80% of PD%s would comply

2. Desirable: between 80 and 90%

3. Optimal: 90% or more PD% would be within the limit defined by BV.

Following the categorisation of EQA programmes described by Miller et al. [7] most of the programmes in the SEQC^ML are category 4 (control material for stabilised human serum, value assigned by homogeneous group, repeated measurements, EPA based on BV); the SCR programme is category 1 (control material for frozen human serum, value assigned by reference method, repeated measurements, EPA based on BV).

The objectives of this study are to understand the evolution of the analytical performance of the laboratories participating in the SEQC^ML EQA programmes over more than 30 years of operation (1981 – 2018), and to compare the analytical performance obtained by laboratories participating in the SEQC^ML EQA programmes with that of participants in other EQA programmes.

Materials and methods

The material used in this study is the reports for the individual and end-of-cycle laboratory, which the SEQC^ML EQA programmes provide to the participants.

Results

Change in the analytical performance of laboratories participating in the SEQC^ML EQA programmes

The individual report provides the deviation from the peer group. Figure 1 shows the percentage of individual laboratory results (taken from individual reports) in 2018, whose deviation (PD%) from the homogeneous group meets the quality specification for analytical ET derived from the BV (minimum, desirable or optimal), on the serum biochemistry programme.

Figure 1:

Compliance with the BV-derived total analytical error specification of the 2018 Serum Biochemistry Programme.

For most biological quantities, 90% of the participants' results meet the specification and even for quantities with higher homoeostatic regulation (lower BV and more restrictive specifications), 50% or more of laboratories are already able to meet the requirement.

A comparison of the 90th percentile PD% for the laboratories participating in the hormone programme between 2012 and 2018, a very important decrease can be observed (Supplementary Figure 3). 90% of laboratory insulin results in 2012 had a PD = 39%, which decreased to a PD% = 14% in 2018. Likewise, the P90 of the C-peptide results decreased from 23 to 14% and the thyrotropin results from 13 to 9%.

The end-of-cycle report shows the indicators of inter-laboratory imprecision (measured as CV) and bias towards the homogeneous group (measured as PD%). Figure 2 shows five time slices of magnitudes chosen as representative of substrates (glucose), electrolytes (potassium), Alanine aminotransferase (ALT) and thyroid-stimulating hormone (TSH):

1. Glucose (Figure 2A): the predominant use of enzymatic methods since the 1990s reduced inter-laboratory imprecision to 6% and bias to 10%. With the use of more specific enzymatic methods, a CV of 4% and a PD% of 7% were obtained in 2018.

2. Potassium (Figure 2B): in 1984 there were more than 10 possible methods for analysing potassium, with a CV of 5.5% and a bias of 7%; from 1989 to 2000 there were flame photometry and potentiometry together and subsequently only potentiometry. Analytical performance indicators gradually improved to a CV of 2% and a bias of 2% by 2018.

3. ALT (Figure 2C): as with other enzymes, the disappearance of colourimetric methods led to a reduction in inter-laboratory imprecision, from CVs close to 50% to CVs below 5%. Bias also decreased dramatically.

4. TSH (Figure 2D): the hormone programme was started in 1999, using different analytical methods with a CV of 12% and a bias of 30%. As of 2010, with the predominant use of luminescent methods, the CV had decreased to 5% and the bias to 15%.

Figure 2:

Evolution of inter-laboratory imprecision and bias towards the homogeneous group.

Discussion

The control material for an EQA should be similar to patient samples and the target value should be determined by certified reference methods [3]. Only one of the 29 SEQC programmes currently meets these conditions (SCR), the main difficulties being a lack of commutable control materials and reference methods for all biological quantities in the clinical laboratory.

If the EQA programme uses commutable material with a value assigned by reference methods, it can evaluate the true accuracy of the individual laboratory, assess analytical methods and estimate bias and promote standardisation. Otherwise, it can only assess the individual laboratory and compare homogeneous groups.

Indicators of analytical performance: The analytical ET, calculated imprecision and bias, and type of EPA used in the SEQC^ML programmes to evaluate participants follow the recommendation of the 1st European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Strategic Conference (Milan 2014) [17].

The participating laboratories have improved their performance over the years, as is shown by increased compliance with the EPA (Figure 1) and a decrease in deviations from target value. The use of the BV-derived criterion to evaluate the results probably contributed to this improvement.

Despite criticism of the use of BV as a performance specification because it is too strict [17], this study shows that laboratories can achieve them for many biological magnitudes.

The decrease in dispersion of methods, the cessation of obsolete methods (enzymes) and the use of chemiluminescence in immunoassays (Figure 2) are contributing factors in improved performance.

It is difficult to compare SEQC^ML EQA programmes with other programmes from different organisations because the reports are only accessible to participants. This study therefore only presents data from programmes that publish their results in international journals or on open access websites. Other difficulties lie in the differences in the control materials distributed, the criteria for forming homogeneous groups, the calculations performed and in the quality specifications applied to assess the results.

The lower PD% of the SEQC^ML compared to ProBioQual (Table 2) are probably due to the fact that the Spanish programme uses BV as a specification, while the French programme uses the more permissive current method performance.

However, the discrepancies observed in the three programmes that use commutable controls with values assigned by certified reference methods (Table 3) cannot be explained in light of the data available in this study.

When the EPAs are based on state of the art, it has been found that most programmes do not define the ‘state of the art’ as the best possible performance of the methods used by the participants, as recommended at the 1st EFLM Strategic Conference in Milan [18], but as the current performance of the methods. This is a further difficulty in achieving harmonisation.

The data shown in this study reveal that, despite the fact that performance is continuously improving within each programme and efforts are being made for harmonisation, the results obtained under different programmes for some biological magnitudes are still not harmonised.

Harmonisation could be achieved through the efforts of all parties involved in the activity of the medical laboratory: international organisations that promote standards and guidelines, scientific societies that disseminate their data, the in vitro diagnostic industry that puts standardised methods and calibrated analytical systems on the market with documented traceability standards, and healthcare laboratories that use the best methods with rigour and protocolisation.

Since 1998 the international community has been making efforts to achieve harmonisation, as reflected in the following documentation:

1. European Directive for the In Vitro Diagnostic Industry (1998), requiring suppliers to ensure traceability at the highest reference level, wherever possible.

2. ISO 17511 (2003): In vitro diagnostic medical devices-Measurement of quantities biological samples – Metrological traceability of values assigned to calibration and control materials. This standard was published following the same requirements as the European directive [19]

3. JCTLM (2002): established by several international bodies that keep a database of reference materials, analytical procedures and reference laboratories as per ISO 17511 [20]

4. ICHCLR (2013): this consortium has a global portal where updated information on the status of standardisation/harmonisation of biological quantities is collected and protocols are developed to establish harmonisation of those biological quantities for which no reference materials and methods exist [21]

5. The new ISO/NP 21151 standard (under development), defining protocols to be followed to ensure harmonisation of quantities without reference materials or methods [19]

EQA programmes play an important role in monitoring the harmonisation of medical laboratory performance, but the use of commutable controls with certified reference values should be encouraged wherever possible and the recommended analytical performance specifications applied.

In conclusions, the laboratories participating in the SEQC^ML EQA programmes have significantly improved their performance over the 30 years. The specifications derived from biological variation are achievable by many laboratories in our country. For biological quantities with no known BV data, the current state of the art is used, rather than the best possible. Organisers of EQA programmes should also join the harmonisation effort by making information on their results available to other interested professionals and by facilitating comparisons between organisations.