Samples

Each laboratory received a set of six commutable frozen human serum samples, regularly used in the national EQA program and manufactured by the Stichting Kwaliteitsbewaking Medische Laboratoria (SKML; Dutch EQA organization). Samples were shipped on dry ice and stored by the laboratories below −70 °C (or, when assayed within 2 weeks, at −20 °C). Samples were thawed at room temperature and analyzed within 8 h after thawing. All target values were assigned with reference measurement procedures (RMP). Preparation, commutability and stability of the samples have been demonstrated and described in previous publications [11], [13]. Table 1 shows the analytes included and their concentration range.

Participants

Table 2 shows the number of participating laboratories specified per country and per manufacturer. The laboratories of Italy, Spain, Portugal, and UK were randomly selected by the respective EQA organizers but with the criterion to achieve a number of 20–30 per country and to cover the common analytical platforms; samples were analyzed in addition to the respective regular EQA programs. In the Netherlands, the samples were part of the regular EQA program and thus analyzed by all laboratories. The analytical platforms of the manufacturers with comparable technology were considered as a group, leading to one group for Roche (all Cobas instruments) and Abbott (all Architect models), and to two groups for Beckman-Coulter (UniCel-DxC and Olympus AU instruments) and Siemens (Dimension and Advia instruments).

Calculations at analyte level

For each sample and each group (group=all laboratories, all laboratories of a country, or all laboratories using the platforms of the same manufacturer), the bias of the mean of that group (bias=mean of the group – target established with the RMP), and the between-laboratory coefficient of variation (CV) were calculated. From these, the mean bias (B) and mean between-laboratory CV (CV_m) of the six samples were calculated and used to calculate the total error (TE) of the group according to TE=|B|+2 CV_m [14]. TEs were compared with the total allowable error (TAE) calculated on basis of the BV taken from the BV specifications database by Ricós et al. [15] and classified as meeting the optimum, desirable, or minimum specification (or as not meeting any of the three criteria). BVs and TAEs are summarized in Table 1. In addition to the results of the individual countries, these calculations were also made for the Italian, Spanish, Portuguese, and UK laboratories as one group. The overall mean is the mean of the Italian, Spanish, Portuguese, and UK laboratories. Results of the Dutch laboratories were excluded from the calculation of the overall mean (to prevent unbalanced data due to the large number of laboratories in comparison to the other countries). For the figures, the model of the IFCC Task Force on HbA_1c was used [16].

Calculations at aggregated level

To summarize the performance of all 17 analytes in one number, the median TE/TAE ratio was calculated: TE/TAE ratio={Σ(TE/TAE_minimum)}/17 in which TAE_minimum is the TAE for the respective 17 analytes at the minimum performance level.

Analyte level

TE’s for country and manufacturer groups are summarized in Table 3. The underlying bias (B) and between-laboratory CV (CV_m) are listed in Supplemental Tables 1 (for countries) and 2 (for manufacturers). Colors indicate whether the TEs met one of the TAE specifications for optimum (yellow), desirable (gray), minimum performance (amber), or not (white). Yellow, gray and amber can also be read as gold, silver, and bronze performance. It can be seen that for sodium (Na), none of the countries and manufacturers even met the minimum specification (thus all white). For glucose, all countries and manufacturers met one of the specifications: gold (one manufacturer), silver (two countries; three manufacturers), and bronze (four countries; two manufacturers). Table 3 shows all numerical TEs; however, when none of the specifications are met, the degree and source (bias and imprecision component) of TE cannot be readily identified. These issues are addressed in Figure 1A and B, with results for total protein enlarged as an example. The TAE for minimum performance of total protein is 5.6% (see Table 1). According to TE=B+2CV_m, a TE of 5.6% can be achieved either with a B of 5.6% and a CV_m of 0.0% or (the other extreme) with a B of 0.0% and a CV_m of 2.8%. These extremes are plotted in the figure: a B of 5.6% on the y-axis and a CV_m of 2.8% on the x-axis. Similarly, the extremes for desirable and optimum performance were calculated (B’s of 3.7 and 1.8%; CV_m’s of 1.8 and 0.9%). The extremes on both axes are the limits of triangles for a golden optimum, a silver desirable, and a bronze minimum performance. The TEs of the countries (left side) and manufacturers (right side) are visualized by plotting the respective B’s and CV_m’s (taken from Supplemental Tables 1 and 2). For total protein, it can be seen that of the countries, only UK (U; red) met the minimum specification. The manufacturer graph shows that Siemens Dimension (D; brown), Siemens Vista (S; purple) and Beckman-Coulter Olympus AU (O; green) were within the minimum specifications. Abbott (A; red) and Roche (R; yellow) were moderately outside, and Beckman-Coulter UniCel DxC (B; blue) was far outside the specifications. The performance of all the other analytes are visualized with respect to country and manufacturer. The enzymes in Table 3 and Figure 1 are from laboratories using methods claimed to be traceable to the respective IFCC RMPs only. The percentages of (excluded) laboratories not reporting in units traceable to the IFCC RMPs are provided in Table 4.

Figure 1:

Performance per country and per manufacturer.

For each of the analytes, bias (y-axis) and between laboratory CV (x-axis) are plotted for countries on the left (I, blue=Italy; N, purple=the Netherlands; P, green=Portugal; U, red=United Kingdom; S, yellow=Spain) and manufacturers on the right (A, red=Abbott Architect; B, blue=Beckman Coulter UniCel DxC; O, green=Beckman Coulter Olympus AU; R, yellow=Roche Cobas; S, purple=Siemens Advia; D, brown=Siemens Dimension). Specifications for minimum, desirable, and optimum performance are indicated by the amber, gray, and yellow triangles, respectively.

Aggregated level

Results can also be reviewed at a higher aggregation level. Especially when a new analytical platform is considered, it is convenient to describe the performance as one number, rather than numbers for each of the analytes. The performance of a platform is expressed as the median TE/TAE ratio of the 17 analytes. When the observed TE of an analyte and TAE at the minimum performance level of that analyte are equal, the TE/TAE ratio is 1.0. At the ratio of 1.0, that analyte just meets the minimum performance specification. The higher the TE/TAE ratio, the worse the performance. The median TE/TAE ratio of the 17 analytes is an overall estimate of performance. Median TE/TAE ratios are shown in Table 3 (bottom line; detailed calculations in Supplemental Table 3) and are visualized in Figure 2. For comparison, the ratios of three analytes (Na, Ca, and CK) are also included in Figure 2. It can be seen that there is little difference in performance between countries or between manufacturers. For Na and Ca, the ratio is high, and for CK, the ratio is low in all countries and for all manufacturers. This also applies for the median ratio with all values within the minimum or just within the desirable performance specifications. Figure 2 also shows the overall ratio (irrespective of country and manufacturer) as an estimate of the analytical performance of the respective analytes: the minimum, desirable, and optimum performance specifications are met by 8, 2, and 1 analytes, respectively. Six analytes did not meet any of these specifications.

Figure 2:

Overall, country and manufacturer specific TE/TAE ratios.

The TAE/TE ratio is plotted on the y-axis with indication of ratios corresponding to specifications for optimum (yellow), desirable (gray), and minimum (amber) performance. Countries (ES, Spain; IT, Italy; PT, Portugal; UK, United Kingdom; NL, Netherlands) and manufacturers (Abb, Abbott Architect; Oly, Beckman-Coulter Olympus; Bec, Beckman-Coulter UniCel DxC; Roc, Roche Cobas; Adv, Siemens Advia; Dim, Siemens Dimension) are plotted on the x-axis. The median ratio of 17 analytes (red dots) and analyte specific ratios of three analytes (gray; Na, Ca, CK) are plotted per country (middle part) and per manufacturer (right part). Gray squares (left part) represent overall ratios for the respective analytes (ToPr, total protein; AP, alkaline phosphatase; ASAT, aspartate aminotransferase; HDLc, high-density lipoprotein cholesterol; Amyl, amylase; Gluc, glucose; ALAT, alanine aminotransferase; UrAc, uric acid; Chol, cholesterol).

Performance at analyte level

Table 3 and Figure 1 clearly demonstrate that the performance of the individual analytes K, uric acid, glucose, and cholesterol performed satisfactory: the minimum performance specification is met in all countries and by all manufacturers. The opposite applies to Na, Cl, and Ca: in that none of the countries or manufacturers met the minimum specification. For Mg, high-density lipoprotein (HDL) cholesterol, and total protein the performance can be considered borderline: in that some countries and manufacturers met the specification whilst others did not. For the enzymes the situation was more complicated in that despite efforts from the IFCC, approximately 20% of laboratories used methods that were not traceable to the IFCC RMPs. Percentages differed per enzyme, per country, and per manufacturer (Table 4). Non-traceable results were excluded from the calculations for Table 3 and Figure 1. The reliability of the remaining results for which IFCC-traceability was claimed is also questionable. Classification was different per country: traceability was either assumed by the laboratories, or assigned by the EQA organizers based on the results (a number of EQA organizers expressed concern that that the laboratories had insufficient awareness of traceability). This is also confirmed by the fact that the between-laboratory CV in Spain, Italy, Portugal and UK is approximately double that of the Netherlands where strict standardization was implemented as part of a concerted awareness and implementation initiative from the association of clinical chemists, manufacturers, and the EQA organizer [3], [13].

Performance at aggregated level

The horizontal distribution of the red dots in Figure 2 demonstrates that the overall performance in the countries and of respective manufacturers is not very different. A similar horizontal distribution is seen for the gray dots of Na, Ca, and CK (and all other analytes – Na, Ca and CK are just an example; see Supplemental Table 3). The TE/TAE ratios for Na (3.5) and Ca (1.7) are much higher than 1.0, indicating that the minimum performance specification for these analytes was not achievable if BV data are used to define performance specifications. This applied to all countries and to all manufacturers. CK is an example of the other extreme where low ratios were observed within the desirable or even the optimum performance specifications for all countries by all manufacturers. The study suggests that the analytical performance of general chemistry analysers is not related to differences in skills in the respective countries and is also not related to analytical platform of a specific manufacturer. However, the performance and therefore the state of the art for each analyte is very different.

Comparison with similar EQA concepts

There are only a few EQA schemes using commutable samples, most of them related to the group of Thienpont et al. [6], [7], [8], [9]. It is worth considering the differences between our two approaches in terms of the findings of our schemes and the use of slightly different samples.

For many analytes, our findings were comparable, e.g. glucose and cholesterol met, quality criteria whereas HDL cholesterol (Abbott) did not [8], neither did calcium and total protein meet the criteria [6]. We found that alkaline phosphatase (AP), alanine aminotransferase (ALAT), aspartate aminotransferase (ASAT), and lactate dehydrogenase (LD) were not traceable to the respective IFCC RMPs. We did not subdivide the transaminases based on the way of pyridoxalphosphate activation as previous investigations showed that traceability may not be substrate dependent [3].

However, some of our findings differed from the Thienpont group. They observed a 3.6% negative bias for chloride with the Roche Cobas system [7]. We believe that this highlighted a temporary effect on the Roche systems. We evidence this on the longitudinal data from the Dutch EQA program which has used commutable materials since 2005. Data from the years 2012–2015 show that the Cobas had a bias of −0.4, −0.6, −2.9, and −0.4% (Roche Modular bias is +1.0, +0.6, −1.1, and −0.3%). The “dip” seen in 2014 was only temporary and has now disappeared. Our data and that of Thienpont’s group may have been affected by this. It is possible that other contrasting findings, e.g. uric acid and Mg (Abbott) had a similar cause [6].

Although we and Thienpont’s group have both used commutable frozen human sera, there are some differences. We used fresh frozen human sera pools, whereas Thienpont et al. used panels of 20 fresh frozen single donation human sera. We do not know whether this complicates any comparison. The advantage of single donation is that assay non-specificity can be more easily detected, but a single donation has a maximum volume of approximately 200 mL and a proportion of each sample has to be used for targeting, homogeneity, and stability testing. In contrast, large sample pools include the option to recruit large numbers of participating laboratories, reissue the same pool over long periods, and the ability to test over a broader range of analyte concentrations. Furthermore, the costs of value assignment with RMPs is relastively lower for pools than for small single donations.

In this paper, we have chosen to focus our evaluation on the performance of groups (either manufacturers or countries) and not on individual laboratories. Results allowed the individual laboratories to evaluate their own compliance to analytical performance specifications in line with the proposals of the EFLM conference on analytical quality [17], which is only possible with a category-1 EQA scheme.