Proficiency testing as a tool to assess quality of data: the experience of the EU Reference Laboratory for chemical elements in food of animal origin

Rationale

Participation in PTs organized by the EURLs is mandatory for laboratories that are designated as a reference in each Member State of the EU. As part of this institutional task, the EURL-CEFAO organized 30 exercises in the period 2006–2018 (Table 1). They were primarily planned as a long-term program to promote harmonization of performance within the network of the NRLs. In particular, the objectives pursued were the following:

1. monitor and improve the performance of their analytical methods;

2. verify the effectiveness of the corrective actions taken (when necessary) by repeating the same matrix;

3. provide long-term follow-up of their performance;

4. promote the improvement of the quality control system

The PTs were focused on chemical elements and foods for which a maximum level (ML) is set in relevant regulations [18], [19], [20], [21], [22] but, starting from 2012, new combinations of analytes/matrix were considered as well. These combinations were selected on the basis of information received from the European Commission (e.g. discussion on the introduction of new limits in relevant regulations; emerging topics) as well as following specific demands from the NRLs.

Thirty PTs were conducted as of 2006 following the procedure schematized in Fig. 1: production and characterization of materials were carried out by the EURL staff as well as the statistical evaluation of results (including the evaluation of participants’ performance and issuing of reports).

Fig. 1:

Scheme of the approach followed by the EURL-CEFAO to conduct PTs.

After an adequate planning, the management of a PT was articulated in some fundamental steps: preparation of PT material; evaluation of its adequacy; statistical evaluation of the participants’ results; assessment of the participants’ performance.

Production of material and assessment of its suitability for PT

Over the years the EURL-CEFAO produced about 4000 PT test items based on different kinds of matrices [23], [24], [25]. Therefore, proper procedures were developed and optimized in order to produce materials adequate for PTs purposes and in a physical form as similar as possible to the samples analyzed by laboratories during their routine work. Liquid samples (e.g. milk), partially liquid samples (e.g. honey), freeze-dried material (e.g. meat, fish, offal, milk) and frozen samples (e.g. meat, fish, offal) were prepared as PT test items. The analytes concentration was often adjusted around MLs or other values of interest. This adjustment was performed either spiking the starting material, purchased at retail stores, with standard solutions of chemical elements or diluting it with a suitable similar matrix. The former was used if the basal content of analytes was lower than the concentration value planned for PT, the latter was used if the basal content was greater than the concentration value foreseen in the final material (e.g. mercury in fish).

Freeze-dried samples were periodically included in the exercises so as to supply the NRLs with samples easy to handle and store that they could use for their internal scope as well.

Once PT test items were prepared, their adequacy for PT purpose had to be assessed by testing stability and homogeneity. In particular, stability had to be guaranteed at least for the duration of PT and the level of homogeneity had to be enough to be confident that differences among test items would not significantly affect the evaluation of participants’ performance. As for the homogeneity, it was evaluated against restrictive criteria using precise analytical methods making the participant feel more confident about the quality of the samples. In fact, the sufficient homogeneity of the PT material was evaluated following an internal procedure based on the International Harmonized Protocol (IHP). The procedure fixes a criterion on the precision of the analytical method to apply: the criterion to be compliant with is that the ratio between the analytical standard deviation (σ_an) and the standard deviation for proficiency assessment (σ_pt) has to be lower than 0.5. The fulfillment of this criterion is recommended to avoid that possible differences among the samples (different test items) are covered up by the analytical variability of the analytical method used. Obviously, the lower σ_pt, the lower σ_an must be to comply with the criterion. As the σ_pt used by the EURL-CEFAO is more restrictive than Horwitz (used for PT in food sector), σ_an must be low enough. Furthermore, the data collected by the EURL-CEFAO over the years showed that this ratio was usually lower than 0.2 so the methods for evaluating the homogeneity were more precise than methods commonly used by the EURL-CEFAO.

As discussed below, all the assigned values were derived by using the consensus from participants. This approach could lead to a biased value but its reliability could be considered a strength point of the EURL-CEFAO scheme that further assured participants about the value of the scheme. In fact, the laboratories belonging to the network had a high level of competence as they were the outstanding ones of the EU. This aspect combined with the strong analytical background of the experts performing the statistical evaluation of results assured the reliability of the assigned values, which was confirmed by the comparability of assigned values (x_pt), values of homogeneity (x_hom) test and expected values (x_exp). The expected value was the value of concentration calculated taking into account the basal content of analytes in the sample and the known spiking.

In addition, some studies were conducted to derive x_pt from a calibration against the reference value of suitable certified reference materials (CRMs). The values assigned using this procedure resulted comparable to the values obtained from consensus, demonstrating the adequacy of the statistical procedure applied to derive the assigned value.

As an example, the expected value (based on the gravimetric spiking), the assigned value, the value from homogeneity test and the value assigned against CRMs are compared in Table 2 for two PTs on milk (14^th PT-2 and 16^th PT). CRMs used for deriving the assigned value (x_pt-CRM) were BCR063R (certified only for Pb) and ERM-BD 150 (certified for both Cd and Pb) for the 14^th PT and the 16^th PT, respectively.

Statistical evaluation of the results

Before starting a PT it is necessary to set the standard deviation for proficiency assessment (σ_pt) as well as to define the procedure for deriving the assigned value. These “reference values” are key to evaluating the performance of the participants. Among the possible approaches, the EURL-CEFAO chose the consensus from participants to derive the assigned value and the fitness for purpose to set σ_pt. The “consensus” is a commonly used approach but the organizers of PTs should be aware of some disadvantages such as the possibility of assigning a biased value. As for the EURL-CEFAO, the probability that a bias occurred in the assigned value was minimized. Firstly, because the results were considered also from an analytical point of view and rejected a priori in case of doubts about how they were obtained. Secondly, because the possible presence of statistical subpopulations, linked to the analytical techniques used, was thoroughly investigated and adequately evaluated.

The σ_pt most commonly used in food sector is derived from Horwitz–Thompson equation (σ_H) that is exclusively based on the concentration value without taking the analyte and/or matrix into consideration. The EURL-CEFAO adopted more restrictive values set according to specific algorithms that were developed taking into account the analyte, the matrix and the level of concentration. This approach was followed because the NRLs were required to perform better than ordinary laboratories. Therefore, the resulting values of σ_pt corresponded to the level of performance that the participants in the PT were expected to achieve. Furthermore, these value were also conceived to correspond to low enough values taking into account the expertise and the role of the NRLs. Both x_pt and σ_pt were usually set by applying robust statistics following a procedure compliant with ISO 13528:2015 [26].

Evaluation of the participants performance

The participant’s performance was evaluated as z-score (z_i) that is calculated according to formula (1).

where x_i is the PT result submitted by the participant. The performance was acceptable for |z|≤2.0, questionable for 2.0<|z|<3.0 or unacceptable for |z|≥3.0.

The participation of the NRLs to the PTs organized by the EURL-CEFAO and the constant follow-up action led to a general improvement of the network performance as the percentage of laboratories satisfactorily performing increased till to reach a steady good performance for almost all analytes/matrix combinations.

As an example, in Fig. 2 the results of similar exercises organized over the years are reported. The PTs and chemical elements used for comparison were selected in order to compare exercises with a similar difficulty. Therefore, the same matrix in the same physical form (e.g. freeze-dried bovine meat vs. freeze-dried bovine meat) as well as similar assigned values of chemical elements were compared.

Fig. 2:

Comparison of the network performance for Cd, Pb and Hg between PTs on the same matrix and similar values of concentration. The color of the bars represents the performance: green for satisfactory, yellow for questionable and red for unsatisfactory.

In terms of z-score the percentage of satisfactory results (green) significantly increased, whilst the number of unsatisfactory (red) or questionable (yellow) data decreased.

The EURL-CEFAO used Shewhart control charts as a graphical means to combine performance scores over several PTs [27]. In general terms, this visual presentation is far more explicative than the interpretation of a single numeric score as it allows participants to check their performance over the time and the PTP to monitor the general performance of the network, also pointing out possible crucial points of the exercises. The charts were updated after each PT and distributed to participants so they could promptly evaluate their performances for a specific matrix/analyte combination. An example of this kind of chart, related to PTs on meat, offal, fish and processed food, is reported in Fig. 3. It is evident that the participant has never had problems to analyse arsenic in these matrices whilst its performance for cadmium and lead has been improved over the exercises.

Fig. 3:

Shewhart control chart of z-scores for As, Cd, Pb, Hg and Cu in meat, fish and offal from a single participant (period from 2005 to 2018).

The improvement of the EURL-CEFAO network was not only proved by the increased percentage of satisfactory z-scores but it was also demonstrated by the harmonization of the submitted results. In fact, the indicators of data dispersion calculated by the participants’ results, namely standard deviation of the mean (SD) and robust standard deviation (s*), decreased over the PTs confirming that data dispersion became more and more narrow. In particular, since 2010 SD resulted lower than σ_pt and close s*. This trend is clarified in Fig. 4 where the following ratio are reported as percentage: SD/x_pt, σ_pt/x_pt and s*/x_pt.

Fig. 4:

Comparison of data dispersion indicators over PTs on meat: SD/x_pt (open square); σ_pt/x_pt (open diamond) and s*/x_pt (open triangle).