Impact of six sigma estimated using the Schmidt-Launsbyn vs. the Westgard equation in the Spanish type I EQA program

Materials

This study is based on the results of the SCR-EQA-SEQC^ML program. This program uses commutable control material from fresh human serum with a value assigned by reference methods (Category 1) [10], [11].

The control materials used for this study are described in the publication by Ricos et al. [10]. Control material was supplied by Stichting Kwaliteitsbewaking Medische Laboratorium Diagnostiek (SKML) and prepared at the MCA laboratory (Queen Beatriz Hospital, Winterswijk, Netherlands). A total of six vials of control material at different concentrations (six levels) stored at −80 °C were sent to each participant in a single shipment. Each vial was measured in duplicate for six consecutive days. The results obtained are available on the SCR-EQA-SEQC^ML website.

Results were included in the study when at least five laboratories participated in the peer group. For all measurands included in the program, means and coefficients of variation (CV, %) were calculated for each method-instrument at each level of concentration. The measurands included in the SCR-EQA-SEQC^ML program are shown in Table 1. The Table includes both the performance specification established by the EQA program based on biological variation [12] and the concentration at which the clinical decision level is defined. This study considers the results obtained collectively – not individually – by the laboratories taking part in peer groups in an EQA, which used the same analytical method/instrument and calibrator traceability [13].

Statistical analysis

The SV was calculated using two different strategies: the equation developed by Westgard (MW) [14], [15], and the Z-transformation method followed by the application of the S-L equation (S-LM). Calculations were made for each instrument-method group and for each of the measurands included.

Westgard strategy for calculating six sigma

Sigma values (SVs) were calculated by the Westgard method (WM) using SCR-EQA-SEQC^ML data for imprecision (CV, %) and bias (SE, %). Imprecision for each measurand and control level was calculated as the percentage of data dispersion. Bias was defined as the difference between the mean value at each control concentration and the value obtained with the method of reference. The specification for the total error (TE, %) of the SCR-EQA-SEQC^ML program was established as the limit of acceptability of data distribution (Table 1). SV was calculated by implementing the following formula to each peer group at each concentration level:

All parameters in the equation are expressed as percentages (%). The SV estimate was converted into DPMOs using conversion charts (https://sixsigmastudyguide.com/process-performance-metrics/).

Z-transformation and Schmidt-Launsbyn equation for calculating six sigma

SV was calculated through the Z-transformation strategy followed by the application of Schmidt-Launsbyn (S-LM) equation from the SD values obtained for each magnitude, control level and TE established as a specification by the SCR-EQA-SEQC^ML program (Table 1). Firstly, the limits to assess compliance with a specification were calculated as follows:

The reference value refers to the target value of the control material, as determined by a reference method (type I EQA program). Normal distribution is defined as a mean (µ) of 0 and an imprecision of 1 (σ). To normalize data distribution, the Z-transformation tool was used [16], which indicates the extent to which a result deviates from the optimal target before falling outside the specification limits. Likewise, Z scores were calculated for the two distribution limits obtained for the peer group at each concentration level. The following formulas were applied:

The difference between the standard normal distribution of the upper and lower z score defines the area under the curve (AUC) [3] contained within the specifications established. The difference between the AUC defined by the specifications and the calculated AUC accounts for the defects occurring during the process. The value of this area is multiplied by 10⁶ to calculate the number of DPMOs. Finally, the SV is calculated from the DMPO value obtained through the S-L equation [6], [7]:

Outlier analysis was conducted using the Tukey method [17].

Statistical analysis and plotting were performed using an Excel spreadsheet (Microsoft Corporation^®, Redmon, Washington, USA).

Comparison of the two methods

Based on the literature available, two methods were selected in the Clinical Laboratory to calculate SV. SVs have been traditionally calculated indirectly using the Westgard method (WM) by establishing a proportional relationship between analytical errors: total error (TE), bias and imprecision [18]. Alternatively, in the recent years, some authors have suggested calculating SM based on a direct estimation of DPMOs [19]. This strategy involves transforming data into a normal distribution through a Z-transformation process, which has been described as the most robust method for calculating SM [5]. The relationship between DMPOs and SV is not linear but exponential [20], [21]. In this study, the S-L equation was implemented to transform DMPOs into a sigma value exploiting the logarithmic relationship between these two values (S-LM). This equation yields a single, directly proportional SV for each DPMO value. In contrast, the WM estimates SV indirectly without considering the DMPOs. As a result, different bias/imprecision combinations generate the same SV (Figure 1).

Additionally, as analytical bias increases, aberrant (negative) SV values tend to appear from approximately 2 × 10⁵ DMPOs. When bias exceeds the established distribution limits (e.g. the quality specification for TE set by the EQA program) the aforementioned negative values begin to appear. This occurs due to the linear treatment of errors (primarily bias) assumed in the Westgard equation [3], whereas the relationship between SV and DMPOs follows a normal distribution [22]. No variations were noted in the behaviors described when all data are considered or when analysis is focused on values close to the clinical decision level. In the light of the problems that calculating SV by the Westgard equation entails, outliers were removed from data distribution (the highest bias values), resulting in a significant – but not total – reduction of negative SVs. On another note, no variations were observed in imprecision according to the method used to calculate SM (Westgard vs. S-L) or the treatment of outliers (presence vs. absence).

Likewise, the average SV for the SCR-EQA-SEQC^ML as calculated by the S-LM method did not vary significantly when outliers were maintained or removed, either considering the entire data set (3.06 vs. 3.10) or solely clinical decision levels (3.20 vs. 3.23). However, variation in the WM SV increased as a function of whether outliers were removed or not, both at the clinical decision level (0.52 vs. 1.33) or when all SCR-EQA-SEQC^ML program data were considered (0.31 vs. 1.14). The bias/imprecision ratio decreased after outliers were removed due to a reduction in bias. Therefore, the SVs obtained by the S-LM method are not influenced by outliers, in contrast to those calculated with the Westgard method, which are influenced by bias. The way bias is approached by each method definitely contributed to the differences observed in SV. The difference between the SVs obtained with the S-LM and the WM method is constant. Hence, WM underestimates the SV, ranging from 2.7 when outliers are included to 1.9 following outlier removal. Differences in SV between the two strategies ((S-L)-W) increase as bias increases (elevated bias/imprecision ratio) (Figure 2A and B).

Underestimation of the sigma value by the Westgard method hinders the achievement of the SCR-EQA-SEQC^ML quality objectives. The WM is significantly influenced by high bias values, thereby resulting in negative SVs. This problem is overcome with the S-LM method, which treats bias value more appropriately. Proper bias management contributes to an enhanced assessment of the performance of methods and instruments, as well as of the EQA program as a whole.

Performance of the EQA program

EQA programs are aimed at assessing analytical performance. The WM approach yields inappropriate sigma values ranging from 1.33 when outliers are considered to 1.14 when removed, both at the clinical decision level. In contrast, the S-LM strategy yields values ranging from 3.23 to 3.10, respectively. This way, the overall performance of the program would exceed the SV limit accepted in the clinical laboratory, established at three sigma. This sigma value provides a general picture of the EQA program performance. The availability of a sigma value for each measurand provides guidance as to the measures to be adopted to improve analytical performance, where appropriate. Only three measurands exceeded four sigma (CK, potasium and enzymatic creatinine); seven measurands received a 4-3 sigma; and eight had a <3 sigma. In view that three sigma is the lowest value, clinical laboratories and/or the in vitro diagnostics (IVD) industry should intensify efforts to improve analytical performance in measurands with a SV<3. Moreover, performance should be enhanced in measurands with a 4-3 sigma to reach sigma 4. Creatinine deserves special attention, as the most precise method –enzymatic creatinine– exhibited a better sigma value (SV>4) as compared to the Jaffé method (SV between 4 and 3). This is a clear example of how clinical laboratories should use the analytical methods with the best performance. The SM calculation method has a significant impact on the SV obtained and affects the fulfilment of EQA program objectives.

The six sigma metrics emerges as a valuable tool in EQA programs, as it provides a picture of the analytical performance (imprecision and bias) of laboratory methods and instruments through a single indicator. In addition, SM facilitates the harmonization of different EQA program, providing a benchmarking parameter.

Currently, the level of implementation of six sigma metrics within EQA programs is limited. The Dutch program SKML provides SV in the performance assessment report, but uses the Westgard method, which has important limitations, as demonstrated in this study. Table 3 displays the SCR-EQA-SEQC^ML measurands grouped according to their six sigma value. The assessment of analytical performance presented in this table is cumulative for each measurand. Thus, each EQA program must be evaluated separately for each instrument and measurand. However, this is out of the scope of our study and will be addressed in future studies. In general, we encourage the IVD industry to enhance analytical methods for measurands with a low sigma value to achieve the analytical performance established based on biological variation. For instance, in the case of sodium, direct potentiometry should be used instead of indirect potentiometry to improve its six sigma value.

In conclusion, the S-LM method complies with the requirements for a correct calculation of SM, including ensuring the normal distribution of the data set. In addition, it overcomes the limitations of the WM methodology. Thus, the S-LM method calculates DMPOs directly and enables subsequent estimation of the SV, in contrast to the Westgard equation. With the S-LM, each DPMO value corresponds to a single SV. S-LM yields more robust SVs, as they are not influenced by outliers. Unlike WM, the S-LM method involves a non-linear treatment of bias. This approach reduces the impact of bias on SV. The results of this study underline the significant impact of bias on SV and the relevance of selecting the most appropriate strategy for calculate SM. A robust SM calculation method will pave the way to its inclusion in EQA programs, thereby resulting in a more objective assessment of analytical performance.