Determination of Sigma metric based on various TEa sources for CBC parameters: the need for Sigma metrics harmonization

Introduction

World Health Organization (WHO) defines the concept of patient safety as “the absence of preventable harm to a patient during the process of health care and reduction of risk of unnecessary harm associated with health care to an acceptable minimum” [1]. Regarding patient safety from the past to the present, laboratory medicine has been focused on the risk of negative consequences associated with laboratory tests and their prevention. Clinical laboratories play an essential role in the management of patient, providing important information critical to diagnosis, treatment and monitorization of diseases [2, 3]. From this point of view, it is very important that laboratories produce reliable results by targeting the strict quality assurance program.

In clinical laboratories, the total testing procedure is categorized in three phases such as preanalytical, analytical and postanalytical. Despite errors in laboratories are more frequently encountered during pre- and post-analytical processes, analytical errors constitute approximately 10% of total laboratory errors and the consequences negatively affect the patient safety. Although comparatively less error rates appear in the analytical phase, it is obvious that the issue of analytical quality is of high importance in terms of determining laboratory errors, and further improvement is supposed to be inevitable in this area [4], [5], [6].

The performance of diagnostic laboratories can be analyzed by the implementation of Sigma metrics during any phase of laboratory process [7]. Six Sigma has been implemented in clinical laboratories since less than two decades [5]. Six Sigma is a modern quality management tool that counts the defects which are known as defects per million (DPM) or quantifies the number of errors existing in an assay, and it can be applied for the improvement of quality control design [8]. The DPM are converted into a scale of Sigma values ranging from 0 to 6, in which six or greater values indicate world-class performance (3.4 DPM). The minimum Sigma value that can be acceptable for the performance of analytical process is three Sigma [9]. The accurate number of errors can be measured by conducting some procedures such as combining imprecision, bias, and total allowable error (TEa) [10]. Unsatisfactory analytical performances ranging from 3 to 4 Sigma have been described in laboratory medicine [2]. TEa defines the maximum potential of errors for patient results that a single result can show with a certain probability and presents the performance specifications and goals which are required for assays to be able to provide clinically valuable results [11], [12], [13]. One of the great challenges in using Sigma metrics is the selection of appropriate TEa goal because many measurands of TEa values show an alteration according to the source. The hierarchy for the selection of an appropriate TEa is recommended and introduced by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM), and the performance specifications depends on three models which are clinical outcomes (Model 1), biological variability (Model 2), and state of-the-art (SOTA) (Model 3). Despite the fact that a well-recommended hierarchy is currently available, which can be taken into consideration while selecting an appropriate TEa, there is no common decision concerning the most convenient source for a given measurand [5, 14], [15], [16].

A complete blood count (CBC) test is one of the most frequent tests applied in laboratories. Hematology analyzers are automated to a large extent, and the interpretation of results in an accurate way requires knowledge of the analytic performance of instruments which are expected to provide clinical significance of the results [17]. There are a limited number of publications related to the assessment of analytical performance of hematological parameters on Sigma scale. The purpose of the present study was to analyze the performance of Sysmex XN-1000 hematology analyzer by utilizing Sigma metrics based on different TEa source and to reveal the variation in Sigma metric calculation.

Materials and methods

This study was conducted in the Clinical Biochemistry Laboratory of the University of Health Sciences, Diskapi Yildirim Beyazit Training and Research Hospital and approved by the Ethical Committee of our hospital. White blood cell (WBC), red blood cell (RBC), hemoglobin (HGB), hematocrit (HCT), mean corpuscular volume (MCV), mean corpuscular hemoglobin (MCH), mean corpuscular hemoglobin concentration (MCHC), red cell distribution width-standard deviation (RDW-SD), platelet count (PLT), plateletcrit (PCT) and mean platelet volume (MPV) were analyzed in Sysmex XN-1000. For all these analytes mentioned above, commercial internal quality controls (IQC) provided by the vendors (XN CHECK^® and e-CHECK^®; SYSMEX Corporation, Chuo-ku, Kobe, Japan, Product Code; L1;213484, L2;213485, L3;213486) were utilized. Three levels of IQC data were collected retrospectively over a period of six months from June 2020 to December 2020. The analyzer and reagents were used according to the instructions outlined by the manufacturer and calibrated according to routine standard operating procedures before any control analysis. All specimens were run on a single instrument. Our laboratory also participates monthly in the External Quality Assessment Scheme (EQAS) of RIQAS (Randox international quality assessment scheme, Randox Laboratories, United Kingdom). The results of EQAS samples between January–December 2020 were included in the study.

Results

The Sigma values were determined by using five TEa sources, as shown in Figure 1, for CBC analysis and a comparison was made between the Sigma values depending on the source. The TEa of each assay was based on five standards: 1) Desirable biological variation (BV) database specifications (BV-EFLM) [18], 2) American Clinical Laboratory Improvement Amendments (New criteria of CLIA) [19], 3) State of Art (SOTA) [20, 21], 4) Standards of Rili-BAEK (Revision of the “Guideline of the German Medical Association on Quality Assurance in Medical Laboratory Examinations [22], and 5) Standards of Spanish (Minimum) [20].

Figure 1:

Sources for TEa specifications used to calculate Sigma metrics for CBC parameters.

Bias%, CV%, and TEa values of CBC parameters are shown in Table 1. For all the analytes, CV values were found within acceptable limits according to manufacturer claims. Sigma metrics of CBC parameters according to five different standards for TEa values were available in Table 2 for each level of IQC. Based on Table 2, according to BV-EFLM (Desirable) TEa, the highest Sigma value was achieved by WBC count for each level and the lowest value was achieved by RDW-SD parameter. The Sigma calculation based on Spanish TEa showed that seven assays achieved Sigma value ≥3 on Sysmex XN-1000 hematology analyzer (Table 3).

In addition to the calculation of the Sigma metric, the Normalized Sigma metric Method Decision chart was utilized to make a graphical summary of instrument performance (Figure 2).

Figure 2:

Normalized method decision chart for CBC parameters comparing Sigma metrics calculated using BV-EFLM, CLIA, SOTA, Rili-BAEK, Spanish TEa targets. X-axis represents imprecision (CV), Y-axis represents bias.

Diagonal lines separate the graph into different Sigma zones, which correspond to the different Sigma metrics levels.

In the review of literature, six studies conducting a research on Sigma metric analysis of CBC were found [23], [24], [25], [26], [27], [28], [29]. The characteristics and results of these studies are summarized in Table 4.

Discussion

Although the Sigma metrics model sets up an objective process to analyze the performance of a method, medical laboratories encounter great difficulty in choosing the most appropriate quality specifications for quality planning [30]. However, the optimum TEa, bias and CV to calculate six Sigma remain unclear, particularly when the sources of TEa vary between laboratories [31].

Preliminary list prepared by the EFLM Task and Finish Group on Allocation of Laboratory Tests to Different Models for Performance Specifications, most CBC components are assigned to model 1 and model 2, as indicated by Ceriotti et al. [32]. It has been proposed that the analytical performance specifications for Hb, platelets and neutrophil measurands of the CBC should be based on clinical outcomes (Model 1). However; no reliable clinical outcomes have yet been available. Model 2 offers an alternative approach for the establishment of analytical performance specifications, and it also provides the opportunity to use BV data for many other clinical applications. Laboratories establish BV based quality goals by evaluating the inherent biological variation of the analyte for three analytical performance specifications: minimum, desirable, and optimum. Many laboratories still use TEa based on the “desirable” specification. In this regard, the EFLM Biological Variation Working Group managed a very important multicenter study which is called European Biological Variation Study (EuBIVAS). The EFLM Biological Variation Database is intended to provide access to the continuously updated and easily accessible database on biological variations. Additionally, the EFLM conducts a meta-analysis of papers on the components of biological variations from which the TEa is derived and according to this information laboratories could decide TEa depending on what is possible and suitable for the laboratory [32], [33], [34].

In this study, five different quality goals have been selected to evaluate the analytical performance of CBC parameters by Sigma metrics. This study aims at evaluating the performance of Sysmex XN-1000 hematology analyzer by using Sigma metrics based on different TEa sources and to determine the effects of these variations in Sigma metric calculation. Our study demonstrates that the implementation of the six-Sigma concept on QC data commonly provides Sigma values below the minimum acceptable value of 3.0 and observations differ depending on the TEa source data used. Inevitably, the question that comes up is whether the six Sigma concept is useful for quantifying the analytical performance of hematology parameters. When using different TEa sources for calculating Sigma metric, a wide range of Sigma values was obtained. Multiple sources of TEa are available with no standardization for TEa specifications for laboratory tests around the world [5, 35, 36]. Our study used five different TEa standards to calculate Sigma values of CBC parameters. One of them is based on The EFLM Biological Variation Database, available at https://biologicalvariation.eu/ [18]. The standard based on BV- EFLM is too strict than others. Rili-BAEK is the German EQA performance specifications of the Rili-BAEK rules [22]. CLIA represents the US CLIA regulations, also new CLIA “proposed rules” were published and defined new criteria for acceptable performance for proficiency testing [19]. Minimum Spanish EQA is the minimum consensus performance specifications of multiple Spanish EQA surveys [20]. The last standard is based on a set of SOTA performance specifications for hematology parameters published by the International Journal of Laboratory Hematology. Vis and Huisman collected acceptable limits for CV and bias values for the blood count parameters according to SOTA. Based upon these data, Westgard has calculated TEa by using SOTA data [19, 20]. TEA values were not available for all CBC parameters in the five TEa sources we used, except BV and SOTA (most parameters). They are quite useful for us to gain insight into areas where previous resources have provided any significant benefit. Accordingly, CLIA does not provide any specifications (TEa) for more of the differentials.

The results of the present study indicate that “desirable” standards based on BV-EFLM data were the most stringent ones because only WBC reached to six Sigma. When using standards of Spanish, all of parameters Sigma score were above three. These results also conclude that Sigma scores have a significant fluctuation based on which TEa sources are used. Furthermore, this study demonstrates that the majority of the basic CBC parameters do not provide the acceptable Sigma value based on desirable BV- EFLM data. As a limitation of this study, we used internal control materials, not human samples, for conducting six Sigma calculations in this study. Although whole blood sample is still the ideal material for quality control of hematology analyzers, this type of specimen cannot be routinely used for many different reasons. For this reason, hematology quality control materials are different from freshly collected patient samples and inevitably have a source of uncertainty. However, in protocol of BV, a number of samples are obtained from a number of individuals whose health status is known and collected during a certain period of time with short, medium or long sampling interval [37, 38]. This means that we should not ignore a certain degree of diversity between control material and whole-blood samples in the use of the TEa based on BV.

The review of published data, predominantly measuring basic CBC parameters (WBC, RBC, Hb, Htc and PLT), reveals that analytical performance on Sigma values are above three, in which more flexible old CLIA standards and TEa values based on BV data updated by C. Ricos et al. [39] were used (Table 4) [23], [24], [25], [26], [27], [28], [29]. By the comparison of various studies, the discrepancy in Sigma metrics can be attributed to selecting different TEa source, method differences in CV and bias calculation. Although analytical problems have improved over the years due to the assay standardization, improved laboratory technology, well-defined rules for IQC as well as effective quality assurance schemes e.g., analytical quality still remains an important problem. Westgard has argued that “estimates on Sigma metric for common clinical chemistry and coagulation tests are unsatisfactory, ranging from 3 to 4 Sigma, at best” [40]. Analytical performances resulted in unsatisfactory outcomes have been identified not only in the area of clinical chemistry, but also in the fields of hematology, coagulation, and molecular biology testing [2].

Many difficulties can be faced that affect the comparability of calculated Sigma metrics among clinical laboratories. One of the most challenging issues of six Sigma is identifying the most appropriate performance specifications. Since there is not an existing quality goals (TEa) universally accepted, harmonization is necessary. Secondly, bias can affect the Sigma metric although bias is typically calculated through the use of IQC, and through either EQAS or rarely, reference materials. The highly recommended method is to use reference materials to measure bias. In addition, the differences in the methods used for CV calculation vary between laboratories. Components in the equation used to calculate the Sigma value such as TEa, bias, and CV remain unclear and vary between laboratories. Thus, there is no specific guideline of a harmonization protocol for Sigma calculation [7, 10]. The results of this study put forward different Sigma values for different parameters that were calculated using different source of TEa. Accordingly, this shows similarity with the study of Sharkawy et al. [41], who suggested a harmonization protocol for Sigma calculation. They provided a road map for TEa selection, bias and CV calculation so as to establish a harmonization between different laboratories. Moreover, Varela B. et al. [30], also proposed a selection algorithm for TEa source with the help of a graphic tool, through the use of IQC and EQAS performances, by which laboratories can evaluate TEa source that best suits the test analytical performance.

In conclusion, although Sigma metrics is a self assessment tool for performance analysis of various test parameters in the laboratory, the largest challenge of Sigma metrics is the selection of appropriate TEa goals and universally lack of a harmonized approach for Sigma calculation. This study highlights the need for Sigma metrics harmonization and standardization. Each laboratory is supposed to select TEa goal depending on clear standardized criteria of selection. Scientific societies and organizations should also make more efforts in harmonizing the entire Sigma metric process. Additionally, the result of the low Sigma values of most CBC parameters is that the use of performance goals based on BV data is extremely challenging in common clinical use. Laboratories, even as they acknowledge the controversy and challenges that will continue to feature in the use of Six Sigma, further studies should be carried out adopting the concept of harmonized approach.”