Evidence-based assessment of the application of Six Sigma to infectious disease serology quality control

Introduction

Numerous papers have addressed the concept of Six Sigma applied to medical laboratory testing [1], [2], [3], [4], [5]. The Six Sigma concept was first implemented by the Motorola Corporation in 1986 [6]. In the manufacturing industry, the number of defects of a particular process can be counted or estimated and then converted to a defects-per-million (DPM) ratio [3]. A process that reports 3.4 DPM episodes is defined as a Six Sigma process. This approach has been applied to many industries including the health industry and is used to improve quality, as practitioners implement practices to drive down the numbers of defects [7]. In medical laboratory testing, rather than counting defects, the Sigma metric (SM) of a testing process is estimated by using measured imprecision and bias of quality control (QC) results, and an estimated or defined allowable total error (TEa) [2], 3], 8]. Using these parameters, SM can be calculated as:

where CV% is the percentage coefficient of variation of the population.

Recently, several authors have described the application of the Sigma metric concept to QC processes for infectious disease serology [9], [10], [11], [12]. Each author of these recent studies has applied different assumptions relating to the determination of imprecision, bias and TEa. Some of these assumptions are not evidence-based, in particular the assumption that bias can be ignored and that TEa can be defined as the difference between the mean of QC sample test results over time and the assay cut-off value. This paper critically reviews and compares the approaches used by each of the four studies, and uses real, comprehensive QC data to demonstrate how these assumptions can impact the estimation of Sigma metrics of an assay.

Methods

Four studies [9], [10], [11], [12] that applied Sigma metrics to infectious disease serology were reviewed and the methods of determining bias, CV% and TEa were critically assessed. To highlight the impact of each assumption in relation to the estimation of bias, CV% and TEa, real laboratory-derived quality control data were obtained from the National Serology Reference Laboratory, Australia (NRL) QConnect™ infectious disease QC program (https://www.nrlquality.org.au/products-services/qconnect/). Laboratories participating in the QConnect™ QC program used Optitrol™ QC samples (DiaMex GmbH, Heidelberg, Germany) which is optimised for each test platform. The QConnect program currently covers all major serology and molecular platforms used across well-resourced countries including those in Europe, North and South America and Oceania, including but not limited to those assays manufactured by Abbott Diagnostics, DiaSorin, Grifols, Roche, and Siemens. The Optitrol QC samples are manufactured to ensure minimal QC lot-to-lot variation. All participating laboratories enter results into NRL’s internet-based software (EDCNet™). For each of the papers reviewed, the assumptions used to define the three elements used to estimate Six Sigma (bias, imprecision and TEa) were tested using QC data extracted from EDCNet™. Unless identified otherwise, QC test results were obtained from a single QC lot number. The period of time the QC were extracted are specified for each example.

The assumptions used to the estimate bias in the studies under review were tabulated and compared. To assess real-world bias experienced by laboratories using the same Abbott Alinity i and ARCHITECT assays used in three of the four Six Sigma studies, QConnect™ data was extracted over a 10-month period from 1st March to 31 December 2023. The bias experienced by laboratories using three different QC lots tested in the Abbott assays was calculated (peer group mean divided by the laboratory mean expressed as a percentage).

The estimated imprecision was calculated from results of testing a single QC lot tested on the same Abbott Alinity i and ARCHITECT assays from data entered into EDCNet™. Data were collected over 10-, 3- and 1-month periods starting from 1st March, 1st October and 1st December 2023, respectively, until 31st December 2023. The CV% was calculated using SD/mean of results reported for the time period.

The methods used by each of the four papers reviewed to estimate TEa was compared and tabulated. The selection of the QC samples used to define TEa, and their suitability for TEa estimation, was critically assessed.

Results

A summary of the key elements used to calculate Sigma metrics by each of the four reviewed papers is presented in Table 1. All four studies used data derived from commercial infectious disease assays. Three of the four studies used data from blood bank screening of donors for blood-borne infections [9], 11], 12]. The other study reviewed data from toxoplasma IgG and IgM testing in a clinical setting [10]. Three of the four studies used automated chemiluminescent assays (CMIA) [9], [10], [11] and one used microtitre plate enzyme immunoassays (MTP-EIAs) [12], which typically have greater imprecision.

Estimation of imprecision

Each study calculated imprecision differently. Three studies used a simple calculation of CV% by dividing the standard deviation (SD) of QC results by the mean of the QC sample reactivity [9], 11], 12], whereas the other study [10] used a pooled CV% incorporating the variation introduced by reagent lot changes (Table 1). The study using the pooled CV% calculation [10] used data collected over a 10-month period, therefore included multiple reagent lot changes. Two of the studies using a simple SD/mean calculation used data from 1- or 3-month periods, where reagent lot-to-lot change was limited. The fourth study did not specify the number of reagent lots or time period of data collection [12].

The CV% of an assay can differ considerably depending on the time period of data collection and the calculation method. The CV% from a single QC lot tested on the same Abbott assays over a period of 10-, 3- and 1-month period was calculated (Table 3). The CV% of QC sample results reported by each laboratory varied significantly, ranging from less than 2 % to greater than 20 %. However, the differences in CV% is not a reflection of the assay performance, but an anomaly caused by the time period of data collection. If the QC data are collected from a single reagent lot, the CV% is generally smaller than if collected over a longer period of time using multiple reagent lots.

Table 3:

Range of coefficient of variation (CV%) reported to EDCNet by laboratories testing relevant OptiTrol™ quality control over a 1-, 3- and 10-month period ending 31st December 2023. The number of laboratories and total number of results and calculated CV% range for individual laboratories and the peer-group are presented.

Assay	1 month			3 months			10 months
	CV% range	No. of laboratoriesa (no. of total results)	Peer-group CV%	CV% range	No. of laboratoriesa (no. of total results)	Peer-group CV%	CV% range	No. of laboratoriesa (no. of total results)	Peer-group CV%
Abbott Alinity i HIV Ag/Ab Combo (anti-HIV-1)	2.66–16.15	22 (644)	13.99	2.80–17.42	38 (2,784)	13.84	4.28–18.96	43 (6,645)	12.94
Abbott Alinity i HIV Ag/Ab Combo (HIV p24 ag)	8.93–24.40	7 (245)	14.57	6.78–24.81	9 (755)	16.12	6.63–21.23	9 (1,642)	15.27
Abbott Alinity i Anti-HCV II	2.11–11.84	23 (667)	5.60	2.36–7.89	38 (2,876)	5.66	2.97–9.07	43 (6,856)	5.77
Abbott Alinity i HBsAg Qual II	2.24–5.66	15 (459)	7.05	2.62–9.73	28 (1,814)	6.11	2.75–14.93	34 (5,112)	9.50
Abbott Alinity i Syphilis TP	1.67–3.56	12 (314)	3.36	1.93–4.97	26 (1,425)	3.88	2.14–5.26	32 (4,085)	4.11
Abbott Alinity i Toxo IgG	3.43–3.56	2 (90)	4.60	3.17–4.82	4 (322)	5.15	4.27–5.30	5 (812)	5.17
Abbott Alinity i Toxo IgM	3.58–7.50	3 (90)	8.94	4.26–9.48	3 (250)	10.75	5.13–9.48	3 (392)	8.73
Abbott ARCHITECT Ag/Ab Combo (anti-HIV-1)	3.58–11.75	6 (128)	9.43	4.32–18.18	13 (620)	11.09	3.39–15.78	20 (1,728)	12.95
Abbott ARCHITECT Ag/Ab Combo (HIV p24 ag)	5.41–20.96	10 (305)	14.94	5.86–19.69	12 (954)	14.66	8.17–17.96	12 (2,009)	13.84
Abbott ARCHITECT Anti-HCV II	2.73–11.33	4 (81)	6.53	2.98–12.08	12 (580)	6.64	3.29–11.95	20 (1,705)	6.48
Abbott ARCHITECT HBsAg II	3.17–10.17	4 (82)	11.79	3.24–12.99	12 (493)	9.97	2.75–12.75	20 (1,499)	10.98
Abbott ARCHITECT Syphilis TP	2.51–2.62	2 (49)	5.97	2.35–11.97	10 (497)	5.34	1.90–11.97	15 (1,428)	4.73
Abbott ARCHITECT Toxo IgG	3.63–3.63	1 (22)	3.77	4.22–5.27	3 (109)	5.05	4.15–5.27	3 (184)	5.03
Abbott ARCHITECT Toxo IgM	3.10–3.10	1 (19)	3.10	3.71–6.13	2 (79)	6.19	5.30–10.64	2 (260)	9.92

1. ^aThe number of laboratories does not include laboratories that reported less than 10 data points in the given time period.

In the calculation of Sigma metrics, the TEa, with or without bias, is divided by the estimated CV%. Therefore, as the CV% decreases, the Sigma value increases. Figure 1 presents representative data sets from two laboratories testing the Abbott Alinity i HIV Ag/Ab combo assay, demonstrating how CV% change over time as new reagent lots are introduced. Each laboratory tested the same QC sample in the same assay (but different reagent lots) over the same time period. As new reagent lots with differing performance are introduced, the QC reactivity changes, as does the CV%. Where the new reagent lots have similar performance, the CV% of the QC sample results remains low for the same time period.

Figure 1:

Two Levey-Jennings charts from two separate laboratories testing the same lot number of quality control material (OptiTrol™ Blue lot DM22121) on the same assay (Abbott Alinity i anti-HIV combi assay) over the same time period. One laboratory (left) displays reagent lot-to-lot variation whereas the other laboratory (right) has minimal reagent lot-to-lot variation. The percentage coefficient of variation (CV%) of each reagent lot and the overall CV% for the time period are presented. The grey horizontal lines indicate the mean plus and minus 3 standard deviations calculated on the QC results obtained by that laboratory over the specified time period.

Discussion

Six Sigma estimation is not a statistical process control methodology for monitoring QC results but informs decision as to the application of QC rules [2], 3], 15]. It has been demonstrated previously that the use of all Westgard rules in infectious disease testing introduces an unacceptable level of false rejections, leading to wasted time and resources in unnecessary investigations [16], 17]. If an assay was deemed Six Sigma or greater, then according to Westgard Sigma Rules, only the 1^3S rule for two control measurements is required to monitor Six Sigma assays [2], 3], 15].

However, the literature is unclear as to the meaning of what constitutes “two control measurements”. It may be interpreted as testing the same QC in duplicate, using two different levels of positive control or employing a positive and negative control. As indicated in one of the studies reviewed, there was an 87 % reduction in “out-of-control” events during the 3-month study period when replacing classic Westgard rules with Westgard Sigma rules, reducing the number of “out-of-control” events from 111 to 15 [9]. Note that, extrapolating these data, the application of classical Westgard rules over a full year would have caused almost 400 false rejections and associated investigations in this laboratory. The authors concluded that this reduction “reduces false rejections but also saves cost and time spent in re-runs and troubleshooting” indicating that the rejections arising from the other Westgard rules were not clinically important. This finding aligns with previously reported comparison of different QC monitoring approaches [16], and highlights that laboratories using classical Westgard rules for infectious disease testing should reconsider their approach to QC processes.

Yet, the use of a single 1^3S rule poses new questions. There is no advice given as to the data set used to establish the mean and SD If the first 20 data points are used as specified in the CLSI Guidelines [18], then the acceptance range will depend on whether the results are obtained using the same or multiple reagent lots, and the performance of those lots. As demonstrated in Figure 1, the CV% can vary considerably. If the CV% is small, then it is more likely that the introduction of a new reagent lot will trigger 1^3S rule rejections forcing more unnecessary investigations. Many laboratories, when faced with a change in reagent lot performance which triggers a QC rejection chose to reset the limits using a new set of data points. However, this approach ignores the root cause of the change, which is a change in bias [17].

NRL previously reported a significant change in reagent lots due to the change in a non-biological component of an anti-HCV assay [19]. In collaboration with the manufacturer, the root cause was identified and rectified, with subsequent reagent lots reporting levels of reactivity seen prior to the change. Figure 2 represents results from a single laboratory using affected and unaffected reagent lots, with associated mean±3SD calculated for each reagent lot. It is evident from the figure that the introduction of the second, affected reagent lot would trigger a 1^3s rule. If the laboratory chose to re-set their limits, as is usual practice when addressing Westgard rule failures, they would have accepted this clinically significant change. This is because they would have chosen to ignore the bias introduced by the reagent lot change. Resetting limits in response to a change in reactivity is a dangerous practice that could allow significant issues to go undetected, as there is no reference point to identify unacceptable bias. The use of historical results to establish an assay/QC specific acceptance range overcomes this methodology deficiency [20].

Figure 2:

A Levey-Jennings chart of monitoring the reactivity of a single quality control sample tested in an anti-HCV assay. The lower results associated with a reagent lot number (represented in lilac) due to an identified change in reagent manufacture. The horizontal red lines represent three standard deviations from the mean of the results of each reagent lot.

Most commentators on QC processes offer opinions, mainly based on published guidelines that were developed decades ago, prior to infectious disease testing becoming automated [21], and thus applying processes used in clinical chemistry without any evidence that they were fit for use in infectious disease testing. Other commentators provide theoretical solutions using mathematical modelling and complex equations, again, without providing real-world evidence using actual data [15], 22], 23]. Others, like the four papers reviewed in the study, use localised QC data generated from single laboratories over a short period of time [9], [10], [11], [12, 14]. Here, we use comprehensive data derived from real laboratory testing of QC samples to review assumptions used to apply Sigma metrics to infectious disease testing.

There have been numerous publications on the application of Six Sigma to medical laboratory testing [2], [4], [5], [6, 24], while noting that its application is different in a laboratory setting compared with other industries [23]. Some authors have questioned its value [1], while others have called for harmonisation [8]. Irrespective, the application of Sigma metric to infectious disease serology QC requires specific review as it includes assumptions not generally required for other medical laboratory disciplines, in particular relating to the definition of TEa and bias. Khelil recently published an article identifying some deficiencies of some assumptions used by a particular study [25]. Each of the four recently reported studies estimating Sigma values of infectious disease serology testing [9], [10], [11], [12] used different assumptions and study designs, indicating a lack of standardisation in the application of this concept. This paper reviewed the methods employed by each of four papers and used real, comprehensive QC data to critically assess the different methods.

There are three main components required to estimate Sigma values, being the estimation of imprecision (CV%), bias and TEa. The CV% varies considerably depending on the data set or the method used for calculation (simple SD/mean or pooled CV%). A laboratory using the same reagent lot number over a short period will have a small CV% whereas data taken from a longer period across multiple lots will result in a larger CV%. This has a material impact on the estimation of Sigma values. Recent data obtained from NRL’s EDCNet™ demonstrates the significant range of CV% reported by laboratories using the same QC and assay over the same timeframe. Data reported within laboratories at 1-, 3- and 10-month periods, identified that the CV% range is not an indication of assay performance but due to larger numbers of reagent lots included in the estimation.

It is important to have defined criteria for the calculation of imprecision for the purposes of applying Sigma metrics. Using data collected during a small period of time such as 3 months or less will result in an artificially low CV% and consequently a higher Sigma value. Laboratories routinely testing QC samples, or even kit control, and using their results for the purpose of estimating Sigma values should have access to considerable data over a long period of time. It is suggested that data from a laboratory-defined period that accommodates normal lot-to-lot variation be utilised to ensure the CV% estimates include all normal lot-to-lot variation.

In infectious disease serology, bias is attributable to many sources of inherent variation (such as local conditions and calibration and performance of instruments) yet is mainly due to the change of reagent lots [16], 19], [26], [27], [28], [29]. This bias can range from being small to significant, so it is not appropriate to ignore bias without first undertaking an estimate. If bias is found to be small, its exclusion could be justified in the same manner as estimating uncertainty of measurement [30]. NRL has previously published a method for estimating bias for infectious disease serology using the mean of QC test results observed by the laboratory, divided by the mean of the peer group of laboratories using the same QC and assay [31]. Only one of the four studies included bias in the calculation of Sigma values. As has been demonstrated before, reagent lot changes can introduce significant bias and ignoring this component is not appropriate unless it has already been calculated and found to be insignificant [31].

Using the difference between the mean reactivity of the selected QC and the assay cut-off to calculate the TEa implies that a change in reactivity of the QC sample between these levels is acceptable (i.e. allowable error). Apart from the study by Khelil, the selection of the QC used to determine TEa seems to be of convenience, rather than selecting a QC with reactivity that was appropriate for the purpose of estimating Sigma metrics.

The reactivity of QC samples used for infectious disease testing should be designed to be on the linear part of the dose response curve of the assay, not just a unilateral decision to be close to a cut-off [32]. Determination of this level requires the performance of a serial dilution series of the stock material as demonstrated by Khelil and routinely used to manufacture Optitrol QC samples used in QConnect™ [20]. If the difference between the mean of the QC results and the assay cut-off is used to define the TEa, the reactivity of the QC sample should be selected to represent this allowable change in reactivity, not for convenience. That is, if the laboratory accepts a decrease in reactivity of a QC sample from S/Co 3.0 to 1.0 (the cut-off) i.e. a fall of S/Co of 2.0, then the QC sample must be manufactured to have a mean reactivity of 3.0 S/Co. Selecting a QC sample with a mean reactivity of 9.74 S/Co and accepting a decrease to S/Co 1.0, as reported in one study is not acceptable. It should be noted that this is not a comment on whether S/Co 3.0 is more or less appropriate than 9.74 in itself, but on the impact of adopting Sigma metrics would inadvertently have on acceptable S/Co ratios for infectious disease QC samples irrespective of where the best fit is for a given assay and its dose response curve. Assessment of an appropriate assay S/Co target is not the aim of this paper.

Whereas some of the kit control’s reactivity may be adequate for determining Sigma values, others were demonstrably not acceptable. The approach taken by Khelil is appropriate, as the QC used in that study were designed to have reactivity close to the cut-off and also on the linear part of the dose response curve and more arguably represented the allowable error for the assays investigated. If QC reactivity is to be used to calculate the TEa, the samples used must be designed and manufactured for this purpose, not selected for convenience.

All four studies demonstrated that almost all the assays under investigation had a Sigma value of >6. This is not surprising, in particular for kit controls and QC samples tested on the fully automated Abbott ARCHITECT and Alinity i platforms. The adoption of Westgard Sigma Rules to only use the 1^3S rule on two levels of QC for Six Sigma assays will reduce the number of unnecessary investigations, saving the laboratory significant resources. This finding confirms previous reports identifying the inappropriate use of classical Westgard rules for infectious disease testing [16].

The use of Westgard Sigma rules in infectious disease testing ignores the points raised in recent publications about the differences between serology and clinical chemistry testing, in particular lot to lot variation [16], 17], 28], 29], 33], 34]. This approach will still cause clinically insignificant flags (false rejections) leading to wasted resources, albeit much reduced compared with classical Westgard rules application. The practice of re-setting of acceptance limits by recalculating mean±3 SD, without reference to an acceptable amount of bias, may result in the acceptance of clinically significant failures. An appropriate QC monitoring process for infectious disease testing must determine what amount of bias is acceptable within a test system.

It is important that methods used for the standardisation and control of clinical chemistry assays are not universally applied across other disciplines without consideration of the differences [33]. This is particularly the case for infectious disease testing which experience significant, but normal lot-to-lot variation. The assumptions used to estimate the Sigma value must be carefully considered. The application of Sigma metrics and the use of a single 1^3s rule on Six Sigma assays will decrease the number of false rejections, benefiting laboratories. However, using a single 1^3s rule will still lead to false rejections and/or the necessity to recalculate acceptance limits and does not solve the underlying issue – that QC rules based on Gaussian distribution are not applicable/appropriate for datasets that are not normally distributed, such as those including multiple reagent lot changes.