Moving average quality control: principles, practical application and future perspectives

Inclusion criteria and truncation limits

The inclusion criteria, or perhaps more correctly the exclusion criteria, aim to remove outliers and extremes from the calculation algorithm to reduce variation in the included assay results. Reducing the variation in assay results reduces variation in MA values and application of more stringent control limits, thereby allowing the detection of smaller systematic errors. Theoretically, the inclusion criteria can be based on any variable considered relevant, e.g. the hospital department [5], in/outpatient populations [2], week/weekend day [3], etc. However, most often and generally supported by MA management software packages, these criteria are based on the exclusion of extreme or not-normal assay results. The limits used for this purpose are also referred to as truncation limits [2], [6]; however, this approach has some limitations. For example, the exclusion of too many results from the MA calculations implies that an MA value is less frequently calculated and that MA alarming might be delayed. Also, when truncation limits are used, extreme systematic errors might become undetectable due to the exclusion of all generated systematic errors containing assay results [6]; this is illustrated in the bias detection curves presented in Figure 1B, C and D. To select optimal inclusion criteria, these effects need to be taken into consideration.

Figure 1:

Bias detection curves with and without truncation limits.

Bias detection curves are presented for sodium moving average (MA) procedures. (A) Bias detection performance of 5 MA procedures using mean calculation algorithm with different batch sizes. Sodium MA using mean calculation of the last 25 results is presented without truncation limits, (B) using truncation that only includes results between 100 and 180 mmol/L (C) and only includes sodium results between 120 and 160 mmol/L. (D) Graphs were obtained using the MA Generator application [7].

Calculation algorithm

The next step is to calculate average values. For this, several algorithms can be used, including the mean [2], [3], [4], the median [4], the exponentially weighted moving average (EWMA) and the XbarB [8]. Each calculation algorithm has its own specific features and the use of one or the other can be based on these features. Nowadays, the availability or absence of these algorithms in MA management software packages for laboratories can be another critical factor in selecting a calculation algorithm [3], [9]. From a statistical point of view, the mean calculation is considered suitable for MA with assay results that are normally distributed and have a low variation in results, whereas median calculations are generally more powerful for assays with non-normally distributed assay results and those with more extreme results or outliers [10]. The XbarB algorithm described by Bull et al. [8], and also referred to as Bulls algorithm, was designed to smooth the moving average deviation in order to minimize the effect of outliers and allow application for non-Gaussian populations. By design, the XbarB works with batches, and new MA values are generated only after filling up a batch, with a potentially unnecessary delay in bias detection [8], [9]. Alternatively, by design, the EWMA calculates a new MA value for each newly available assay result and, therefore, represents a more true and continuous “moving” average. For median and mean calculations, in terms of filling a batch or “walking” batches, both designs are available. Alternative measures or algorithms include exponentially adjusted moving mean [11], Bhattacharya calculation [12], moving standard deviation and moving sum of outliers [13], number of positive patient results [14] and the average of delta [15]. However, no clear performance-based guidance is available concerning which specific MA algorithm should best be used.

The second part of the calculation algorithm is the selection of variables used in the calculation algorithm. For the mean, median and XbarB, a batch size needs to be selected, whereas for the EWMA, a weighting factor between 0 and 1 is required. Based on the results of Bull et al. [8], an XbarB with a batch size of 20 results is generally used as a standard for the MA procedure for erythrocyte indices and other hemocytometer analytes. In general, with the use of large batch sizes (or a low weighing factor), less variation in MA values is observed and smaller systematic errors can be detected. The trade-off is that the detection of larger systematic errors requires more assay results and can be delayed [5], [6]. A calculation feature that is considered necessary to meaningfully interpret the obtained average values in an MA control chart is that all MA values are computed using the same number of included results and are not, for example, based on a daily average; this avoids MA variation related to the use of different calculation procedures [16].

Control limits

The control limits are assigned MA values that, when exceeded, indicate an out-of-control status of the assay. These limits are generally based on population-based SD rules [4], [17], the reference change value [2] or other related statistical approaches [18]. When using these approaches, the upper and lower control limits are related to each other and have the same distance from the MA mean. More recently, control limits were set by the primary requirement of a manageable or acceptable number of false MA alarms [4], [6]. For this purpose, after running the MA procedure on a training set obtained from the laboratory, the observed minimum and maximum MA values were used as control limits [6]. Using this approach, non-symmetrical and non-SD-based control limits are obtained to meet the requirement of no false MA alarms while using the most stringent control limits [6].

Power function analysis

Others have used power function analysis to study MA characteristics and to obtain optimal MA settings for routine practice [17], [19], [21]. Power function analysis was first described and used to thoroughly investigate the performance of statistical QC rules for internal QC results [22], [23]. In line with this application, power function analysis was then used to assess the performance of MA QC procedures [19]. Although highly successful for the analysis of statistical QC performance, this approach has some limitations when used to investigate MA QC. The first limitation is that the analysis is based on computer-generated data with a Gaussian distribution [19]. Although this seems to work when describing internal QC results, this does not apply for most measurands in medical laboratories that often show a severely skewed distribution [2], [6]. Also, the samples and, thereby, the distribution of assay results are generally not randomly presented to the laboratory. They are usually determined by logistics and patient care settings, such as clinical rounds (intensive care, nephrology, pediatrics, etc.), week and weekend days, in- and outpatient programs, etc.; however, all of these have a significant effect on the normal MA pattern or “MA fingerprint” of a laboratory [2], [6]. Another limitation of power function analysis is that the true effect of truncation limits on potentially delaying error detection is not reflected [5]. When truncation limits are applied, large systematic errors can result in the exclusion of many results from the calculation algorithm, thereby significantly delaying error detection [5], [6]. Last but not least, the onset of error in the MA values is not reflected in power function analysis. A larger batch size always results in better error detection [5], [19]; although this is basically true, at the same time a larger batch size always delays the detection of (especially) larger errors [5], [6]. The latter becomes more relevant when MA QC is used for continuous and real-time QC purposes. An advantage of this approach is that nomograms are available for implementation and represent a more universal and less laboratory-specific optimization method [19].

Simulation methods using results needed for systematic error detection

More recent MA optimization approaches use advanced simulation strategies in which systematic error is introduced, and the number of assay results needed for systematic error detection is used as read-out or output [2], [6], [20]. These approaches aim to simulate the true MA performance, as observed in daily practice. These simulation results can be presented in either the mean number of results needed for systematic error detection as is reflected in the ANPed parameter [2], [20], or the median number of results needed to detect a systematic error [6]. When the simulations are performed for various systematic errors, the results can be plotted in specifically designed bias detection curve graphs in which, for multiple systematic error, the median number of assay results needed for error detection is presented. When multiple MA procedures are plotted, their bias detection performance can be compared (Figure 1A).

Additional information is needed to be able to properly interpret studies using these approaches. Because the results are based on a reference data set, this set has to be representative for a true, in-control, run. Therefore, the method of interest should have been in-control during the reflected data-sampling period. Second, the order of the obtained results should be maintained during the simulations; this is necessary to be representative of the MA fingerprint as would be observed reflecting, for example, week and weekend profiles, in- and outpatients logistics, intensive care, pediatric blood sampling rounds, etc. [2], [6]. Alternatives that apply random sampling from the data set or computer-generated assay results can only estimate a “normal” MA pattern. They do not take into account the true logistic variables that determine the true MA fingerprint [20]. Furthermore, the use of either a mean or median value to interpret the simulation results has different meanings. The median has statistical power in the sense that in 50% of the simulations the systematic error is detected in less than the represented number, and in 50% of the simulations more results were required. For this reason, the median value has an understandable meaning and can be used as a risk parameter [6], whereas the mean value does not provide this kind of information because most of the obtained simulation results are not normally distributed (Figure 2 ). Alternatively, also the 90%, 95% or the minimum/maximum number of results needed for bias detection can be obtained that allow a more risk-based estimate of the MA QC performance. Therefore, this information (as presented in MA validation charts) can be used to design more risk-based MA QC procedures (Figure 2) [6].

Figure 2:

Moving average (MA) validation chart.

A MA validation chart is presented for a sodium MA procedure without truncation limits, calculating the mean of the last 25 results and using 137.5 and 143.6 mmol/L as lower and upper control limit, respectively. Bars represent median number of results, and error bars represent minimum and maximum number of results needed for error detection as observed in the simulation analysis. The graph was obtained using the MA Generator application [7].

Figure 3:

Moving average (MA) graph presented in accuracy plot.

The MA graph presents a sodium MA procedure. The sodium MA procedure is the same as that used for the MA validation chart in Figure 2. The green area shows the MA values between the control limits determined according to van Rossum et al. [5], [6].

A drawback of these advanced simulation strategies is that the outcomes are probably rather laboratory specific. Furthermore, advanced statistical modeling and/or software is required that is generally not available in medical laboratories, although the web-based application MA Generator has become available online [7]. This application allows laboratories to use the MA optimization approach described by van Rossum et al. [6], using laboratory-specific data sets.

Optimal MA requirements

The last point of discussion concerning MA optimization methods is what parameter to use to guide the MA optimization process. Generally, MA optimization focuses on the detection of a specified systematic error, often the TEa [2], [4], [18]. The risk of using this approach is that by focusing on this specific error, the detection of systematic errors larger than the TEa is potentially compromised, e.g. by the use of truncation limits (as shown in Figure 1). Conceptually, TEa or related performance criteria reflect a threshold that determines a clinically relevant error, and therefore, all larger errors are of at least equal importance [24], [25]. In this respect, MA QC differs from statistical QC in the sense that larger errors will always be detected with a larger probability; for MA QC, this depends on the use of truncation limits in an MA procedure and cannot be assumed. This is illustrated in Figure 1. Here the use of truncation limits in Figure 1C and D compromises the detection of larger systematic errors. The MA presented in Figure 1D is unable to detect a 20% systematic error, whereas a systematic error of 10% is rapidly detected. An alternative approach is to use the bias detection curve itself as an optimization parameter. This optimization approach is less clearly guided but allows to take into account the overall systematic error detection performance of MA procedures [6].

Long-term assay stability

One application of MA QC is the assurance of long-term assay stability. One reason for this application is the issue of non-commutability that can be associated with internal and external QC samples [30], [31], [32], [33]. Examples of using MA approaches as a quality assurance tool include the stability of FT4, TSH, calcium and phosphate using a 50th percentile algorithm. Using a similar approach, clinically relevant shifts in sodium were observed that coincided with reagent lot shifts [34]. Others detected a shift in bone-specific alkaline phosphatase using an MA, which was not visible using external and internal QC [35].

More recently, a more or less pragmatic MA approach was introduced. It comprises two web-based and freely available applications called the Percentiler and Flagger [36], [37]. They are part of a larger empowerment project developed to enhance the communication on analytical performance of commercial test systems between laboratories and in vitro diagnostic manufacturers [26], [27], [31]. The Percentiler database comprises instrument-specific daily medians calculated from outpatient results and sent by e-mail from all over the world. Data are peer grouped by platform/assay. Via a user interface, participants have access to the graphical presentation of the course of their moving medians in comparison with the peer group one. This allows inference of the mid- to long-term stability of analytical performance at the individual and peer group level, and/or occurrence of shifts/drifts [26], [27]. The Flagger application is designed to monitor the stability of the percentage of results flagged when they exceed the cutoff points used by individual laboratories. From combining the Percentiler and Flagger observations, users can infer how the change in flagging rate in their laboratory is related to the analytical variation [38].

Real-time/continuous QC

The second major application of MA QC is continuous (preferably real-time) QC. For the long-term assay stability application of MA QC, generally one MA QC is generated on the longer term, such as on a daily [27], weekly [34] or monthly basis [32]. However, MA QC values can be calculated for every new, or a couple of newly generated assay results, resulting in a more continuous QC process. When the MA QC can detect errors within one or a couple of assay results, e.g. as shown for large systematic errors for sodium MA (Figure 2), it could be claimed that, for these errors, MA QC supports real-time QC. However, by design, erroneous samples have to be generated before MA can detect such an error.

MA QC to support statistical QC

When considering continuous laboratory production processes, bracketed internal QC (iQC) is applied for analytical quality assurance [39]. In current practice with short turn-around times, continuous analysis and the release of multiple diagnostic test results, many laboratory results will be released before a confirmatory QC is performed. When a continuous production process is analytically controlled using scheduled iQC, there is a risk of reporting erroneous results by either rapid onset of (large) error in between the scheduled QC [3] or temporary assay failure in between the scheduled iQC, as demonstrated for a sodium MA alarm case [9].

Furthermore, by design, statistical QC is limited in its ability to detect clinically relevant errors for low sigma processes [39], [40]. Interestingly, these low sigma processes (characterized by a low biological variation/analytical variation ratio) generally have the characteristics for optimal MA performance [19]. Furthermore, when risk-based statistical QC schedules are used for high-volume low sigma (<3) assays, statistical QC becomes almost unmanageable [39]. For these reasons, MA QC can further support and extend the analytical quality assurance of statistical QC [39]. However, to my knowledge, as no study has designed and validated integrated statistical QC and MA QC strategies, there is no clear guidance regarding how to design risk-based QC plans incorporating MA QC. Furthermore, it should be noted that, for several reasons, it is unlikely that MA can replace the measurement of QC materials (internal and/or external) [33]. For example, analytical failure detected by MA should be confirmed, e.g. by QC samples, and MA error detection performance for many measurands is limited. Also, after performing potential error-introducing procedures (such as large maintenance, new reagent lots, calibrations, etc.), QC measurement is required before analyzing samples in order to confirm proper assay performance, which cannot be done by MA QC [33].