Assessing the measurement uncertainty of qualitative analysis in the clinical laboratory

Introduction

Measurement uncertainty is one of the most powerful tools for expressing the dispersion of measurement procedures in clinical laboratories [1], [2], [3]. According to the “Guide to the expression of uncertainty in measurement”, measurement uncertainty is defined as a “parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand”. Considering this definition, measurement uncertainty is expressed as the confidence interval in which the unknowable true value is believed to lie [4]. To estimate measurement uncertainty, all potential sources of uncertainty should be elucidated and the contribution of each uncertainty budget should be estimated by proper models [5]. The estimation of measurement uncertainty can provide a quantitative overview of a test result and give opportunities for quality improvement [6]. Recently, a number of reports have described examples of estimating the measurement uncertainty in various quantitative analyses [7], [8], [9], [10], [11], and the evaluation and expression of the measurement uncertainty is gaining importance in clinical laboratories [5], [10].

Unlike quantitative analysis, qualitative analysis is reported by a dichotomous result, such as “positive or negative” or “pass or fail”. This qualitative result renders it difficult to express measurement uncertainty in a similar manner to quantitative analysis. As the typical measurement uncertainty is generally probabilistic in nature, it is reasonable to express the measurement uncertainty in qualitative analysis as the probability of making a wrong decision. Previously, several articles have discussed the application of the measurement uncertainty to qualitative analysis, with numerous methods based on mathematical approaches having been introduced and successfully applied [12], [13], [14], [15], [16]. Although approximately two decades have passed since this issue was reviewed, the practical application of the measurement uncertainty in qualitative analysis in clinical laboratories remains unaddressed. Even in the cases of various international guidelines, no mention is made of the measurement uncertainty in qualitative analysis [2], [3].

Thus, we herein review three applicable approaches for quantifying the measurement uncertainty in qualitative assays from a clinical laboratorian point of view. More specifically, Bayes’ theorem, the normal distribution method, and the information theoretic approach are examined, which are based on probability calculations and can be reported as probability, likelihood ratios, or entropy [12], [13], [14], [15], [16]. In addition to the mathematical descriptions, we also show applicable examples that can be readily employed in a real clinical laboratory.

Measurement uncertainty using Bayes’ theorem

Bayes’ theorem is perhaps the oldest known method for determining conditional probability [14], [17], [18]. Using this method, we can estimate the probability of an event based on a prior probability of conditions that would be associated with the event [12], [19]. For example, one can calculate the probability of the hepatitis B virus (HBV) active carrier status when the hepatitis B surface antigen (HBsAg) test is positive. From a clinical aspect, this method is one of the best candidates for the application of measurement uncertainty to qualitative assays.

Bayes’ theorem can be expressed as outlined in the following equation:

where P(A) is the probability of an event A (presence of the measurand, for example, HBsAg for an HBV active carrier), and P(Pos) is the probability of a positive test result (in this case, HBsAg-positive result). In addition, P(A|Pos) is the conditional probability of event A given that the test result is true (the probability of an actual HBV active carrier whose HBsAg test result is positive). In the above equation, a prefix “not” (e.g. not A) denotes “absence of event”, and therefore, P(Pos|A) and P(Pos|not A) are the conditional probabilities of a positive result given that event A has or has not occurred already (the conditional probabilities of HBsAg-positive results in an HBV active carrier or in an HBV-free person).

In common clinical practice, the probability P(A) can be replaced by the probability of a particular medical condition (disease prevalence), and P(A)+P(not A)=1, because P(A) and P(not A) are mutually exclusive (i.e. the patient is an HBV active carrier or not). P(Pos|A) is equal to the proportion of true-positive test results in patients with event A, and this represents the sensitivity of the assay. Similarly, P(Pos|not A) is the proportion of false-positive test results in patients free from event A, which represents 1−specificity. Bayes’ theorem can therefore be revised as follows:

Based on the aforementioned points, the measurement uncertainty of qualitative assay can be easily quantified because it is equal to the positive predictive value using the analytical performance of the assay and the disease prevalence [15]. These data can be collected with previously published reports for technical specifications of assays. Conversely, it should be noted that a negative predictable value (P(not A|Neg)) is the measurement uncertainty of the opposite case (i.e. the absence of event A in a given negative test result).

The application of Bayes’ theorem can be illustrated by the following example. Assume a rapid test for the presence of HBsAg using test T_A. This test showed a 92% sensitivity and a 98% specificity in the performance evaluation report. To calculate the conditional probability, which is the likelihood of the patient being an actual HBV active carrier (P(A)) given that T_A is a positive result (P(Pos)), the country prevalence of the HBV active carrier is assumed to be 0.001. From equation 2, the measurement uncertainty is P(A|Pos)=(0.92×0.001)÷[(0.92× 0.001)+(1−0.98)×(1−0.001)]=4.4%. However, this small posterior probability is the result of screening the entire population of a country. As such, we can adjust the prevalence to a hospital-specific prevalence for a more rigorous uncertainty evaluation, and this is significantly larger than that of a country’s population, as the HBsAg test is employed only when symptoms are suspected. If the prevalence of patients referred to the test is known to be 0.01, the adjusted measurement uncertainty is P(A_adjusted|Pos)=31.7%. This posterior probability represents the likelihood of the patient being an actual HBV active carrier who visited hospital (P(A_adjusted)) given the positive test result. When the test result is negative in the HBV-free person, the negative predictive value is (P(not A_adjusted| Neg)=[0.98×(1−0.01)]÷{[0.98×(1−0.01)]+(1−0.92)× 0.01}≈100%. It is important to note that the posterior probability varies depending on the sensitivity, specificity, and prevalence and the change of posterior probability is visualized in Figure 1.

Figure 1:

Change of posterior probability according to the sensitivity and specificity.

The disease prevalence is set to 0.001 in (A) and 0.01 in (B).

The measurement uncertainty of qualitative assays can also be expressed using a likelihood ratio, which is the ratio between two conditional probabilities [12], [14]. The two likelihood ratios can be expressed as follows:

Thus, a positive likelihood ratio represents the probability that a given positive test result would be expected in a person with the disease compared to the probability that a positive result would be expected in a disease-free person. It should be noted that the likelihood ratio has been employed in forensic science to evaluate identification certainty [12], [17], [20]. Due to nature of forensics, in which an assay result can be used as evidence in a court, it is recommendable to express the expert evaluation of likelihood as a form of word. According to the magnitude of the likelihood ratio, the degree of support can be expressed with verbal equivalents (Table 1) [17].

From equation 3, returning to the previous example (T_A), the positive likelihood ratio is 46, which would be considered a “moderate support” for the diagnosis of an HBV active carrier. In the same way (equation 4), the negative likelihood ratio is about 12, which is considered “moderate support” for the absence of HBsAg.

The key advantage of Bayes’ theorem is the easiness to quantify the uncertainty when prior probabilities (i.e. sensitivity, specificity, and prevalence) are available [12]. In addition, the uncertainty in a specific medical environment can be estimated with adjusting prevalence. When an epidemic occurs in a specific area, we can re-estimate the positive predictive value using the epidemic data or hospital-specific prevalence. However, Bayes’ theorem also has several drawbacks [14]. Firstly, the nomenclature employed is complex and not familiar to a clinical laboratory. In addition, a priori probabilities must be available to quantify the conditional probability, and the measurement uncertainty cannot be calculated when the assay specificity is 100%. To avoid such issues and obtain adequate estimates, a high number of tests should be carried out.

Most importantly, the key weakness of Bayes’ theorem is that this method fails to take into account the parameters associated with the result of each measurement, and the measurement uncertainty of each sample is always the same whether the amount of the measurand in each sample is small or large [14]. This means that the uncertainty calculated using this method is the measurement uncertainty for the measurement system, and not for the individual measurement. This would be particularly problematic when the measurand concentration is close to the limit of detection, because the probability of determining the correct result would be reduced in the sample of this concentration interval.

Measurement uncertainty using the normal distribution approach

A number of automated assays use quantitative signals to conclude qualitative results. These quantitative signals, such as the sample to cut-off ratio (S/Co) and the cut-off index (COI), are compared to cut-off values set by the assay manufacturers or by international consensus guidelines [21], and the test results are reported as either “positive” or “negative”. Although these assays are categorized as qualitative analysis, the instrumental response is a continuous value and has quantitative properties [22], [23]. We can therefore assume that the instrumental signal follows the normal distribution and the measurement uncertainty of the instrumental signal can be estimated in a quantitative analysis manner. Previously, a number of instrumental responses, such as the S/Co of HBsAg and the COI of the HIV test, were hard to calculate the measurement uncertainty because of the absence of a reference method or reference materials. However, recent methods for evaluation of the measurement uncertainty using external quality assessment and proficiency test (EQA/PT) data have been introduced [9], [10], [24], and it is now possible to calculate the measurement uncertainty for almost all signals obtained in a clinical laboratory. With this information, the true-positive rate (or true-negative rate) and the false-negative rate (or false-positive rate) can be calculated using a normal distribution curve (Figure 2). The normal distribution curve of patient result is

Figure 2:

Relationship between the instrumental signal value and the cut-off value.

The true positive rate and the false negative rate are computed through the normal distribution curve. LR(+), positive likelihood ratio; LR(−), negative likelihood ratio. (A) r_result<r_cut-off, (B) r_result=r_cut-off, and (C) r_result>r_cut-off.

where μ is the patient result (instrumental response), and σ is the measurement uncertainty of instrumental response. From equation 5, the true-positive and false-negative rates are

where r_cut-off is the cut-off value of qualitative assay using continuous instrumental response. Then, the likelihood ratio equal to the measurement uncertainty of qualitative analysis can be quantified as follows [20]:

As indicated in Figure 2, there are three cases between the measured result (r_result) and the cut-off value (r_cut-off), i.e. (A) r_result<r_cut-off, (B) r_result=r_cut-off, and (C) r_result>r_cut-off. When the measured result is larger than the cut-off, the qualitative result is “positive”, and we can calculate the true-positive and false-negative rates using the measurement uncertainty of the instrumental signal. The positive likelihood ratio can then be expressed by simply dividing the true-positive rate by the false-negative rate. Conversely, the negative likelihood ratio for r_result<r_cut-off can be quantified by dividing the true-negative rate by the false-positive rate. In the case of r_result=r_cut-off, the true-positive and false-negative rates are equal to 0.5 (likelihood ratio=1), and therefore, we cannot conclude the binary test result.

A second example illustrates the expression of measurement uncertainty involving a quantitative property. The instrument T_B is assumed to be an automated HBsAg assay to screen for the HBV infection (Table 2). The result of T_B is reported as an S/Co value, and is assigned as “positive” when the S/Co value is >1. Using the calibrator information, EQA data, and internal quality control data, the expanded measurement uncertainty of the S/Co value is quantified as ±0.08 S/Co. When the measured result of instrument T_B is 1.15 S/Co in the HBV-suspected patient, the reported S/Co value can be expressed as 1.15±0.08 S/Co (k=2, approximately 95% coverage probability). In this case, the true-positive rate is about 99.99%, and the false-negative rate is about 0.01%. The calculated likelihood ratio of a positive result is 11,309, and the measurement uncertainty can be reported in terms of a “very strong support” for the presence of HBsAg. Figure 3 illustrates the true-positive rate and the positive likelihood ratio according to the instrumental signal.

Figure 3:

The positive likelihood ratio increases rapidly with increasing value of the instrumental response.

Variation in the true positive rate (A) and positive likelihood ratio (B) based on the instrumental response in example 2.

In contrast to the Bayes’ theorem approach that estimates the uncertainty based on the overall behavior of previous measurements, the normal distribution approach considers individual measurements for calculation of the uncertainty [14]. However, this method also has several disadvantages. For example, a dichotomous result cannot be concluded when the measured signal is equal to the cut-off value, as the possibility of a true-positive result is equal to that of a false-negative result. At this point, the normal distribution approach does not provide any further information about the test result.

To assign the measurement uncertainty of the instrumental signal, the interlaboratory bias should be calculated using information from EQA/PTs [9], [10], [24]. However, due to the nature of the qualitative assay, the majority of PT/EQA programs collect only binary results rather than instrumental responses such as S/Co and COI. This method could not be applicable to qualitative assays using continuous measurement values when sufficient information is unavailable.

Measurement uncertainty using the information theoretic approach

The information theory has also been introduced to clinical laboratories for measuring the uncertainty of binary outcome [16]. The entropy, one of the key concepts of information theory, quantifies uncertainty of binary result using the probability of possible outcomes [25], [26]. If p is the probability of a binary outcome for a specific assay, the entropy, S, is

The logarithm is taken with base 2 for binary test result, yielding the entropy in units of “bits”. The point of minimum entropy occurs as the probability approaches 0 or 1. As the probability approaches 0.5, the entropy approaches 1.

For the previous example 2 in the normal distribution approach, the true-positive and -negative rates correspond to the probability to calculate the entropy. From the equations 6, 7, and 10, we can calculate the entropy using the instrumental signal and cut-off value. When the relationship between the instrumental response and the entropy is plotted, the shape of the curve for the entropy shows an inverted U shape; decreased entropy for low and high instrumental signals, and increased entropy near cut-off values (Figure 4).

Figure 4:

The binary entropy function graph based on the instrumental signal.

Because the information theoretic approach uses the continuous instrumental response to calculate the entropy, this approach shares the same advantages and disadvantages as the normal distribution approach. However, the complex terminology with unfamiliar units (bits) would be one of the most potential hurdle to implement the information theory in the community of laboratory medicine [25].

Conclusions

The qualitative tests differ from the quantitative tests principally because there are no numerical results but dichotomous results. For many years, most of clinical laboratorians have only emphasized the final binary result, and focused on the concepts of sensitivity, specificity, and receiver operating characteristic curve [16], [27]. Although various mathematical methods have been applied to the measurement uncertainty of qualitative analysis [14], [15], [16], [22], the concept of measurement uncertainty is not commonly used in qualitative measurement that laboratory tests provide.

The measurement uncertainty can be described as a quantifiable parameter associated with the dispersion of measurements. As such, three parameters, known as the conditional probability, the likelihood ratio, and the entropy, permit us to express the measurement uncertainties of qualitative assays using statistical models. Although the mathematical expression of the measurement uncertainty differs slightly between qualitative and quantitative analyses, the significance of the measurement uncertainty does not change. It is a “non-negative parameter characterizing the dispersion of the quantity values (or qualitative result) being attributed to a measurand, based on the information used” [3]. As in the case of quantitative methods, we can assign the probability that a reported qualitative result is actually true based on a priori information or on the measurement uncertainty of instrumental responses.

However, these methods exhibit a number of major limitations [14], [20], [25]. For example, in the case of the Bayes’ theorem method, following the calculation of the measurement uncertainty, the uncertainty is fixed and cannot consider any individual measurements. In addition, the normal distribution method and information theoretic approach cannot be applied when the measurement system does not use the quantitative signal for determination of the binary result. However, these approaches still exist as statistical methods for quantifying the measurement uncertainty of qualitative analysis in the current state of the art, with a number of adequate examples being previously reported [15], [16], [28], [29].

Regardless of method, all measurement procedures are affected by many potential sources of variation, and the results cannot be exactly determined [8]. The concept of measurement uncertainty reflects incomplete knowledge of the test result, and provides the statistical dispersion of the values attributed to the final result. With the measurement uncertainty, we can provide the probabilistic information about the binary results in the qualitative assays. In this paper, we have reviewed various approaches of how the measurement uncertainty can be applied to the qualitative test, and believe that it will contribute to help us understand the more information that the qualitative tests provide.