VeRUS: verification of reference intervals based on the uncertainty of sampling

Introduction

Clinical decisions, including diagnosis and treatment, are frequently guided by numerical laboratory test results [1], 2]. A commonly used tool for interpreting these results is the reference interval (RI) [3], 4], which usually describes the central 95 % of values observed in a reference population consisting of apparently healthy subjects [5]. Inclusion of RIs in laboratory reports is mandated to facilitate the interpretation of biomarker measurements [6]. Currently, most laboratories provide RIs in their reports that were established externally, such as RIs taken from manufacturers’ package inserts, expert panel recommendations, guidelines, or publications [4], 7]. To ensure that the RIs used in laboratory reports describe the local population adequately, regulatory bodies require laboratories to regularly verify their RIs. However, the exact procedures to comply with these standards are not defined [6].

The Clinical Laboratory Standards Institute (CLSI) EP28-A3c guideline recommends using a binomial test for the verification of RIs [5]. Briefly, the CLSI binomial test evaluates a RI by determining the proportion of reference samples falling outside its limits. The RI is verified if this proportion is ≤10 %. If this proportion is between 10 and 20 % it is recommended to repeat the test once [5]. This binomial test can be conducted with 20–120 reference samples [5]. However, due to the substantial effort involved in sample collection, a number at the lower end of this range is usually chosen [7]. Notwithstanding its utility, the binomial test has several known limitations, primarily the practical challenges associated with collecting reference samples [8]. The collection of samples for the purpose of verifying pediatric RIs is a particularly challenging endeavor, primarily due to the ethical considerations involved [9], 10]. These challenges are amplified when verifying RIs partitioned by categories such as age and sex, which require separate sample cohorts for each category. Furthermore, and clinically important, the test is inherently unable to detect too wide RIs [5], even though too wide RIs expose patients to the risk of inadequate clinical decisions.

Alternatively, several studies suggest that RIs can be verified by comparing them to RIs estimated using indirect methods, which infer the reference distribution a posteriori from routine data, employing a variety of different approaches. These approaches include the subjective comparison of numeric reference limits [11], 12], distribution medians, or flagging rates [4], 8]. Another approach involves assessing whether the CIs of indirectly established RIs overlap with those of the candidate RI or encompass its limits [13], 14]. Since indirect methods typically use large datasets, these indirectly estimated RIs often have comparatively narrow CIs. These narrow CIs may lead to rejection of candidate RIs even when differences lack clinical significance [15]. Therefore, RI verification should be based on more objective acceptance criteria that are independent of the size of the input data set.

One such approach utilizes adapted equivalence limits (ELs) as acceptance criterion, which estimate the permissible analytical standard deviations at each reference limit. These ELs are calculated with a simple algorithm that necessitates several assumptions, including the assumption that all biomarkers are lognormally distributed [16]. This approach is integrated into multiple applications [17], 18]. However, to our knowledge, there exists no systematic comparison of the characteristics of the binomial test and ELs, which would allow an objective assessment of each method.

Here, we propose a novel approach for the verification of RIs based on indirect methods: “Verification of Reference Intervals based on the Uncertainty of Sampling”, short VeRUS. VeRUS combines the practicality of indirect methods with acceptance criteria based on the uncertainty associated with the nonparametric direct method of establishing RIs as recommended by the CLSI EP28-A3c guideline [5]. We implemented VeRUS as part of the open-source R-package refineR version 2.0.0 available on CRAN (CRAN. R-project .org/package=refineR). Guidance on how to apply the VeRUS algorithm can be found in the ‘verification’ vignette of this R-package. Additionally, we present the first systematic comparison of these RI verification methods, namely the binomial test recommended by the CLSI EP28-A3c guideline, ELs, and VeRUS, using simulated datasets from RIbench [19] to evaluate strengths and weaknesses of each approach.

Materials and methods

Description of VeRUS

The fundamental concept of VeRUS is the comparison of the candidate RI with an RI estimated from local routine data (local RI) by assessing the numerical differences between the individual reference limits. The criteria when the differences are considered acceptable are derived from the statistical sampling uncertainty of the nonparametric direct approach for establishing RIs. A schematic representation of the VeRUS methodology is shown in Figure 1.

Figure 1:

Schematic overview of VeRUS for the verification of a candidate reference interval (RI). (1) The local RI is derived after modelling the non-pathological fraction of routine data with mean (μ₁), standard deviation (σ₁), power (λ₁), and shift (δ₁) parameters. (2) Assuming similar distributional shapes (λ₁=λ₂ and δ₁=δ₂), mean (μ₂) and standard deviation (σ₂) of the reference distribution underlying the candidate RI are inferred from its limits. (3) Uncertainty margins (UMs) for both RIs are calculated based on the uncertainty of the nonparametric direct method (n=120). (4) Verification requires overlap of all corresponding UMs.

The first step of VeRUS is modeling the distribution of the local reference population from readily available routine data, i.e., test results obtained during patient care, using the refineR algorithm [20]. With this model, the local RI is estimated (Step 1). As VeRUS relies on accurate RI estimation, assessment of the routine data and critical evaluation of the resulting refineR model is strongly recommended [13], 20]. Subsequently, the underlying distribution of the candidate RI’s reference population is modeled. If the reference values are available, this may be done as described in [21]. In the more common case where no distributional information is available, VeRUS operates on the assumption that if two populations are comparable enough to consider RI transfer (as per CLSI EP28-A3c guidelines regarding, e.g., age, sex, geographic location [5], 8]), their underlying distribution shape is likely to also be comparable. This is implemented by transferring the shape parameters, power (λ) and shift (δ), from the local refineR model to characterize the candidate distribution. The location (μ) and scale (σ) parameters of the candidate distribution are then estimated based on the candidate’s reported upper and lower reference limits, as they are directly susceptible to inter-laboratory variation such as imprecision or bias (Step 2). With these estimated distribution parameters uncertainty margins (UMs) are calculated for each reference limit of both the local RI and the candidate RI (Step 3). Specifically, the UMs are obtained by first approximating the confidence intervals (CIs) for the percentiles defining these RIs (usually the 2.5th and 97.5th percentiles) of the standard normal distribution applying a formula published by Serfling [22]. To obtain the UMs, these CI approximations are then transformed to the appropriate scale, using the distribution parameters determined in steps 1 and 2. The resulting UMs can be interpreted as the minimal width of the CIs for reference limits, as sources of uncertainty other than those caused by random sampling, such as measurement imprecision or bias, are not taken into account [15], 22]. The candidate RI is verified if the UMs of each corresponding reference limit of the candidate RI and the local RI overlap (Step 4). A more detailed description of each step and an evaluation of the Serfling formula for approximating the CI of percentiles (Supplementary Figure 1) is provided in the Supplementary Material.

By default, VeRUS requires a two-sided candidate RI, and a set of routine data. However, some adjustments enable the comparison of one-sided RIs, two numerical RIs and the direct comparison of two indirectly estimated RIs. These adjustments and assumptions are described in the Supplementary Material.

Establishing RIs for heavily right-skewed biomarkers with limited sample sizes (e.g., 120) results in extensively large CIs of the upper reference limit [15], while the lower CI can converge towards a width of zero. This effect is a direct consequence of the skewed shape of the biomarker distribution, which is also reflected in the width of UMs. Extensively narrow UMs at the lower limit may cause acceptable candidate RIs to be rejected. To address this issue, we implemented a correction in the form of an artificial shift applied to the inverse Box–Cox transformation when calculating UMs. This asymmetry correction is only applied if the upper UM width is greater than 75 % of the RI width, i.e., in the case of extremely skewed distributions. This correction broadens the lower UM, while the upper UM width is limited to a maximum of 75 % of the RI width.

Results

Comparing the extent of ELs or UMs around reference limits shows that ELs and the UMs of VeRUS at default settings are essentially identical for normally distributed biomarkers (Table 1). For skewed distributions, ELs aligned more closely with VeRUS UMs computed with n=1,000 at the upper limit, whereas at the lower limit ELs were closer to VeRUS UMs at default settings. Overall, VeRUS at default settings was the most permissible method, followed by ELs, and VeRUS with n=1,000 resulted in the smallest margins around the reference limits.

In our systematic assessment of the three verification methods, we compared the fraction of test cases verified by each approach for a wide range of plausible, to-be-verified candidate RIs using graphical representations (Figures 2 and 3; Supplementary Figures 2–7). On the universal percentile scale, the CLSI binomial test results in the same pattern across all assessed biomarkers (Figures 2 and 3; Supplementary Figures 4, 5).

Figure 2:

Comparison of RI verification methods: CLSI binomial test, equivalence limits (EL), and VeRUS. Each point represents a candidate RI consisting of a lower (Y-axis) and upper (X-axis) limit. Axes are scaled linearly to the percentile values of the simulated distribution. Colors indicate the probability of RI verification (binomial test) or the fraction of verified RIs estimated from simulated test sets (ELs, VeRUS). Red lines mark the ‘true’ reference limits.

Figure 3:

Comparison of RI verification methods: CLSI binomial test, equivalence limits (EL), and VeRUS. Each point represents a candidate RI consisting of a lower (Y-axis) and upper (X-axis) limit. Axes are scaled linearly to the percentile values of the simulated distribution. Colors indicate the probability of RI verification (binomial test) or the fraction of verified RIs estimated from simulated test sets (ELs, VeRUS). Red lines mark the ‘true’ reference limits.

Similar to the CLSI binomial test, the plots generated with VeRUS are comparable for all biomarkers (Figures 2 and 3; Supplementary Figures 2–7). Slightly different shapes of the acceptance areas can be observed at TSH and IgE, biomarkers categorized as heavily skewed (Figure 3; Supplementary Figures 3, 5, 7). ELs were considerably less consistent in verifying candidate RIs of different biomarkers than the other two methods. This is evident as the highest degree of variation in the sizes of acceptance regions, ranging from the smallest for TSH to the largest for Ca, is observed for ELs. The transition from highest to lowest probability of verifying the candidate RI was most gradual for the binomial test, whereas it was relatively sharp for ELs and VeRUS. The most gradual transitions for ELs and VeRUS occurred with skewed biomarkers, e.g., GGT, TSH, IgE.

Verification rates for candidate RIs representing ‘too narrow’ (central 80 %), “correct/true” (central 95 %), and ‘too wide’ (central 99.8 %) intervals are presented in Table 2. All three methods verified the ‘correct/true’ candidate RIs consistently at high rates. The CLSI binomial test is designed to accept RIs which encompass 95 % of a distribution (“correct/true” RIs) with 99.2 % probability [5], while ELs and VeRUS depend on the accuracy of parameter estimation with refineR. Despite this dependency, ELs verified true RIs in 85.4–100 % of cases (mean 94.3 %, SD 6.0 %), and VeRUS achieved verification rates between 96.2 and 100 % (mean 98.2 %, SD 1.2 %).

The CLSI binomial test incorrectly verifies the 80 % “too narrow” candidate RI in 29.3 % of cases. Using IgE as an example, this indicates a 29.3 % probability of accepting a candidate RI of [5.2 × 10³, 80.5 × 10³ IU/L] when the true RI is [1.5 × 10³, 114.0 × 10³ IU/L]. ELs verify these “too narrow” candidate RIs generally at lower rates between 0.0 and 99.7 % of cases (mean 21.7 %, SD 40.9 %). However, ELs hardly rejected the “too narrow” intervals for Ca and Hb with verification rates of 99.7 and 87.5 %, respectively. For the other biomarkers, the “too narrow” RI was verified between 0.00 and 0.4 %. In comparison, VeRUS verified the “too narrow” candidate RIs more consistently at low rates between 3.1 and 18.4 % (mean 7.2 %; SD 4.7 %).

The 99.8 %, “too wide” RI was falsely verified by the CLSI binomial test with a probability of 100 %. Unlike the binomial test, ELs and VeRUS are designed to reject too wide RIs. The successful rejection of candidate RIs with low lower limits and high upper limits can be observed in most biomarkers (Figures 2 and 3). Indeed, the “too wide” candidate RIs were verified at lower rates by ELs between 0.0 and 94.8 % (mean 10.8, SD 31.5 %), and by VeRUS between 0.0 and 6.6 % (mean 4.2, SD 2.3 %).

Discussion

We have developed a novel method for the “Verification of Reference Intervals based on the Uncertainty of Sampling”, short VeRUS. Our systematic comparison of VeRUS to the current de facto standard, the CLSI binomial test [5], and to ELs [16] demonstrates that VeRUS is a robust and consistent method for the verification of RIs.

Our results (Figures 2 and 3; Supplementary Figures 2–7) show that VeRUS and the binomial test verify fairly consistent ranges of candidate RIs on percentile scale, whereas ELs yield varying outcomes depending on the specific biomarker distribution. This difference likely arises because ELs assume log-normal distributions [16], while VeRUS utilizes estimated distribution parameters provided by refineR which better reflect the biomarker specific distribution.

Deviations from the generally consistently shaped acceptance ranges are observed when verifying the highly skewed biomarkers TSH and IgE (Figures 2 and 3; Supplementary Figures 2–7) with VeRUS. These deviations are a direct consequence of the integrated asymmetry correction. The asymmetry correction is a crucial feature, as perfectly consistent acceptance areas on highly skewed biomarkers translate into unrealistically narrow UMs of the lower reference limits, thereby reducing practical utility. Therefore, the asymmetry correction ensures more applicable UMs in these scenarios. Additionally, VeRUS generally verified the “correct/true” RI at higher rates than ELs (Table 2).

VeRUS and ELs both overcome the main disadvantages of the binomial test. Neither requires the collection of dedicated reference samples when combined with indirect methods like refineR and both are capable of rejecting inappropriately wide candidate RIs [5], 8]. Furthermore, the binomial test with n=20 has a particularly gradual transition from high rates of verification to effectively 100 % rejection. This indicates a high probability of accepting RIs as verified that almost certainly should not be considered equivalent. However, by increasing the sample size the transition gets substantially less gradual (Supplementary Figure 10). In comparison, the gradual transitions observed in ELs and VeRUS caused by imperfect model estimation were minor. If the refineR model would always correspond to the ground truth, i.e., eliminating errors occurring in the estimation step, it would result in perfectly sharp transitions (Supplementary Figures 4–7).

However, there is no universally accepted definition to what degree RIs are allowed to differ to be considered equivalent. Like any statistical test, the verification process is ultimately a compromise between acceptable false positive and false negative rates. As RIs are guiding clinical decisions, the ideal verification method would maximize the rejection of RIs supporting adverse decisions with respect to clinical outcome and accept RIs that, while differing from the theoretical optimum, maintain clinical utility. These considerations require biomarker-specific rules depending on the clinical significance of each biomarker, which is beyond the scope of this paper.

In contrast, the methods for RI verification compared in this study apply generic rules for acceptance independent of the biomarker to be evaluated. The CLSI binomial test is defined purely statistically. ELs are derived from the concept of permissible imprecision [16], while VeRUS relies on UMs that are based on the uncertainty of sampling generally accepted for establishing RIs with the nonparametric direct approach [5].

As UMs are conceptually derived from CIs, it should be noted that while non-overlapping CIs indicate a statistically significant difference, overlapping CIs do not imply equivalence with the same strength as a formal hypothesis test. Achieving this level of significance may require that the UM for each reference limit includes the point estimate of the corresponding reference limit [25]. This more stringent acceptance criterion results in a much narrower range of verified candidate RIs compared to the CLSI binomial test (Supplementary Figures 8, 9). Conversely, the acceptance criterion of overlapping UMs yields a similar range of verified upper and lower limits as the CLSI binomial test, while also rejecting RIs that are too wide (Figure 2; Supplementary Figures 2, 4, 6). These findings suggest that VeRUS, at its default settings, serves as a reasonable tool for verifying RIs.

In the common case that only the limits of the candidate RI are available but no additional distributional information, VeRUS assumes a shared distributional shape (λ and δ) of the candidate and local reference distributions. It is advisable to assess whether this modeling choice is appropriate within the respective context. The underlying premise is that a biomarker’s fundamental distribution shape is likely primarily governed by physiological factors, which should be conserved between the comparable populations required for RI transfer [5]. In contrast, inter-laboratory variations in analytical methods and preanalytical processes, e.g., specimen collection, transportation, and handling, manifest more directly as changes in µ and σ. Thus, this data-driven approach of inferring the shape from the local data may be more plausible than the common practice of assuming a Gaussian or log-normal distribution for all biomarkers when no additional information is available [16], 17], 23].

The universal acceptance criteria compared in this study might not be the perfect choice for every biomarker. For example, the defaults of VeRUS are based on the approximation of the 90 % CIs of the nonparametric method of establishing RIs, using 120 samples. However, this sample size is recognized as potentially inadequate for establishing RIs for highly skewed – e.g., log-normal – distributions, often yielding excessively wide upper CIs [5], 15]. This inherent distributional property is also mirrored by VeRUS that directly applies the n=120 nonparametric uncertainty via UMs resulting in the aforementioned impractically narrow UMs around the lower reference limit and extensively wide UMs around the upper limits (Supplementary Figures 3, 7). Hence, the default n=120 assumption itself might be suboptimal for verifying certain highly skewed biomarkers depending on clinical considerations, although VeRUS’s asymmetry correction mitigates the most extreme effects. Recognizing the issue of excessively wide upper CIs in the context of RI establishment, larger sample sizes are recommended for biomarkers with skewed distributions [5], 15]. Importantly, VeRUS is a flexible approach allowing users to adjust the parameters used for UM calculation including the underlying sample size or the width of the approximated CIs (see Table 1). This flexibility of VeRUS allows for adaptation of the verification stringency to better suit specific biomarker characteristics or differing levels of required clinical certainty.

Building upon its flexibility, VeRUS can be extended to quantify the similarity between two RIs. The UMs are defined as the approximate CIs of the population quantiles for a given sample size. Consequently, larger sample sizes lead to smaller UMs and vice versa. The degree of similarity between the two RIs can be quantified by the maximum sample size for which all corresponding UMs of the two RIs overlap.

Conclusions

In this study, we introduced VeRUS, a novel method for verifying RIs, and compared it to the CLSI binomial test and ELs. Since VeRUS and ELs make use of already available routine data, they are substantially less resource intensive than the binomial test. As no collection of reference samples is required, these methods enable the verification of RIs in scenarios where RI verification is often restricted, such as in resource-limited settings, for highly partitioned RIs or in pediatrics. Our analysis revealed that VeRUS exhibited robust statistical properties while verifying fairly consistent ranges of candidate RIs. Removing the need for sample collection, its consistency and flexibility make VeRUS a valuable tool for regular RI verification.