NUMBER: standardized reference intervals in the Netherlands using a ‘big data’ approach

Introduction

Interchangeability of laboratory test results across laboratories and in time is a major topic in laboratory medicine. The implementation of the Dutch External Quality Assessment (EQA) Program ‘SKML Combi New Style’ in 2005, using commutable and targeted sera, has proven to be very effective in reducing median inter-laboratory coefficients of variation for electrolytes, substrates and enzymes in the Netherlands [1]. However, despite the standardization of medical tests, national reference intervals and decision limits are still lacking for many of these tests in the Netherlands. Reference intervals are still established per laboratory using variable not evidence-based approaches: e.g. based on information from manufacturers’ product inserts, from literature or from healthy controls analyzed by the laboratory itself. As a result, reference intervals differ according to manufacturer, method principle and/or method generation and are often based on more strict pre-analytical conditions compared to those applied in daily clinical practice, leading to higher between-laboratory variation in reference intervals than the analytical variation in measurement results [2], [3], [4]. This hinders the national use of reference intervals in clinical guidelines, prevents adequate interpretation of laboratory test results and leads to incorrect and unequal treatment of patients [3], [4]. Interchangeability of laboratory test names, units and reference intervals in the Netherlands has become an absolute necessity due to recent developments, such as the introduction of electronical patient records in which laboratory results from different laboratories are combined, and the re-organization of the Dutch healthcare system in which patients with multiple health conditions are treated in more than one hospital or are treated by different physicians in primary, secondary and tertiary care.

The conventional settlement of reference intervals with the direct method, e.g. collecting and analyzing material from healthy control donors, is a time consuming and costly process. Following the CLSI protocol EP28-A3c, at least 120 samples should be tested, for each subcategory when applicable [5], leading to sometimes >1000 measurements per analyte when age and sex subcategories are necessary. The Committee on Reference Intervals and Decision Limits (C-RIDL) from the International Federation for Clinical Chemistry and Laboratory Medicine has published a detailed protocol on conducting multicenter reference interval studies, including requirements for recruitment of healthy volunteers (using health questionnaires), sample collection, cross-checking, ethics and data-analysis [6], [7]. These recommended methods for sample collection, centrifuging and storage differ from everyday practice [8], leading to reference intervals that will not always be applicable to routine practice because it will lead to excess flagging [9]. Given the workload, burden and costs of the direct method of establishing reference intervals, alternative methods are explored for establishing reference intervals, as described by the IFCC Committee on Reference Intervals and Decision Limits [10].

Under the umbrella of SKML/Calibration 2.000 [11], we have initiated a national endeavor named NUMBER (Nederlandse UniforMe Beslisgrenzen En Referentie-intervallen), in order to accomplish standardized reference intervals in the Netherlands. In this project, we adopt an indirect ‘big data’ approach to determine reference intervals [12], based on the evidence-based approach that was applied in Australia and New Zealand (https://www.aacb.asn.au/aboutus/harmonization-committee) [4], [13], [14], using millions of test results that are readily available in existing clinical laboratory databases from patients visiting general practitioners. The ‘big data’ approach is particularly usable in the Netherlands, because a unique, category 1 EQA system with value assigned, commutable EQA-materials, covering the clinically relevant concentration range, is in place. The Dutch EQA organization SKML collaborates with JCTLM-listed reference laboratories which perform value assignments [15] using primary reference methods and reference materials [16]. Furthermore, a multi-sample evaluation scoring system (MUSE) has been introduced [17], based on the Six Sigma metric combined with the bias and imprecision criteria from the Milan conference models [18], allowing monitoring of trueness and precision over time, based on predefined scientific criteria. This approach is also a method which allows periodic review of the established reference interval, as required by ISO 15189 (Chapter 5.5.2). We initially focused on SI-traceable general chemistry tests and enzyme tests defined by their reference measurement procedure, in which trueness can be verified 2-weekly by the Dutch EQA scheme ‘SKML Combi General Chemistry’ [15] using its category 1 trueness verification program.

Materials and methods

A schematic overview of the project outline is shown in Figure 1.

Figure 1:

Project outline.

Results

Sixteen clinical laboratories across the country provided anonymous test results from patients visiting general practitioners over a period of 1 year, leading to a total of 7,574,327 analyzed results. The 16 laboratories represented the complete geographical area of the Netherlands, the four main clinical chemistry analyzer manufacturers and both heparin plasma and serum matrices (Figure 2). Thirteen laboratories used lithium heparin collection tubes, and three used serum tubes, either with or without clotting activator (Becton Dickinson or Greiner). The majority of laboratories (85%) used tubes with a gel for separation. Besides providing the test results for the data analyses and additional information on pre-analytical conditions and test characteristics, the laboratory specialists in clinical chemistry, residents in clinical chemistry or senior technical staff of the participating laboratories took seat in the NUMBER expert meetings.

Figure 2:

Map of the Netherlands indicating the location, platform and matrix of the participating laboratories.

During the three expert meetings organized in 2017, consensus was obtained about (a) the standardized reference intervals for ALT, albumin, ALP, amylase, (anorganic) phosphate, AST, (total) bilirubin, calcium, chloride, GGT, iron, potassium, creatinine, LD, magnesium, sodium, total protein and urea, and (b) the need for partitioning in sex, age, matrix or method categories for each analyte. An overview of the standardized reference intervals is presented in Table 1.

Flagging rates were calculated in five representative independent datasets to verify the plausibility of the calculated reference intervals using the ‘big data’ approach. Flagging rates were all within the plus or minus 5% (±2%) range (Figure 3), which was considered acceptable during the expert meetings (consensus decision). The higher flagging rates below the lower limit of the reference interval for iron were expected, as no hemoglobin data were available in the validation data sets.

Figure 3:

Percentage flagging rates when applying standardized reference intervals in an independent dataset.

No consensus has been reached yet for implementation of the calculated reference intervals for creatinine kinase and uric acid, as these reference intervals were substantially higher than currently applied in the participating laboratories and higher than to be expected in healthy individuals. For glucose (fasting and non-fasting), it was decided in expert meeting 1, in compliance with the Milan hierarchy [18], to recommend the use of outcome-based clinical decision limits rather than a reference interval, such as provided by the WHO and American Diabetes Association [31], [32]. As a result of this decision, we did not calculate standardized reference intervals for glucose.

We observed substantial differences in the calculated standardized reference intervals using data from laboratories using the BCG and the BCP method for albumin, confirming the need to introduce separate reference intervals for BCG and BCP (Figure 4). In addition, we observed larger variation in currently applied reference intervals for potassium (based on manufacturers’ product inserts, literature, or healthy controls analyzed by the laboratory itself) in the participating laboratories than in the calculated reference intervals using the ‘big data’ approach with separate reference intervals for plasma and serum matrices (Figure 5). The observed differences in reference intervals for laboratories using heparin plasma and serum for potassium and total protein were in accordance with previous studies [26], [27], [28]. For plasma/serum ALT, ALP, AST, GGT and creatinine, considerable differences were found between the different age groups (Figure 6).

Figure 4:

Reference intervals (calculated as mean±2 SD) for serum/plasma albumin stratified by method (BCP vs BCG).

Figure 5:

Current reference intervals for potassium in participating laboratories (left panel), calculated reference intervals for potassium in participating laboratories (middle panel) and proposed standardized reference intervals (right panel) for potassium in serum and heparin plasma.

Figure 6:

Age and sex effect on the reference intervals for ALT and creatinine.

Discussion

Building on the successes of the Dutch National Endeavor Calibration 2000 (now called Calibration 2.000) [11], [33] and inspired by the evidence-based approach that was applied in Australia and New Zealand [4], we set up a project and road map for sustainable determination of actual reference intervals, eventually stratified by gender and age, using a ‘big-data’ approach and medical test results from patients visiting their general practitioners. As metrological traceability of test results produced by medical laboratories in the Netherlands is guaranteed through the category 1 EQA scheme with value-assigned trueness verifiers, standardized reference intervals should be feasible. The fully commutable, targeted EQA materials that have been introduced in the Netherlands since 1998, in combination with the MUSE scoring system [17], have significantly reduced inter-laboratory variation for electrolytes, substrates and enzymes in the Netherlands [1], and has led to significant, further harmonization of test results [34]. Unfortunately, the standardization of tests has not yet led to harmonization of the total test process, because of substantial variation in both the pre-analytical phase and post-analytical phase. Reference intervals that are currently applied by Dutch medical laboratories originate from variable sources such as literature, manufacturer inserts, data mining from the laboratory information system, or regional agreements, leading to higher variation in reference intervals than the variation of analytical results from the different clinical chemistry platforms [2], [3], [4]. This situation is suboptimal and may lead to misinterpretation of the generated results by treating physicians, additional unnecessary diagnostic evaluations, over- or under-treatment or inaccurate estimations about prognosis [3], [4]. The use of standardized reference intervals for medical tests for which the traceability chain is in place, is also an essential prerequisite for standardizing the total test process [35]. Standardized reference intervals will facilitate the uniform interpretation of test results in the case of referral of the patient to a different hospital or from primary to secondary care and vice versa. The upcoming digitalization in the medical world, and the development of national or personal health records is another driver for standardization of the total test process.

Because the establishment of reference intervals according to CLSI protocol EP28-A3c [5] and C-RIDL [7] is a time consuming, unsustainable and a costly expedition, we searched for an efficient and future proof approach for periodic (re-)establishment of standardized reference intervals in the Netherlands. We therefore adopted the evidence-based approach from Australia and New Zealand using existing data from patients visiting general practitioners to establish the reference intervals using a ‘big data’ approach. The calculated standardized reference intervals in our project are comparable to the reference intervals found in other important reference interval studies such as CALIPER (direct method) [36], ARIA (indirect method) [4] and NORIP (direct method) [8] (Supplementary Material Table 2), supporting the chosen approach. The reference intervals that we propose can be implemented in the near future by all Dutch laboratories if they (a) adhere to the ‘Venipuncture’ guideline of the Dutch Society for Clinical Chemistry and Laboratory Medicine [19], (b) have comparable pre-analytical work-up conditions (such as collection tubes, transportation, temperature monitoring and centrifugation conditions) as applied by the participating laboratories in this project, (c) are using selective tests with clearly defined measurands that are traceable to JCTLM-listed reference methods and/or reference materials, and (d) have a MUSE performance score of at least one (sigma 2 or higher) [17] in the SKML EQA scheme (Supplementary Material Figure 1).

Interestingly, our study shows significant age and/or lifestyle effects for ALT, ALP, AST, GGT and creatinine. The proposed standardized reference intervals for these tests are substantially higher than the (IFCC) reference intervals that are commonly applied for these tests in the Netherlands [37], [38], [39], [40], [41], [42]. The results with respect to creatinine build on evidence from earlier studies and are likely to reflect an aging effect. It is well known that in old age renal function decreases as a result of loss of glomeruli and decline in renal blood flow [43], [44]. This is illustrated by results from the Leiden 85-plus Study, a population-based prospective follow-up study of 85 year olds in Leiden The Netherlands, in which median creatinine clearance (estimated by the Cockcroft-Gault formula at the time) at age 85 was 45 mL/min [44]. However, the age-dependent increase found in the reference intervals for ALT, AST and GGT might be due to natural aging of the organs and/or due to lifestyle of the Dutch population, where obesity, metabolic syndrome, lack of exercise and alcohol consumption are prevalent [45], [46]. The elevated reference intervals are not found in the oldest age category (81+), suggesting that aging is not solely responsible for the increase in reference intervals for these enzymes. Data are increasingly becoming available that question the extrapolation of ‘common’ medical knowledge into the highest age groups. The effects of some classical (laboratory) determinants of disease and mortality in middle age, such as hypothyroidism [47], hypercholesterolemia [48] and elevated ALT [49], [50], [51], have been shown to disappear or even reverse in the oldest old, indicating complex relationships between common laboratory values and clinical outcomes in the elderly. This is an important topic of further research. The increased upper limits of the liver enzymes compared to the IFCC reference intervals that are commonly applied in the Netherlands [37], [39, 40, 41, 42, 52] was discussed extensively during the expert meetings. Even though a lifestyle component may be involved, our results are supported by the flagging rate analyses. This is important, as frequent flagging (marked on the laboratory rapport) may distract attention from true pathological results [9].

Our study has several strengths. First, compared to the direct method of establishing reference intervals, the applied ‘big data’ approach in NUMBER is cost-efficient in the sense that it avoids collection and analyzing material from healthy control donors. Secondly, the chosen indirect method and statistical methodology has the potential to be a sustainable method that can be repeated at set time intervals to re-evaluate the applied reference intervals in the future, to identify effects of population changes due to epidemiological transitions, exposure to the microbiome, aging and increasing welfare or advances in pre-analytic and analytic methods. When centrally organized, this will facilitate the Dutch laboratories in their duty to monitor and evaluate applied reference intervals conform ISO15189. Third, the participating laboratories represented the complete geographical area of the Netherlands, the four main clinical chemistry analyzer manufacturers and both plasma and serum matrices. This is of vital importance for the national implementation of the standardized reference intervals that were obtained from this project. Fourth, the Dutch EQA system uses, where possible, commutable, targeted materials. This is a minimum requirement for determining standardized reference intervals, applicable to all laboratories that show a Six Sigma score of at least 2 in the MUSE scoring system.

Some limitations also have to be acknowledged. First, as a consequence of using anonymous laboratory test results, clinical information was not available. We tried to select a ‘healthy’ population by (1) using laboratory tests ordered by general practitioners (that in general order blood tests to rule out disease, meaning that the a priori chance of illness is low), assuming that the majority of individuals are free of disease, (2) we excluded laboratory results when phlebotomy was performed at home, (3) used large numbers of test results in order to have confidence about the distribution of the central majority of the cohort [53], and (4) excluded results from related tests when results that were supposed not to belong to the reference population were detected. However, we cannot fully exclude the possibility of selection bias, especially for those tests that are routinely performed to monitor patients with chronic diseases (like risk management for cardiovascular disease) or are performed only for a very distinct indication. This is, in particular, the reason why we did not reach consensus yet for the implementation of the calculated reference intervals for CK and uric acid. As recommended in the GP guideline ‘Cardiovascular Risk Management’ of the Dutch College of General Practitioners [54], CK is commonly measured in patients using cholesterol synthesis inhibitor (statins), which are known for side effects such as myopathy [55], [56] and are used by more than 11% of the Dutch population [57]. In the case of uric acid, there is no universal definition for hyperuricemia. A cut-off defined by the solubility limit of uric acid (>0.42 mmol/L for men and >0.34 mmol/L for women) is commonly applied [58], based on the study by Campion and colleagues [59]. Others have already suggested that the reference intervals for uric acid may need to be redefined, because of a ‘shift to the right’ [60]. However, as uric acid is primarily requested for the diagnostic indication gout/arthritis, as is recommended in the Dutch GP Guideline ‘Arthritis’ [61], our dataset with results from patients visiting general practitioners might not be representative for the normal/healthy population for this test. Therefore, the preliminary calculated standardized reference intervals for CK and uric acid have to be evaluated in a healthy control group or population-based study such as LifeLines [62] before national implementation can take place. Second, the data mining method requires large numbers of individuals [53]. Although more than seven million test results were available from 16 participating laboratories, in some subgroups still very small numbers of tests results were available. Therefore, it was not always possible to propose reference intervals for the youngest age groups. A different representative population should be sought for the youngest age categories (0–1 year and 1–5 years). The CALIPER study determined reference intervals in children separately on several platforms, but did not standardize reference intervals across methods [36]. Zierk and colleagues applied an indirect method to generate pediatric reference intervals for biochemistry analytes analyzed on one platform during clinical care in a tertiary care center in Germany [63], [64]. The Canadian and German results can therefore not be translated to the Dutch medical laboratory setting. However, the continuous reference intervals that were provided by both projects are a topic of further study in the Netherlands.

In conclusion, with the use of a big data approach we were able to calculate age and sex stratified standardized reference intervals for 18 clinical chemistry tests that can be implemented by each Dutch laboratory with a Sigma score of at least 2 in the SKML EQA program and platform with a full traceability chain in place. Nationwide implementation of these established reference intervals has the potential to improve unequivocal interpretation of test results leading to better diagnosis, treatment, risk stratification and follow-up. As a next step, a fourth expert meeting with the ambassadors of the NUMBER initiative will be organized early 2018 to come to agreement for national reference intervals for CK and uric acid after evaluation in a healthy control group. In addition, promotion material will be developed and a detailed plan for the implementation of the reference intervals will be established (Figure 1, phase 5). The working group and ambassadors nationally and locally promote adherence to the standardized reference intervals in all Dutch laboratories according to a national implementation plan. The promotion of the use of the national reference intervals will not only lead to better exchangeability of laboratory results between different medical laboratories, but will also enhance the harmonization of methods, because the laboratory specialists are encouraged to calibrate their methods to achieve a bias of almost zero. Moreover, in cooperation with SKML and Calibration 2.000, our ambition is to formulate formal recommendations regarding more selective methods, e.g. enzymatic method for creatinine and the BCP method for albumin, to achieve the most reliable and exchangeable results for these parameters. Furthermore, we will set up a sustainable national surveillance system to structurally monitor constancy in reference intervals with time in cooperation with SKML using their existing digital infrastructure for all medical laboratories in the Netherlands (Figure 1, phase 6). When this is accomplished, not only biological effects on reference intervals but also the effects of evolution of test performance and test standardization on the reference interval will be periodically evaluated.