Reliability of biological variation data available in an online database: need for improvement

Introduction

Knowledge of biological variation (BV) of measurands enables a model suggested to set quality specifications for the analytical phase within medical laboratories [1]. The model is one of five identified at Stockholm consensus conference on quality specifications in laboratory medicine [2]. The BV model was recognized as the second in the hierarchy to fulfill medical needs.

BV data, if correctly defined, have other highly useful and important applications. Those include the assessment of significance of change in serial measurements observed within a subject [3], definition of analytical goals to enable safe application of laboratory measurements in a clinical setting [4] and derivation of an index of individuality (II) to enable assessment of the utility of conventional population-based reference intervals [5]. Given the importance of BV data, it is clear that access to this data is essential to the clinical laboratory, and that users should be confident that they are well characterized and fit for purpose.

Compiled sources of this data have been made available to the laboratory medicine specialists. Fraser published a collection of BV data (including data from 1988 to 1991) in 1992 [6]. In 1999 Ricos et al. [7] compiled and made available a database that included BV of biochemical and hematological measurands. This has evolved into a BV database (BV-DB) gathered from 240 publications that continues to be expanded, updated and published online [8]. It is extensive and presents data and derived analytical quality specifications for imprecision (I%), bias (B%) and total error (TE%) for more than 350 measurands. It is however, only recently that Perich et al. have published criteria used to evaluate the reliability of the data included in the BV-DB [9].

The BV-DB provides an important source of BV data that is widely available, used and referenced for clinical application. However, some limitations of this approach have, been identified at a meeting of over 40 medical laboratory opinion leaders [10]. Concerns have been further specifically identified in published reviews of the BV of glycated hemoglobin (HbA_1c) [11], C-reactive protein (CRP) [12], urinary albumin [13] and some enzymes [14]. These publications lead to the conclusion that there is a need for a standardized approach to derivation and publication of BV data to enable their safe accurate and effective application in a clinical setting.

This paper explores the further reliability of the online BV-DB and identifies ongoing work to address some of the limitations of the approach and work to be undertaken to enable an expansion of the available data through co-operative working by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM), Biological Variation Working Group (BVWG) [15], in collaboration with a Spanish group responsible for the BV-DB updating [8].

Materials and methods

Results

Figures 1 and 2 are derived from examination of the BV-DB. The database lists data for measurands in serum, plasma, whole blood and urine. Figure 1 graphically displays a breakdown of numbers of publications referenced per measurands in each matrix. It is clear that the data base entries are mainly derived from a single publication. Where multiple publications are available the database entry represents a median value of those published [9].

Figure 1:

Distribution of measurands stratified by the number of cited references upon which the BV-DB entry is based.

The four panels display data for measurands in different sample matrices included in the database, (A) serum, (B) plasma, (C) whole blood and (D) urine.

Figure 2:

Distribution of 247 papers included in the database stratified by year of publication.

Figure 2 shows the distribution of 247 papers stratified by years of publication included in the database and used in order to calculate BV data.

Of the 247 references cited in the database 36% were published before the seminal paper of Fraser and Harris in 1989 [17] describing a preferred protocol for the generation of data. Furthermore a total of 271 of the database entries were based on evidence from either one or two publications. Of these 28% were published before 1989 with a further 33% between 1989 and 2000.

Discussion

Critical review of the BV-DB highlights a number of factors that might impact on the reliability of the BV data entries and translation into current practice. When taken into account alongside a number of systematic reviews [11, 12, 14] of BV data, some key issues identified need to be addressed to enable safe, accurate and effective application of these data.

Non-compliance with best practice delivers uncertainty around the data and the fact that a high proportion of the data sets are generated from single non-corroborated studies leads to uncertainty around the value and transportability of the data. It has been identified that age of some of the studies and subsequent developments in methodologies mean that data may not be suitable for application in today’s practice.

The availability of multiple studies per measurand highlights inconsistencies in approach to protocols applied in the various studies, issues with data management and other important factors that will impact on the absolute values obtained and make it difficult to combine data to enable singles data entry within the BV-DB. The existing data sets may however still be of use and there is a need to be able to assess their quality and value for inclusion of historical data within any future database. The provision of key data to enable users understanding of the quality of the data and the delivery of CIs around the estimates of BV data are essential for the future development of any BV-DB.

Roraas et al. showed [21] it is necessary to know analytical standard deviation (SD_A) and intra-subject standard deviation (SD_I) (to calculate the ratio SD_A/SD_I), number of replicates (there must be at least two of these replicates), number of samples/subject, and the number of individuals, in order to calculate CI around BV data. This information allows the calculation of the CIs and enables determination of the power of studies to deliver the required data. The magnitude of the CI of the SD_I [and also reference change value (RCV)] varies with the number of individuals, the number of samples from each individual, and the number of replicates [for ratio SD_A/SD_I>1] [21]. The analytical I% must be taken into consideration in order to obtain good estimates of BV: the higher SD_A is the wider CI will be. Study design, number of replicates, number of samples, and number of subjects have a great impact on the reliability with which we can estimate the SD_I and RCV of an analyte, and that the effect of these variables varies with the ratio of SD_A to SD_I. Fraser and Harris [17] stated that: “the components of variation can be obtained from a relatively small number of specimens collected from a small group of subjects”, this may be the case, but the work of Roraas et al. enable [21] the truth of this statement to be tested.

In the absence of a defined standard for the production of BV data, the work of Fraser and Harris [17] provides an excellent framework for determination of meaningful BV data. This is not always closely followed. They highlighted the need for duplicate measurements to be undertaken in a single run under conditions that minimize analytical I%. It is clear from this review that many studies do not follow this approach which makes it impossible to calculate CIs to be used in the assessment of the data. In addition, lack of knowledge of CIs opens up the possibility that data from non-compatible studies are being combined within the BV-DB and delivering error.

A further problem with the current BV-DB is that clearly not all studies are captured and the user is not able to consult the original papers to study the detail as they may have limited accessibility. This was the case with the HbA_1c example mentioned in this review. This might be considered to be less of an issue if recognized standards existing for generation and publication of data. In general assessment of the publications used to construct the current BV-DB it is worrying that the majority of papers failed to indicate the application of statistical tests of normality to the data [11, 12, 14]. By definition the CV of a Gaussian distribution cannot exceed 33.3% [14, 21], but, as shown in the results section, 10% of CV_I values reported in BV-DB are >33%. In the PCR example given in this paper the CV_I was seen to be >33%. In this and other cases the calculation of RCV is invalid known that the formula to calculate the RCV [2.77(CV_A²+CV_I²)^1/2] for bidirectional change at p<0.05, is valid only if the distribution of the data is Gaussian [3]. Suitable “health warnings” should accompany the data in such cases.

As recently underlined by two editorials published in Clinical Chemistry and Laboratory Medicine [27, 28] and by the editorial published in Annals of Clinical Biochemistry [29], a number of issues clearly undermine the value of the current BV-DB. These issues are being considered by the EFLM BVWG [30], in collaboration with a Spanish group responsible for the BV-DB updating [8].

In the absence of standards for generation and reporting of BV data the BVWG have proposed the use of a checklist that will assist laboratory medicine professionals to generate, compile and publish high quality BV data and to objectively assess the content and quality of existing publications and enable referees of future BV publications [30]. This is based on expert opinion and consideration of existing best practice (e.g., Fraser and Harris [17], Roraas et al. [21]). There is a need to deliver updated studies on biological variability using contemporary analytical techniques. Delivery of valid and appropriately powered studies is time consuming and organizationally difficult, and mechanisms are being developed to facilitate the delivery of materials in the form of a bank of specimens to be made available to enable updates of this type.

BVWG is working on a new sample collection in order to update the existing BV data.

The project “Samples collection from healthy volunteers for BV values update” has just been approved by the Ethical Committee of San Raffaele Hospital Milan, Italy, and is going to start in April 2015. This project is a multicenter study involving seven European laboratories with all the pre-analytical requirements complying with the ideal protocol identified by Fraser and Harris [17]. Samples will be collected from 105 healthy volunteers and stored at a central facility as a resource for the delivery of contemporary BV data.

This dual approach enables a driving up of the quality of BV data with the checklist enabling assessment and validation of historical data and provision of a framework to drive up the quality of future studies and their reporting to enable safe, accurate and effective application of the data. The approach of a specimen’s collection will enable delivery of up-to-date and valid BV data using contemporary analytical methods.

Conclusions

The concept of the BV-DB is powerful and the current database represents the product of an immense amount of work. It has proven to be a valuable resource, but it is clear that there are a number of issues that impact upon its current value.

There is massive variation in the quality of the studies delivering data reported in the current database. The delivery of a complete database that includes the products of appropriately powered studies delivering indices of BV with confidence intervals provides challenges. In the absence of recognized standards application of a BV critical checklist based on expert opinion will enable historical data to be reassessed and drive up the quality of future reported studies. This will improve the usefulness of future databases.

The need to deliver contemporary data can be facilitated by delivery of a bank of appropriately collected specimens which will enable new and valid BV data sets to be delivered for key target areas yet to be identified.