A checklist for critical appraisal of studies of biological variation

Introduction

Biological variation data have many applications in laboratory medicine [1]. Those include setting of analytical performance specifications based on components of biological variation [2]. Models attempt to minimise the ratio of “analytical noise” to the biological signal within clinical laboratory measurements and to ensure that population-based reference intervals are transferable over time and geography. If analytical goals are achieved then this implies that there is no advantage in further improvement of the method in terms of the derived quality standard. The valid use of biological variation data (BVD) in this and other applications requires that they are robust and have characteristics concordant with those of the population to which the measurement procedure is to be applied. This requires that BVD are appropriately quantified, well defined, characterised and understood to enable their translation into safe and effective applications and transportability across populations and health care systems.

There are parallels to be drawn between the production and use of BVD and production and use of reference values [3–8]. The requirements for delivery and characterisation of the latter have been clearly identified by the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) and more recently in guidance issued by the Clinical and Laboratory Standards Institute (CLSI) [9]. The approach identifies need for characterisation of populations studied, methods for production of data, and the statistical treatment of data. A need for this degree of definition is accepted in the context of population-based reference values and is as important in the context of BVD. There are currently no recognised international standards for the production and reporting of BVD.

Review of the literature relating to biological variation (BV) identifies a significant volume of work stretching back over 40 years. The papers published are of varying quality in terms of study designs and presentation. This delivers a high degree of uncertainty around published estimates of BV [10–12]. The heterogeneity in quality of BVD and the use of non-standardised terminology to describe the data in publications are also problematic [13] and provide further complexity for the user. Attempts to make BVD accessible to laboratory medicine specialists have resulted in the delivery of a biological variation data base by Ricós and colleagues which is currently hosted online [14–16]. The criteria they used to construct the database have been published recently [17]. The authors recognised that there is a need to further develop criteria to better characterise BVD and enable selection of BVD from publications for inclusion in their database. In the absence of such criteria compiled data collections are readily available in accessible formats that potentially enable an uncritical application of often poorly characterised data sets.

Work has been undertaken to develop the critical appraisal checklist presented here by the Biological Variation Working Group (BVWG) [18] established by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM). The checklist is similar to that published as part of the Standards for Reporting of Diagnostic Accuracy guideline (STARD) which aimed to raise the quality of publications in that area [19, 20]. The checklist proposed here, by the BVWG, is a tool that will assist laboratory medicine professionals to generate and publish high quality BVD accompanied by relevant metadata to enable safe accurate and effective clinical applications [21]. It provides a framework for end users of BVD to critically appraise existing publications, and for reviewers of future BVD publications to assure a standard of reporting that enables valid clinical application of new BVD studies by those same end users. Studies and publications that are compliant with the checklist will allow effective transportability of appropriately derived and characterised BVD across health care systems as reference data. It follows that valid application of BVD by laboratory medicine specialists at other locations or times requires recording and transmission of key metadata [22]. Those metadata describe and give information concerning the key attributes of BVD that impact on transportability (e.g., demographics of the population from which they were derived, description of the analytical methods used etc). The metadata can be further grouped into a defined “data archetype” to enable consistency and constancy of transmission of BVD through information and knowledge management systems as reference data. It has been proposed that definition of BVD as transportable reference data requires that key metadata forming the archetype can be clearly identified within six domains (e.g., study characteristics, population characteristics, and data characteristics) [21, 23]. The key metadata from each of those domains delivers a minimum data set (MDS) that can be used to define the data archetype as part of a future health informatics standard for onward transmission of BVD.

Adherence to STARD guidelines is required by many journals for studies on diagnostic accuracy providing an important checklist of items to be included in publications. The positive impact of the STARD guidelines has been acknowledged by a Consortium of Laboratory Medicine Journal Editors [24]. The importance of the detail required in publications to enable the insight of readers into the value of research is recognised. The Biological Variation Data Reporting critical appraisal Checklist (BioVarC) is proposed to deliver a similar approach and benefit. It stands as a precursor initiative to production of any formalised standards for delivery and reporting of BV studies, being based on an evaluation of current best practice and the need to ensure incorporation of key metadata into publications that impact upon the utility of BVD. Such standards may enable delivery of a need identified in 1989 by Fraser and Harris to be able to ensure comparability of data by use of common study design and analysis of data [25].

Materials and methods

The BVWG established by the EFLM consisted of laboratory medicine specialist with a remit to establish a critical appraisal checklist for publication of biological variation data. The group has studied existing BV literature and databases and undertaken discussions to enable construction of a critical appraisal checklist applicable to existing and future publications of BVD. The group have further identified a MDS required by users to enable transportability of BVD into local clinical practice.

Results

The checklist is shown in Table 1. It is based on the same structure as the STARD table and identifies six main items for focus with a number of sub items. The sub items have been additionally mapped to minimum data set domains (Table 2; MDS: A–F) previously identified by the BVWG [21, 23]. Domain (F), which relates to a data rating concept, is not included in the checklist at this time as this is a quality measure that requires further development. The attributes identified in MDS domains A–E are identified as describing key metadata to enable safe, accurate and effective use of BVD by third party users. Domains E and F will provide further sources of information to support a users decision-making processes in the context of their clinical practice and support delivery of data through media such as online databases.

Discussion

There are currently no clearly defined internationally recognised standards for production, reporting and transmission of BVD. If BVD are considered to be reference data it follows that they should be characterised and described with sufficient key metadata to enable valid applications in clinical settings. If they are to be used to set quality standards then users of the data must have confidence that the data are the product of appropriately designed and delivered studies and further aware of the confidence limits around the estimates of the variability which they are about to use. Delivery of confidence in both senses will allow appropriate contextual application of BVD sets in clinical settings across populations and health care systems (transportability).

Reviews of BVD available for a range of analytical targets highlight many issues in study designs and reporting [10–12]. This provides a major challenge to users trying to translate the content of individual publications into practice and to those attempting to collate valid data sets into databases for use by multiple users [17]. The problems associated with, and the needs for standardisation of, terminology used in publications of BV studies have also been highlighted by Simundic et al. [13].

The critical appraisal checklist presented here follows an approach that has been shown to raise standards of reporting of studies in other settings (STARD). The BVWG have attempted to identify major items, sub items to be considered in the design, delivery and reporting of BV studies. It should apply equally to laboratory based measurements and quantitative physiological measurements (e.g., blood pressure). Compliance with the checklist will enable authors, reviewers and journal editors to assure that studies are fit for purpose, appropriately powered [26], share common terminology [13] and deliver estimates of BV accompanied by key metadata required to enable valid application of the BVD described [20–22]. Use of BV estimates accompanied by an MDS outlined in Table 2, delivers key metadata to enable transportability of data and further enable compilation of a database of BVD for use in setting of quality standards and other applications. Metadata could include the use of recognised coding systems to enable ease of transmission of relevant detail. Logical observation identifiers names and codes (LOINC) [27], the systemised nomenclature of medicine (SNOMED) [28] and the nomenclature, properties and units coding system (C-NPU) [29] provide examples of such. The MDS provides the foundation for construction of a data archetype to enable consistency and constancy of transmission of BVD through information and knowledge management systems as reference data. This is an important concept. Transportation of poorly defined and characterised BVD to populations that do not share characteristics may not only lead to setting of erroneous quality standards, but may also deliver patient safety issues. As an example of the latter if BVD are used to set reference change values, significance of change may be misidentified in the target population if they do not exhibit the same biological variation as the population from which they were derived.

The concept of scoring publications containing BVD has been described by Perich et al. [17]. It is proposed by the BVWG that a more sophisticated score should be included in the MDS to accompany BVD as a quality measure to further aid to users as a quality measure [18]. This concept needs to be further developed and has parallels with the scoring of medical evidence.

The checklist described here is based on expert opinion and provides an interim framework that may be used prospectively to improve future reporting of BVD and retrospectively to enable critical appraisal of existing publications. It will benefit from future iterations and develop in the event of delivery of defined and agreed standards for generation and reporting of BVD. Development of specific standards for the generation, reporting and transmission of BVD should also be considered by appropriate bodies [21]. Until such are available the detailed supporting information could be supplied by a series of publications similar to those developed by the IFCC and applying to reference values [3–8].

The practical application of this current checklist will be aided by current and future developments. Currently delivery of a pro forma set of focused questions by the BVWG and others will enable users to deliver a practical objective assessment of compliance of BVD studies with the high level checklist. This tool is being developed for future publication and to be made available online. In the future standardised approaches to data management might be greatly aided by the creation of a bespoke statistical package that supports appropriate design of studies and data analysis. Availability and internet-based access to such tools and supporting information should increase understanding of factors that impact upon the utility of BVD, enable valid application of the data and drive up the quality of future published studies.

Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

Financial support: None declared.

Employment or leadership: None declared.

Honorarium: None declared.

Competing interests: The funding organisation(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.