A standard to report biological variation data studies – based on an expert opinion

Background

Biological variation (BV) data have many important applications in laboratory medicine [1]. Those include definition of analytical performance specifications (APS), assessment of the validity of population-based reference intervals, assessment of significance of change of sequential clinical laboratory test results for one person, and estimation of personalized reference intervals [2]. The consensus statement arising from the first European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Strategic Conference in 2014 highlighted the need to be able to carefully assess the relevance and validity of BV data prior to their application [3]. It was concluded that the utility of BV data is compromised by confounders affecting the quality of the studies generating data and by the quality of reporting of BV studies. Deficiencies in the study quality (design and execution) affects the quality and veracity of the data with impacts upon the value of clinical applications of BV data [4, 5]. Deficiencies in the reporting of BV studies are just as problematic as they deny end users the opportunity to assess the quality and veracity of BV data and deliver uncertainty around the relevance and applicability of the published data to their clinical practice. This supports the view that published BV data are reference data [6]. To ensure the validity and transportability of reference data across time, geography, and clinical practices, they must be identifiable as robust and relevant. Thus, producers of BV data need to be aware of what constitutes a well-executed study to enable appropriate reporting of the key metadata required to support valid clinical applications by end users. A degree of understanding and rigor needs to be applied to production, reporting, and application of BV data paralleling that identified for population-based reference values published by the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) [7]. It follows that there is a standards and guidance requirement for both BV data generation and reporting. Delivery of such standards enables critical appraisal of historical BV studies and prospectively assures adoption of compliant best practice for future studies.

Standards for generation and reporting of BV studies and data

The position in 2024 is that there are no formally agreed standards or guidance for generation and reporting of BV data, or formally recognized process in place to ensure the quality and confidence issues extant around BV data are addressed. In 1989 Fraser and Harris described a prospective experimental approach to generation of BV data [8]. They stated that it was their hope that comparability of BV data might be provided by use of a common study design and analysis of study data. In the absence of any other formally agreed standard, the approach they described has sat as a de facto standard and has been used as such in most published prospective BV studies to date.

A recently published review, by Sandberg et al., of developments and future challenges around BV, highlights some developments with the aim to address the standardization and quality issues applying to BV data [9]. One of these, the Biological Variation Data Reporting Critical Appraisal Checklist, developed by the EFLM Biological Variation Working Group (BV-WG) [5], was first presented to the 2014 EFLM first Strategic Conference. This aimed to identify a standard for BV data publications to ensure that information (metadata) was available to enable safe and accurate use of these data across healthcare systems (transportability). The checklist maps to domains of a minimum data set (MDS) (Supplementary Material, Table S1), facilitating transportability of BV data as reference data (Table 1). Domains A to D of the MDS relate specifically to report content, Domains E and F are important to users wishing to apply data listed in centralized BV databases. Location of, and links to, original published study reports (Domain E) deliver ease of access to users to finer details of studies, enabling understanding of the provenance of the data and opportunity to appraise quality and relevance of the data prior to application. The Domain F concept provides the user with a succinct indicator of the comparative quality of the data against a common set of defined quality items (QI). Further work by the BV-WG and the EFLM Task Group for the Biological Variation Database, established following the EFLM first Strategic Conference, enabled a realisation of the Domain F concept delivering a system for grading of BV studies. The resultant Biological Variation Data Critical Appraisal Checklist (BIVAC) has been evaluated and published [10]. BIVAC incorporates QI that are used to deliver an overall grade for BV data studies. The studies are graded on a scale of “A” to “D” where “A” indicates full compliance, while a “D” grade identifies a study as non-compliant, egregiously flawed and at risk of delivering unusable data. In practice, the checklist and BIVAC grading system have subsequently underpinned the collation and publication of BV data in a publicly available online BV database, the EFLM BV Database [11]. Additionally, it has enabled metanalysis and combination of published data sets to improve the confidence limits around global point estimates of BV [10].

In summary, we see that the de facto standard approach to the delivery of BV data described by Fraser and Harris and the work of the EFLM groups provide a nascent standard and guidance framework for delivery of BV data to be published as reference data.

Disseminating and enabling compliance with emergent guidance for reporting and generation of BV data

The EFLM BV database now makes available 3069 BV records gleaned from 581 referenced publications, each having been critically appraised by the BIVAC and thus associated with a BIVAC grade. BIVAC has further enabled the members of the EFLM groups to undertake systematic reviews of published studies of the BV of important measurands [12], [13], [14], [15], [16], [17]. Those reviews found that a high proportion of studies are not fully compliant with the BIVAC (i.e. BIVAC grade of B, C and D). The challenges are typified in the critical appraisal and meta-analysis of BV estimates for kidney related analytes published reported by Jonker et al. [13]: of the 61 papers reviewed, only three received a BIVAC A grade. Four papers received a grade of B, the majority, 48 papers, were awarded a C, and 5 grade D. The most common reason for allocation of a C-grade related to deficiencies in the statistical analysis of the data. Those being failure to report on homogeneity of variances (BIVAC QI 10), failure to report number of data points used to calculate the BV estimates (QI 12), and failure to report and perform appropriate outlier analysis (QI 8). Grade D was awarded to papers that lacked standardisation of phlebotomy timing (QI 3) or issues related to the measurand and measurement method (QI 4). BIVAC QI 10 is particularly important, as lack of homogeneity of variances across the range of the measurand results (heteroscedasticity) will affect the generalisability of the BV estimates. More recent reviews indicate an improvement in the quality of BV studies published that state compliance with BIVAC [18].

These observations, and further work undertaken by the EFLM BV-WG and TG-BVD, have enabled development of a mind map which details the current state-of-the-art for delivery and reporting of BV studies employing the classical experimental prospective approach as described by Fraser and Harris [8]. The primary aim with the mind map is to deliver an accessible resource for researchers when planning and reporting a BV study compliant with the de facto standard approach to the generation of BV data. Compliance with the embedded guidance will deliver studies with reports classifiable as grade “A” on application of BIVAC and ensure that the recommended MDS is present to enable transportability of BV data as reference data. A proposal for the use of standardised terminology put forward by Simundic and colleagues when describing BV data sets is also embedded within the guidance to further enable convergence upon a BV standard [19].

The mind map was constructed using a commercially available package (Mindjet, MindManager 2018) which has functionality to enable generation of the interactive online version incorporated into the EFLM BV Database site (https://tinyurl.com/bvmindmap) [20, 11]. The content of the mind map reflects a consensus of the thinking, work and discussions of an established international group of experts in the field of BV focused through the EFLM BV-WG and TG-BVD [21, 22]. The mind map is incorporated in the EFLM BV Database website and it is thus freely available to anyone with access to the internet [20]. It was designed to enable users writing a paper on BV, to identify what is to be expected in the various sections of publication to attain a BIVAC grade A and includes hyperlinked access to key information to do this, in the form of embedded notes presentations, and relevant key references (Figure 1 and Supplementary Material, Figure S1). The map reflects the structure described in 2014 with main topics that equate to the sections for a publication, each of which expands to a varying number of sub-topics [6]. For example, the Method topic has 43 further sub-topics, 27 of which relate to study design (Figure 1). The embedded materials are made accessible to users as they expand the mind map. The user is for instance presented within the study design sub-topic related links to a publication describing how to assess the power of BV studies. There are also attached notes describing the relevant BIVAC scoring criteria. The Data Analysis sub-topic similarly includes links to online full text key references on statistical approaches to analysis of BV data (e.g. impact of data distributions upon the estimates of within person BV estimates and reference change values (RCV)). The user can consequently interact with the mind map to drill down on issues of interest. So, if they are interested in the issue homogeneity of variances (QI 10) mentioned earlier, notes embedded on the subtopic not only present the BIVAC grading criteria for the QI, but also indicate acceptable statistical tests to be used to assess variance homogeneity (Figure 1).

Figure 1:

Sample of the on-line mind map topic showing part of the sub-topic expansion with a check list for the method section with call outs illustrating embedded links to full text key references and links to sub-topic specific BIVAC QI grading information.

Benefits of the approach

Discussions within the EFLM groups around the value of the mind map indicated potential for realisation of benefits on its application to both historically published reports in addition to the delivery of new reports. Four improvement areas were identified.

The first improvement area considered is quality assessment and assurance of BV data. This initiative establishes a framework for designing, delivering, and reporting studies that align with a standard that document quality items (BIVAC QI) and ensures the reporting of valid BV data along with a required MDS. The second improvement area concerns consistency and comparability of BV studies. It introduces a structured framework for reviewers and end-users to apply to BV studies. Compliance will facilitate objective assessment of datasets from well-documented studies performed in compliance with a standardised approach. Consistent reporting of the MDS and BIVAC grading enable meta-analysis and combination of datasets from various studies to enhance confidence limits around BV data estimates and the identification of true differences, e.g. due to population differences or disease effects. This framework enables producers of BV data to consistently deliver reports for the publication of BV data as reference data that are therefore objectively comparable and combinable.

The third improvement centers on decision making. Compliance with standardized reporting guidelines ensures that BV data are delivered in a format that enables user to assess the relevance of published data to their clinical practice. Therefore, compliance enables decisions to be made around relevance and thus transportability of BV reference data. Adherence to standards delivers quality and instills confidence in users to apply the data.

The fourth and final improvement area relates to health informatics. A checklist approach and adoption of standardized terminology guarantees the reporting of an agreed MDS with the BV data to enable development of automated clinical informatics applications. The approach enables delivery of key metadata for the development of, and compliance with health informatics standards for the transmission and integration of BV data into information and knowledge management systems. It supports the delivery of BV data as reliable reference data and facilitates data sharing through online databases and other media.

Continuing improvement and the scale of benefits realization will deliver a need for further future iterations of the mind map content as the approaches to BV data delivery evolve, and the standard is refined.

Significance of the mind map initiative

Clinicians, laboratory professionals and other stakeholders need access to reliable knowledge and information to objectively evaluate various characteristics of BV data prior to its clinical application. We recognised that poorly executed BV studies and inadequate reporting of well executed studies impact on the utility and transportability of published BVD [4], [5], [6]. The mind map delivers access to content that can be used to help address BV data quality issues [11]. The embedded guidance will assist producers of BV data to write papers that report studies conforming to a state of the art delivery of the de facto standard for BV data generation [8] by the prospective experimental approach. Mind maps are a recognised tool for communication and education, enabling understanding and knowledge transfer around complex topics [23]. Use of mind map software to deliver an online guideline for writing papers about BV is innovative. If successful the approach will in time enable convergence of the delivery and reporting of BV studies to standards embedded within EFLM checklists [5, 10]. The numerous links embedded within the on-screen graphic view of the mind map to relevant notes, PowerPoint presentations and full text versions of key references will enable researchers to adopt best practice for reporting and thus delivery of fit for purpose BV reference data (Figure 1). It will support processes to address quality and confidence issues around BV data and thereby facilitate transportability and safe application of published estimates of BV as reference data [11, 12, 21, 22]. Additionally, when used in presentation mode, its web-based dissemination allows it to function as an educational resource, enhancing general knowledge of a crucial topic in the practice of laboratory medicine. We anticipate that the rich granular content will assist both BV data producers and users to identify, understand and assess the various factors that impact upon veracity and quality of BV data. The online mind map provides a flexible tool to enable this. Rapid access to relevant content can be attained using the on screen search facility, and functionality is available via a menu button to enable user defined printing of sections of the map for offline usage. Future developments will need to address BV data delivery by other methods that use big data/data mining approaches of the type recently reported by Marqués-García et al. [24].

A standard for reporting biological variation data studies

Content and format of the mind map has parallels with the materials used for delivery of the Standards for Reporting Diagnostic Accuracy (STARD) initiative, which describes what is expected from authors in developing sufficiently informative study reports of diagnostics [25]. STARD provides a consensus guideline and checklist describing what is expected from authors in developing sufficiently informative study reports. Editorial policy of many peer-reviewed journals publishing such studies requires that authors self-assess their work and confirm compliance with STARD prior to submission of draft manuscripts for review and publication. That approach has delivered quality improvement impacts in that reporting context [26]. Adoption of an STARD like approach to assessing BV studies we believe will similarly promote consistency and comparability of BV data reports making it easier to assess the veracity and quality of both old and new reports of BV. Additionally, application of identified best practice will deliver new BV reports that should achieve a grade of “A” on application of BIVAC. Indeed a recent review of studies of BV of troponins indicates an improvement in the quality of BV studies published that state compliance with BIVAC [18]. This will drive up the quality of data assimilated into the EFLM BV database and enable inclusion of a higher proportion of data from grade “A” studies in meta-analyses [11]. Other benefits to be delivered will accrue because of improvements in the usability and transportability of BV data as reference data.

Adopting the position that BV data are reference data imposes requirements for study delivery and reporting that are addressed in the mind map. Transportability of the data will be improved by ensuring that key metadata identified in the embedded EFLM checklists are collated as components of an essential MDS, recorded, and reported as required for reference data [5, 10]. This will lead to improvement in the quality of BV data which will translate into user confidence; enabling decision support in the contexts of utility and applicability of published data in the user’s clinical practice across different healthcare systems applied to varying populations.

The mind map we thus believe delivers a guideline with embedded checklists enabling compliance with an emergent Standard for Reporting BV Data Studies (STARBIV). If editorial boards of journals were to mandate STARBIV compliance through an agreed process, similar to STARD, it could enhance quality and address concerns raised at the EFLM first strategic conference [3]. The process may involve the use of a high-level checklist to provide evidence of compliance (e.g. Supplementary Material, Table S1) with the supporting detail and guidance for compliance provided through links to the mind map to enable items to be addressed. STARBIV-compliant authors ensure quality by identifying, recording, and reporting key metadata from the mind map. This process yields checklist-compliant studies with sound statistical analysis, providing BV reference data reports open to critical appraisal by users. Online tools are under development that will enable assessment and recording of compliance with STARBIV high level checklist and embedded BIVAC (www.starbiv.eu) [27].

Adoption of the proposed STARBIV approach delivers a necessary framework that will enable an increased awareness and understanding of the factors impacting upon quality and confidence issues around BV data delivering many potential benefits (Supplementary Material, Table S2). The mind map provides an open access tool considered by the EFLM BV groups to have content indicative of the current state of the art regarding the prospective experimental approach to delivery and reporting of BV studies. Further iterations of the map will be required in response to the future development of new approaches to delivery of BV data (e.g., big data/data mining approaches) [24].

Summary

In summary, an online resource for design, production, delivery and reporting of BV studies has been developed by international expert groups established by the EFLM. It takes the form of an online interactive mind map [20], that enables users to both plan and check that studies delivering BV data by the prospective experimental approach are compliant with a working standard, reporting a considered minimum data set and are compliant with the embedded BIVAC. The approach to its use parallels that of the STARD initiative. In the absence of formally agreed standards, use of STARBIV will enable convergence of the reports of BV studies upon a standard and thereby address quality and confidence issues around BV data and their use and application as reference data.

Outlook

There is a need for standards for generation and reporting biological variation (BV) reference data. The absence of agreed-upon standards compromises the utility of BV data due to deficiencies in study quality and reporting. The first European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Strategic Conference in 2014 emphasized the importance of assessing the relevance and validity of BV data before application.

The innovative solution reported here provides an online interactive mind map to serve as a guideline for planning, performing, and reporting BV studies. This tool, designed for application to both historical and future studies, aims to enhance data quality, consistency, and comparability. Its benefits extend to aiding decision-making and supporting health informatics. Dissemination through the EFLM BV database website ensures accessibility, fostering a culture of compliance with reporting standards.

Looking ahead, the mind map approach, akin to the STARD initiative for diagnostic accuracy studies, holds promise in improving the reporting quality of BV studies. Embedding compliance within editorial policies can lead to a standardized approach (STARBIV), offering numerous benefits, including increased user confidence, decision support, and critical appraisal of BV data. However, ongoing refinement will be necessary to adapt to emerging methodologies, such as big data approaches. Overall, the concerted efforts of the EFLM groups signal a positive trajectory towards enhancing the validity and applicability of BV data in clinical practice. The current iteration of STARBIV, presented online, provides an expert opinion in a format that can respond dynamically to relevant developments delivered by other organisations and groups. Alternative delivery formats may need to be developed following discussions with other interested parties around implementation, accessibility and version control.