The final part of the CRESS trilogy – how to evaluate the quality of stability studies

Aims

We describe a tool to evaluate the quality of stability studies, aiming to standardize and harmonize the grading of such studies for improved comparison and meta-analysis as a starting point, contributing to the development of a database of graded stability studies.

Background

Laboratory medicine plays a critical role in a significant proportion of the clinical decisions regarding patient management [1]. It is therefore of the utmost importance that published laboratory results are reliable and accurate. However, as surmised by the phrase ‘Garbage in, garbage out’, the quality of the results can only be as good as the quality the samples delivered to the laboratory. Laboratory medicine has established excellent procedures to ensure quality in the analytical phase during the last century. In addition, over the last few decades, significant progress has been made in establishing awareness of quality in the extra analytical steps factoring in all aspects of the total testing process, from the clinical decision to take a sample to the interpretation of laboratory results [2], [3], [4]. In 2012 the laboratory standard ISO15189 [5], introduced a requirement for laboratories to improve and maintain the quality in the preanalytical phase which has led to an increase in interest and in publications on this topic. To avoid poor quality samples with subsequent poor quality results, it is essential that laboratories are able to determine the quality of the sample and the associated quality of the analytes which are to be tested. In order to do this, laboratories need to know the stability limits for every analyte in a variety of different storage conditions and matrices. This knowledge allows laboratories to define the analyte-specific acceptable delay between sample collection and centrifugation/analysis necessary to guarantee results and interpretation of sufficient quality to answer the question asked by the requesting clinician.

When reviewing published stability studies, a certain level of redundancy of tested analytes as well as a heterogeneity in applied methodology and results can be found, making it difficult to transfer stability criteria from such studies onto the local setting. Our Working Group for the Preanalytical Phase (WG-PRE) of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) therefore established a 4-step plan, aiming to standardize and harmonize stabilities data for better comparison and meta-analysis (Figure 1). The first step was to produce a checklist to guide reporting of stability studies which was published in 2020 [6] and the second step, published in 2023, was to produce a how to guide for performing stability studies [7]. The final 2 steps involve a mechanism to evaluate published stability data and to then incorporate this into a central free-to-use resource.

Figure 1:

The EFLM WG-PRE 4-step plan, aiming to standardize and harmonize stabilities data for better comparison and meta-analysis.

What is stability?

Defining stability in the context of a biomarker can be difficult. The international Vocabulary in Metrology (VIM) [8] defines stability only for analytical instruments as a ‘property of a measuring instrument, whereby its metrological properties remain constant in time’. This statement is applicable to analytes by slight rephrasing to: ‘a property of a measurand, whereby its metrological properties remain constant in time’. Furthermore, this definition could be extended, stating analyte stability as the timeframe during which the measurand is being stable under defined conditions, or by stating that a particular analyte changes less than a defined criteria (percentage) over a defined period under defined conditions [6]. In other words, is it defining a limit past which the sample should not be used for the respective analyte testing. Ideally, the result deterioration is expressed by a regression equation, making it possible to calculate individual acceptance limits for the local setting. Either way the main goal should be to ensure that laboratory professionals can add the highest value possible to a laboratory report to ensure laboratory results of sufficient quality for patient’s safety.

Variables affecting stability

There are many contributing variables that may affect the stability of a sample and it is important that these are all understood and accounted for when determining whether the sample is of sufficient quality for analysis. This range of variables needs to be controlled and documented when performing stability studies.

Apart from the time between collection and analyte measurement, variables include the sample type (e.g. blood, serum, plasma, cerebrospinal fluid, saliva or other bodily fluid) or the collection tube type (e.g. EDTA, Citrate, Heparin, etc.) which can vary among manufacturers for the same tube. Sample mixing with the additive may also have an impact on the quality, as inadequate mixing can lead to poor stabilisation of the sample by additives and over vigorous mixing can cause cell damage. Additionally, the tube filling volume may contribute to poor sample quality in terms of potential dilution effects or inadequate additive effect. Another major variable affecting sample stability is the temperatures the sample is exposed to during transport, centrifugation or storage. The centrifugal force during plasma/serum separation is another important factor that can influence the sample quality, e.g. if separation is incomplete. Additional variables potentially affecting sample stability and quality include light exposure if the analyte is light labile, sample evaporation and specifics of the laboratory instrumentation and reagent reaction kinetics [9].

Sample stability may vary between individuals as some have cells which leak more readily than others in vitro (e.g. potassium), and others have different enzyme activities, cell counts, protein concentration, all of which potentially influences the analyte’s stability. The metabolism in the sample may also lead to the production of the analyte of interest and increase its concentration.

Methods

In order to evaluate sample stability studies, we used the EFLM Checklist for Reporting Stability Studies (CRESS) as a foundation to structure 20 sections against which stability publications would be assessed [6]. Each section then had 4 levels of quality grading, ranging from A for “Best in Class” to D for a fail in that section (Table 1). Initial requirements for the categories A to D were produced following expert discussion among EFLM WG-PRE members. Six published stability studies from the last 10 years were then circulated among these members to test the methodology [10], [11], [12], [13], [14], [15]. The manuscripts were sent to members of the EFLM WG-PRE alongside the scoring criteria. Category grading was then refined following these pilot results and the feedback from members of the group.

The scores of A to D for each criteria was then converted to a numerical value (4 for an A down to 1 for a D) and each of the 20 sections was weighted ranging from 0.5 to 3, depending on the importance assigned to each section based on the expert opinion of the EFLM WG-PRE. For example, a low weight was applied to questions on funding and ethics as these factors play a minor role in evaluating sample stability, compared to details about the study population, samples used and the analytical method which are some of the highest weighted criteria (Table 2). Unweighted results were also calculated to demonstrate the merit of using weighted scores.

For weighted scores the maximum score possible was calculated by multiplying the maximum score for each category from the CRESS checklist by the criteria weighting and adding them all together. A final score of A to D was then calculated, based on a final score being a set percentage of the maximum achievable. Percentages used were 80, 60 and 40 % following expert discussion.

For the test stability studies a weighted score for each category from the CRESS checklist stability study was calculated by multiplying the average score from the experts for each category from the CRESS checklist by the weighting and adding them together. For both weighted and unweighted a final score of A to D was then calculated, based on the final score being a set percentage of the maximum achievable as above.

Results

Table 1 presents a standardized model utilizing the CRESS checklist strategy for evaluating the quality of stability manuscripts. This framework covers typical sections and important parameters commonly found in stability manuscripts, encompassing 20 distinct evaluation criteria. Defined by consensus, each item includes a specific question and the definition of four levels of quality, ranging from best in class (A) to Fail (D), thereby providing a score from 4 to 1. Additionally, weighted scores based on the importance of each criterion were also determined and are detailed in Table 2. To validate this strategy, six stability manuscripts were evaluated by members of the WG-PRE.

There were varying levels of heterogeneity between both assessors and manuscripts but this would be expected to some extent due to the remaining subjective nature of the process and the piloting phase of the evaluation process. However, even at this stage, the majority of scores differed only by a single category. The observed grading variations were lower in the higher scoring studies, compared to the lower scoring manuscripts.

The final scoring of the manuscripts ranged from A to C and the ranking itself for the chosen manuscripts didn’t display any difference. Table 3 displays the scores for weighted and unweighted results alongside the percentage scores. However, the percentage scores did differ ranging from 52 to 81 % for the non-weighted scores and 51–84 % for the weighted scores. This emphasises that weighting is of importance to demonstrate the differences between stability studies.

Discussion

Patients have the expectation that results of the ordered tests are accurate and reflect the true state of their health. Laboratory medicine professionals are aware that there are various factors that can affect the accuracy of results, of which most are to be found in the preanalytical phase, affecting sample stability, among others. It is critical that laboratories are aware of the impact of any delays and conditions samples are exposed to during this delay. For many laboratory specialists, consulting the literature is the first place to search for stability data. The current problem is that the measurands evaluated in the published studies overlap and the methodologies differ, making it hard or impossible to retrieve the desired information or to apply the results onto the local settings. To that end, as discussed above, the EFLM WG-PRE have put together a package to guide and standardize the conduct and evaluation of stability studies. The final part of this package, as detailed in this manuscript, was to produce a standardized evaluation process to assess the quality of stability studies, following the sections detailed in the CRESS checklist. This guide will allow laboratorians to identify studies that have followed a standard methodology and will contain all the information they require to make an informed decision. Of the six manuscripts evaluated, 2, 3 and 2 scored an A, a B and a C, respectively. There were no scores of D (fail) which perhaps reflects the quality of the peer review process in eliminating the poorest quality manuscripts. It is also worth noting that although this process identifies high quality studies, manuscripts that score low are not necessarily of poor result quality, they just do not adhere to all of the CRESS checklist and therefore will not have all the information now stated to be required in a good quality manuscript.

The quality ranking (A–D) did not vary whether weighted or non-weighted scoring was used, but there was a difference in the percentage scores. This indicates that for a larger volume of manuscripts a difference in categorisation would occur and would differentiate between best in class papers and lower quality manuscripts.

The aim of providing this package of guidance on sample stability is to encourage people to not only perform high quality stability studies but to report their results in high quality manuscripts and to share their raw data. As a fourth step, the EFLM WG-PRE is currently working on a database which will contain published stability data in a structured way and to which local (unpublished) stability data can be uploaded. The vision is that the relevant manuscript is linked to this data and subsequently representatives of the EFLM WG-PRE will be able to perform meta-analysis and assess the manuscript to assign a quality score of A–D to it. That way a laboratory medicine professionals can search for stability information on a particular analyte and filter for the storage conditions in question and be able to apply the data to their own setting.

The big limitation of the proposed evaluation process is the subjectivity of some aspects of the process. We saw differences in the ranking of sections of the different manuscripts assessed and while this was useful in allowing us to eliminate areas that could be misclassified or where the degree of overlap was too great, it will always be a point of variation in this type of process. That said it is very unlikely to impact on the final outcome of any classification due to the wide ranges. It could also be mitigated by having 2 verifiers for all studies.

It is the hope of the EFLM WG-PRE that this final paper in the trilogy in combination with the other manuscripts will lead to an improvement in both the way that stability studies are performed, as well as in the way that they are reported and evaluated. We believe that adhering to the proposed guidance will result in improvements in both quality and standardisation of stability studies.