“Good samples make good assays” – the problem of sourcing clinical samples for a standardization project

Introduction

In vitro diagnostic (IVD) testing on clinical specimens is for the greater part serum- or plasma-based. For a test result to truly reflect the concentration of an analyte in the blood circulation, it is important that the testing was done on a sample with a good preanalytic status. This requires phlebotomy with minimal disturbance of the in vivo situation, and processing of the specimen until analysis (clotting, centrifugation, separation of serum from the clot, storage etc.) under conditions maintaining sample integrity [1, 2].

For obvious reasons, native samples also form the cornerstone in all activities involved with the development of an IVD assay. Indeed, they are essential tools for the initial discovery of new biomarkers and evidence-based documentation of their clinical utility. When the manufacturers decide to initiate the development of an assay for IVD testing of the marker, they need clinical samples to define the assay design and optimize the test conditions. After preliminary evaluation of the assay’s performance attributes by use of “surrogate” materials (e.g., control or spiked materials), manufacturers again need native samples to conclusively prove that the quality is commensurate with the intended use. The same applies for the assessment of diagnostic accuracy and specificity, and for the establishment of reference intervals or cut-offs for medical decision-making. In addition, IVD manufacturers consider patient samples of the utmost value in the process of establishing/validating metrological traceability of their assays to a higher order calibrator and measurement procedure. Indeed, it is generally recognized that their use in the calibration hierarchy, more particularly at the level of the calibrator of the manufacturer’s selected or standing measurement procedure, is the best practice to circumvent commutability issues potentially encountered with processed or artificial calibrators [3–5]. However, it should be noted that even clinical samples may be non-commutable with an assay, due to a lack of specificity of the latter for interfering substances. When indeed such samples are included in a commutability assessment and measured in parallel with patient samples without interfering substances, it might be that their measurement results are not within the prediction interval containing 95% of the results. Therefore, the Clinical and Laboratory Standards Institute (CLSI) C53-A recommends to exclude “problematic” samples from the panel of native specimens used in commutability assessment, which is per se not a problem since most manufacturers declare the potential interference with their assays [6]. In view of this role played by patient samples in the context of metrological traceability, relevant panels of high quality samples are also crucial in IFCC standardization projects [7]. These typically apply a step-up approach, i.e., start with panels of high volume single donations (200 mL of serum/plasma), then proceed to relevant clinical samples of medium-sized volume (15–30 mL).

For all aforementioned applications, the quality of the samples goes beyond blood collection and processing. Other influencing factors relate, among others, to the compliance of the samples with in- and exclusion criteria, the representativeness of the donors for the application (e.g., donors meeting tightly defined criteria for health or diseased status and clinical prognosis), and the availability of in-depth patient information (age, gender, ethnicity, relevant medical history, co-morbidities, current and past medication, etc.).

In spite of the described pivotal role of high quality clinical patient samples, all involved parties experience sourcing of the specimens particularly challenging [8]. The easiest accessible source, i.e., residual samples from completed laboratory testing, is mostly of not much help, due to the low volumes (0.5–3 mL) available, and local ethical rules which prevent laboratories from providing key patient information. Instead, dedicated supply sources have to be used, either commercial vendors or individual hospitals. The advantage of the former is that they usually have an established network of collaborating centers, so that they can deliver in a timely way and accommodate varying sample specifications. In contrast, individual hospitals mostly require a very long set-up phase to find volunteer clinicians with sufficient personnel resources to ensure that they can meet all key factors and requirements associated with sample procurement. A major obstacle is also the time-consuming procedure to achieve approval by local Ethical Committees or institutional review boards.

Below, we describe the “torturous” experience in this regard of the IFCC Working Group for Standardization of Thyroid Function tests (WG-STFT, subsequently transformed into a Committee).

A first surprise in the phase I study

In the phase I method comparison study, the WG-STFT used a panel of 40 single donation specimens from apparently healthy individuals. Blood collection and serum generation were done according to the CLSI C37-A protocol, except that the serum was not filtered nor pooled [9]. In short, this protocol comes down to: 1) collection of individual donor units of whole blood in a plastic blood-pack put in an ice-water bath; 2) separation of plasma from cells by centrifugation at 4°C (within 5 min of unit collection, before unit clots); 3) aseptic expression of plasma into a sterile borosilicate glass; 4) clotting at room temperature until the clot retracts fully from the serum (clotting time 3 to 4 h); 5) centrifugation of each unit and aseptic transfer of the clear serum layer to a clean and sterile borosilicate glass bottle, with testing for absence of clotting elements; if positive, steps 4) to 5) are repeated for one additional cycle but if positive result persists, the unit is rejected; 6) subsequently, the desired volume aliquots of serum are aseptically transferred to final containers, which are capped, sealed, and frozen (while kept in an upright position and in direct contact with the metal shelf of the freezer). Final storage of the serum is done at –70°C. Note that all steps from blood collection until freezing are completed within 24 h. In the method comparison for free and total triiodothyronine (FT3/TT3) measurement, some assays showed on several sera a very bad quality of performance in terms of excessive scatter of results and outliers after recalibration in comparison to the reference measurement procedure [10, 11]. Strangely enough, parallel analysis of other thyroid hormones [free and total thyroxine (FT4/TT4) and thyrotropin (TSH)] in the same samples did not reveal performance issues. The two FT3 assays of concern gave results that strongly deviated in a positive direction (ranging from 128% to 159%) in four samples in comparison to the other samples, which of course heavily affected the correlation coefficient (r²<0.30; after recalibration). Moreover, three TT3 assays produced grossly increased (by 120% to 174%) results for 10 samples, giving an r²<0.60. Note the performance of one assay was problematic for both FT3 and TT3. In an attempt to explain the peculiar observations, we re-measured the sample aliquots used for the assays of concern with our TT3 reference measurement procedure, however, we did not find increased concentrations. From this experiment we could exclude impaired sample integrity (e.g., wrong labeling, contamination, bad storage conditions, etc.) as the cause. We also could not find a direct association with the concentrations of other biochemical analytes measured to characterize the samples. Finally, in consultation with the WG, we investigated whether the observed effects might be associated with the C37-A clotting procedure, originally developed and validated for cholesterol measurement. The guideline recommends clotting at room temperature and allows that 3–4 h may be required until the clot retracts fully from the serum. Our concern was that this clotting time is much longer than needed for regularly collected blood samples. Therefore, we thought that maybe the peculiar observations in our study indicated the need to verify the validity of the C37-A protocol for other analytes. In this regard, we focused on accomplishing complete clotting in an accelerated and consistent manner by adding human thrombin to the plasma of 20 blood collections [2 U/mL plasma, Sigma (St. Louis, USA)]. We then measured the specimens in replicate within one run with both an “affected” and “unaffected” assay. Figure 1 presents the results by the two assays on the thrombin and C37-A clotted serum samples in a percentage difference plot. The experiment demonstrates in general a beneficial effect from the addition of thrombin on the performance of the affected assay: for FT3, the scatter and number of outliers became less pronounced, for TT3 they even disappeared.

Figure 1

Difference plots for FT3 (A) and TT3 (B) measurement results generated by an “affected” and “unaffected” assay.

Results for analysis of the C37-A clotted samples of the phase I method comparison are represented by open triangles; those for the thrombin clotted specimens by closed diamonds.

The unsolved mystery in the phase II study

In the phase II study of the WG-STFT, we were faced with a peculiar observation in the TSH method comparison, more in particular in the high measurement range [12]. In a scatter plot of the results of all assays vs. the all-method trimmed mean, we observed that the between-assay variation was much higher in the TSH concentration range >12 mIU/L, compared to the euthyroid range. To further investigate this observation, we made for each individual assay a ratio plot of its results to the trimmed mean and found that seven out of 10 assays had a moderately to strongly biased (negatively or positively) performance on the samples with hypothyroid vs. euthyroid concentrations. Different hypotheses were made in the WG regarding the deviating performance of some of the assays: Differential capture by the antibodies of the glycosylation forms typical for TSH in hypothyroidism [13], inclusion of samples from patients under recombinant TSH therapy [14], a difference in matrix of the samples (=commutability issue), or an aberrant processing of the concerned samples (in terms of clotting, time-to-clot, storage before centrifugation, etc.). We first investigated the last hypothesis, in particular because the samples from euthyroid donors had been collected/processed according to the C37-A protocol, whereas those from non-euthyroid patients had been obtained from a commercial source. The supplier confirmed that blood drawing had been done with regular red cap coagulation tubes (without additives) and in experienced collection centers included in his network. On the basis of this information, we had no reason to doubt the quality of the clinical samples. In particular, triggered by the hypothesis of a difference among assays in recognition of the TSH glycosylation forms, we investigated whether the observed difference could be reproduced. However, to exclude the influence of the collection side and procedure, we did the experiment with 30 samples from hypothyroid and another 30 from euthyroid patients, all obtained from the same hospital and according to the local standard operating procedure (in short: blood collection in red cap tubes (with complete filling of the tube), immediate transport of the tube to the laboratory, clotting during 30 min, centrifugation during 10 min followed by immediate transfer of the serum retracted from the clot to another vial). All samples were analyzed in duplicate with two assays, randomized, and in the same run. One of the assays (Y) had shown in the phase II study a clear difference in performance, the other (X) not.

Figure 2A presents the ratio of the results by the two assays (Y over X) in the above experiment in comparison to that in the phase II study. Clearly, in the most recent comparison, assay Y behaved similarly on the hypo- and euthyroid samples. We thus provisionally concluded that the effect must have been caused by an (unidentified) matrix problem, most probably associated to the high clinical samples. It remained necessary to investigate the hypothesis regarding the inclusion of samples from patients under recombinant TSH therapy. Indeed, literature gives evidence that assay antibodies may differently capture glycosylation forms specific in recombinant vs. hypothyroid TSH [14]. We did a similar experiment as explained before, now with 30 euthyroid and 30 TSH-treated patients (continuum of concentrations). As shown in Figure 2B, again no difference in performance was observed on the two groups of samples. After this second experiment with two representative assays, we finally concluded that the peculiar observations in the phase II were presumably due to unexplained matrix effects, which left us with an unsolved mystery.

Figure 2

Ratio plots of the TSH measurement results by two assays.

(A) Diamonds refer to the phase II study, whereby the closed diamonds stand for the ratio on euthyroid samples, the open ones on hypothyroid samples; the crosses and stripes refer to the here described experiment (all samples collected in one hospital according to the standard operating procedure): the crosses stand for the ratio on euthyroid samples, the stripes on hypothyroid samples. (B) Closed triangles refer to the ratio on samples from euthyroid individuals, open ones to the ratio on specimens obtained from patients under recombinant TSH therapy.

The long and winding road to clinical samples for the phase III study

For the phase III study, the WG-STFT planned to collect two sets of clinical patient samples: one should comprise 90 samples to be analyzed for FT4, the other 100 samples for TSH. As these two panels should reasonably cover the complete measurement range for both FT4 and TSH, the WG defined the different concentration categories to include, as well as the number of samples per category (see Table 1). In addition, specifications for sample collection and processing were agreed upon, as well as criteria for in- or exclusion of patients. The first criteria relied on the diagnosis hypo-, eu- and hyperthyroidism, whereas the exclusion ones were absence of informed consent, recombinant TSH therapy and diagnosis of severe illness potentially affecting the thyroidal status [i.e., chronic renal failure, liver cirrhosis, advanced (active) malignancy, sepsis, trauma, prolonged fasting or starvation, heart failure, myocardial infarction or psychiatric disorder]. If a patient was undergoing treatment for thyroid dysfunction, he/she could be included provided the type of treatment and the starting time was captured. For all donors, information on co-morbidities and medication should be provided, and of course, also the usual information, such as the donor’s age and gender. From each donor, 15 mL of serum was required, to be aliquoted in 0.5 mL portions and stored at –70°C.

The WG first called upon clinicians from all over the world for sample procurement from the patients they see for evaluation of thyroid disorders. A lot of clinicians declared their interest, however, after some negotiation, the need for approval by the local Ethical Committees or institutional review boards was the major obstacle to the requested support. Therefore, the WG asked a commercial supplier, who declared himself to have the experience and collection capacity to commit for sample recruitment with full compliance in regard to the set specifications and deadline. The price quotation was relatively high, but the IVD manufacturers participating in the project were so eager to enter the phase III study that they agreed to entirely cover the expenses. Unfortunately, on the basis of regular contact with the vendor, it soon became clear to the WG that the enrolment of thyroid diseased patients was extremely difficult and the progress in sample procurement too slow. The deadline was extended, but in spite of additional efforts and resource input by the commercial supplier himself (e.g., he committed to contract new collection centers and hospitals), the new deadline again was not reached. Therefore, two other commercial suppliers were contacted for additional sourcing of samples in the hypo- and hyperthyroid categories. One was finally selected and delivered, but again, reaching the collection completion timescales was extremely challenging. Meanwhile we continued to search for clinicians willing to help. Some new contacts at first looked promising, but again failed, until the chairholder got into contact with two enthusiastic endocrinologists at the annual meeting of the Belgian Thyroid Club, where she was invited speaker. With their help, a number of clinical samples were obtained immediately, but again problems showed up in that the requested volume of 15 mL was not always available. As a result of the described difficulties to obtain clinical samples, the WG was faced with a delay of more than 1 year for the phase III study, which is currently going on.

Conclusions

We hope that with this description of the torturous way to obtain clinical samples for the thyroid standardization project, we draw attention to the importance for similar projects of thinking in a sufficiently early phase about the timely procurement of clinical samples of the required quality. Likewise, we hope that our report may be the incentive for all parties, who believe that “good samples make good assays”, to start a collaborative effort towards establishing a sample procurement infrastructure, based on a solid relationship not only with commercial suppliers, but also with a significant number of committed clinicians in hospitals.