Performance evaluation of cobas pure integrated solutions at multiple sites in Europe and Asia

Daily QC

At all five sites, the Roche-manufactured and BioRad QC materials were measured using selected applications (ISE, clinical chemistry [including enzyme, substrate, and specific protein], and immunochemistry) at two analyte concentration levels over 14–16 weeks. Analyte recovery per QC was monitored; the recommended analyte recovery range was within ±2 standard deviation (SD) of the assigned target value, with the exception of chloride and urea, where the range was extended to ±3 SD (due to electrode exchange and adjustment of the rinse station).

Precision

At four sites (Ludwigsburg, Seoul, Visp, and Wroclaw), precision experiments based on the Clinical and Laboratory Standards Institute EP05-A3 protocol were conducted, with two runs per day over a 21-day period for 34 selected applications (ISE: 3 analytes/6 applications [serum/plasma and urine]; clinical chemistry: 10 analytes/15 applications [serum/plasma, urine]; immunochemistry: 13 analytes/13 applications [serum/plasma]) that represent a typical routine panel of assays and cover differences in assay protocols (such as different viscosity of reagents or pipetting volumes). Pooled control materials (Roche-manufactured and BioRad) were used to minimize the potential influence of sample handling between sites and operators; two control samples were measured per application (low and high concentration).

Coefficients of variation (CVs) for repeatability (within-run precision), intermediate precision (within-laboratory precision), and reproducibility (across-laboratory precision) were calculated and compared with study-specific acceptance criteria. The study-specific acceptance criteria are detailed in the supplementary materials (Supplementary Table 2).

System functionality

At four sites (Heidelberg, Ludwigsburg, Visp, and Wroclaw), routine simulation imprecision experiments were conducted to test the interaction of hardware, software, assays, and samples, and evaluate recovery and precision under various stressed routine-like conditions, including provocations.

These experiments comprised a reference part, where a defined number of aliquots per assay were processed consecutively (within-run batch), and a random part, where all assays and materials were requested in a randomized order (random access mode). For selected experimental runs (two per site), operators challenged the system functionality by applying provocations within the random part of the experiment. The provocations performed represented typical user scenarios applicable to daily routine laboratory work, such as sample short, short turnaround time requests, reagent changeover, reagent short, sample clots, etc. CVs and median CVs of the reference part and random part were calculated and compared.

Reliability

At all five sites, system reliability – measured by the percentage of analysis runs completed without system-related interruption – was assessed.

Routine simulation method comparison

At four sites (Heidelberg, Ludwigsburg, Seoul, and Visp), routine simulation method comparison experiments using residual routine samples were conducted to evaluate overall functionality under routine-like conditions and comparability of cobas pure integrated solutions with respective routine analyzers: Beckman Coulter AU5800 Clinical Chemistry Analyzer (Seoul), Abbott Alinity i (Seoul), Siemens ADVIA Centaur XPT Immunoassay System (Seoul), and Roche COBAS INTEGRA^® 400 plus (Roche Diagnostics GmbH, Mannheim, Germany; Visp), cobas e 411 (Visp), cobas pro (cobas c 503 and e 801; Ludwigsburg), and cobas 8000 (cobas c 702 and e 801; Ludwigsburg and Heidelberg). In total, 47 analytes with 53 applications covering ISE, clinical chemistry, and immunochemistry were assessed. To introduce variability within and across sites, testing was performed for up to 10 days at each site. The analyte concentration ranges covered per assay reflected those of the individual sites’ routine samples at that time. The residual routine samples were independent from their specific matrices, including type of serum/plasma collection tube and type of urine specimen (e.g. spot samples and samples collected over 24 h). Pre-collected or spiked samples were not included.

Routine workloads (100−300 samples) were replicated and reprocessed, in part or total, on cobas pure integrated solutions using WinCAEv – a Code of Federal Regulations Title 21 Part 11 compliant electronic data capture software developed and validated for Roche-sponsored studies [2] – to capture routine analyzer requests via an electronic file from the Laboratory Information Systems (LIS, Host). This experimental design enabled laboratory personnel to assess overall system functionality, including sample and reagent handling, data flagging and alarm generation, and flow of workload processing [3], as well as reliability under different laboratory conditions. Samples were stored for 1−3 days at 4 °C between the initial routine measurement and the subsequent measurement on cobas pure integrated solutions.

Results per sample were compared with those previously generated during routine operation; data pairs were automatically evaluated using Passing–Bablok regression [4], and slopes, intercepts, and correlations for method comparisons were calculated and compared with study-specific acceptance criteria.

Workflow

At four sites (Heidelberg, Seoul, Visp, and Wroclaw), operational performance was evaluated by recording time to first result (from sample loading to first result), time to last result (from sample loading to last result of the entire workload), and output (results generated per hour). The results considered eight different workflows, with variable test request patterns and different clinical chemistry/immunochemistry test ratios. All runs were defined and performed as routine simulations to verify the throughput of cobas pure integrated solutions under user intended conditions, using pooled Roche-manufactured QC materials as samples. By loading the samples onto the analyzer as the starting point, the same conditions were ensured for all systems.

Inter-laboratory survey (ring trial)

At four sites (Heidelberg, Ludwigsburg, Visp, and Wroclaw), result accuracy for 48 selected applications (ISE: 3 analytes/3 applications; clinical chemistry: 26 analytes/26 applications; immunochemistry: 19 analytes/19 applications) was validated in a ring trial – a validation study that evaluates a diagnostic method using identical samples in several laboratories.

Ring trial samples included 16 commercial serum samples and two Roche-manufactured QC samples (low and high concentration anti-hepatitis A virus immunoglobulin M), which were measured in triplicate over 3 consecutive days. Commercial samples are often used for ring trials due to their availability for comparison measurements. The median percentage difference between site-specific recovery and overall recovery was calculated (maximum 10 % deviation permitted).

Practicability and usability

At the five sites, user feedback was captured at the end of the study via a 200-item questionnaire comprising 17 question categories: installation environment, location of components, operator training, user documentation and support, product design and labelling, daily workflow, reagent handling, workflow, timing/productivity, data processing environment, quality assurance, calibration, QC, maintenance, troubleshooting, versatility, and consolidation. Questions were graded on a 10-point scale, from 1 “does not meet expectations” to 10 “exceeds expectations”. For each question, a mean score was calculated across the five sites; an approximate score of 1–3.3 = “does not meet expectations”, >3.3−6.6 = “meets expectations”, and >6.6–10 = “exceeds expectations”. Users could provide qualitative feedback to report advantages of the analyzer, incidents of malfunction, and suggest possible improvements.

Formative usability testing was performed according to the ISO standard IEC 62366-1:2015. Trained operators completed tasks relying on their training and training material alone, while observed by trained test conductors from Roche Diagnostics.

Statistical analysis

Data were captured, analyzed, and stored by WinCAEv; data captured offline were verified using review by Roche Diagnostics.

Daily QC

In total, 25,826 QC results were included in the analysis. Analyte recovery for all assays was within ±2 SD of the preliminary assigned target value; except for chloride and urea, which showed QC recoveries within ±3 SD. The chloride electrode was exchanged and optimized adjustments were applied to the rinse station used for urea. Following this, chloride and urea showed recoveries within ±2 SD.

Precision

In total, 612 CVs were calculated, including repeatability (n=272), intermediate precision (n=272), and reproducibility (n=68) (Figure 1; Table 1). For repeatability, 34/48 CVs were <1 % for ISE assays, 116/120 CVs were ≤2 % for clinical chemistry assays, and 90/104 were <2.5 % for immunochemistry assays. For intermediate precision, 47/48 CVs were ≤2 % for ISE assays, 114/120 CVs were ≤2 % for clinical chemistry assays, and 77/104 were <2.5 % for immunochemistry assays. For reproducibility, 11/12 CVs were ≤3 % for ISE assays, 27/30 were ≤3 % for clinical chemistry assays, and 14/26 were <2.5 % for immunochemistry assays.

Figure 1:

Distribution of (A) repeatability, (B) intermediate precision, and (C) reproducibility CVs for QC low and high concentration samples for ISE, clinical chemistry, and Elecsys immunochemistry assays^a. ^aCombined dataset, including serum and urine matrices. ^bTarget CVs for repeatability: ISE <1 %; enzyme and substrate assays <2 %; homogeneous immunoassays and urinalysis <3 %; heterogeneous immunoassays <5 %. ^cTarget CVs for intermediate precision: ISE <2 %; enzyme and substrate assays <3 %; homogeneous immunoassays and urinalysis <4 %; heterogeneous immunoassays <8 %. ^dTarget CVs for reproducibility: ISE <3 %; clinical chemistry, homogeneous immunoassays, and urinalysis <5 %; heterogeneous immunoassays <10 %. ^eThe reproducibility analysis yielded two CVs per analyte (QC low and high), therefore, there is no variability (minimum, maximum, or mean). ALT, alanine aminotransferase; AST, aspartate aminotransferase; B12, vitamin B12; Ca, calcium; CA 15-3, cancer antigen 15-3; CEA, carcinoembryonic antigen; CHOL, cholesterol; Cl, chloride; CREA, creatinine; CRP, C-reactive protein; CV, coefficient of variation; E, estradiol; Fe, iron; FOL, folate; FT4, free thyroid hormone thyroxine; GLUC, glucose; HAV, hepatitis A virus; HCG+β, human chorionic gonadotropin; IgM, immunoglobulin M; ISE, ion selective electrolyte; K, potassium; Na, sodium; PHOS, phosphate; pro-BNP, pro-B-type natriuretic peptide; QC, quality control; TESTO, testosterone; TNT-hs, troponin T-high sensitive; TP, total protein; TSH, thyroid stimulating hormone; TSHR, thyroid stimulating hormone receptor.

System functionality

For the routine simulation imprecision experiments, 18 runs with 604 CV pairs were included in the analysis. All introduced provocations were handled as expected by cobas pure integrated solutions. All runs including provocations are shown in Table 2.

Most reference and random part CVs were ≤2 % (reference part: 94 %; random part: 93 %) as shown in Figure 2; median reference and random part CVs were 0.58 and 0.74 %, respectively. cobas pure integrated solutions consistently demonstrated very good precision results in batch and random access mode over an experiment run time of ∼4 h.

Figure 2:

Distribution of (A) reference part and (B) random part CVs for cobas pure integrated solutions. CV, coefficient of variation.

Reliability

The overall system reliability of cobas pure integrated solutions was 99 %. In total, 271/288 runs were completed without interruption across all five sites during the 4-month study period. Only three interruptions were system related, the additional 14 interruptions were due to human error and, therefore, excluded from the analysis.

Routine simulation method comparison

Four of the sites (Heidelberg, Ludwigsburg, Seoul, and Visp) participated in the comparison of routine results with those generated on cobas pure integrated solutions from the same routine samples. More than 55,000 data pairs were generated from 7,254 routine samples.

In total, 218 method comparisons generated a median slope of 1.00, median bias at the medical decision point of −0.1 %, and median Pearson’s r of 0.998. The median bias at the medical decision point was 0.0 and −0.2 % for clinical chemistry and immunochemistry assays, respectively. For ISE, all comparisons were within ±5 % bias at the medical decision point. Overall, 168/187 method comparisons vs. Roche methods met the Passing–Bablok slope and intercept criteria for standard batch method comparisons (Table 3; Supplementary Table 3). Standard batch method comparison is where a number of samples are run in sequence, i.e., they are not randomized. This reduces the time between measurements taken on the investigational device and measurements taken on the reference (comparator) device and also limits potential interactions between different samples and reagents. In contrast, the routine simulation method comparison mimics the situation in real laboratories, where samples are tested in random access mode and multiple applications are used concurrently. Depending on the availability of samples, the time span between testing can be 1–3 days for the routine simulation method comparison.

These results were obtained even though the four sites used different reagent and calibrator lots for routine testing and there were delays of up to 3 days between measuring samples on cobas pure integrated solutions following routine analysis.

Workflow

Summaries of the workflow metrics for the eight different workloads analyzed are shown in Tables 4 and 5. The time to first result was as expected (0.5–2 min for ISE results and 20–21 min for the immunochemistry assays), and time to last result was highly dependent on workload (a rise in requests for immunochemistry testing from 14 to 17 % increased the time to last result from 45 to 55 min, while the absolute number of requests stayed the same). The introduction of two short turnaround samples per 200 samples did not significantly change the time to last result. However, the percentage of pre-dilution or 27-min assays in specific immunochemistry blood screening workloads did increase time to last result; these assays require additional pipetting steps, thus, affecting time to last result.

Output (results generated per hour) correlated with workload and immunochemistry/clinical chemistry ratio. Output for blood screening workflows correlated with the use of pre-dilution or 27-min assays. Output of serum work area workloads was not affected by an increase in immunochemistry/clinical chemistry ratio (from 14 to 17 %), nor the introduction of two short turnaround samples.

Inter-laboratory survey (ring trial)

For the 48 applications included in the result accuracy analysis using ring trial samples, 80.4 % of median recoveries per assay and site were within ±2 % of the median per assay for all sites, and 99.7 % of median recoveries per assay and site were within ±10 % of the median per assay for all sites. Only folate exhibited increased imprecision due to the complex assay format (Figure 3).

Figure 3:

Inter-laboratory survey (ring trial): Median within-lab recovery vs. median across all sites for (A) clinical chemistry and (B) Elecsys immunochemistry assays. The dashed lines represent median recoveries within ±2 % of the median per assay for all sites. ALB, albumin; ALP, alkaline phosphatase; ALT, alanine aminotransferase; APO, apolipoprotein A1; AST, aspartate aminotransferase; B12, vitamin B12; BILD, bilirubin direct; BILT, bilirubin total; Ca, calcium; CA 15-3, cancer antigen 15-3; CEA, carcinoembryonic antigen; CHE, cholinesterase; CHOL, cholesterol; CK, creatine; Cl, chloride; CREA, creatinine; CRP, C-reactive protein; E, estradiol; Fe, iron; FERR, ferritin; FOL, folate; FSH, follicle stimulating hormone; FT3, free thyroid hormone triiodothyronine; FT4, free thyroid hormone thyroxine; GGT, γ-glutamyltransferase; GLUC, glucose; HAV, hepatitis A virus; HCG+β, human chorionic gonadotropin; HDL, high density lipoprotein cholesterol; IgM, immunoglobulin M; K, potassium; LDH, diglycosylated luteinizing hormone; LDL, low density lipoprotein cholesterol; LH, luteinizing hormone; LIP, lipase; Mg, magnesium; Na, sodium; PHOS, phosphate; pro-BNP, pro-B-type natriuretic peptide; PROG, progesterone; TESTO, testosterone; TNT-hs, troponin T-high sensitive; TP, total protein; TPSA, total prostate-specific antigen; TRIGL, triglycerides; TSH, thyroid stimulating hormone; TSHR, thyroid stimulating hormone receptor; UA, uric acid.

Practicability and usability

In total, 99 % of questions answered were graded as “meets expectations” or better. With a mean grading of 8.6 and a median grading of 9.0 across all sites, practicability was categorized by all sites combined as “exceed expectations” (Supplementary Figure 1). Usability testing did not reveal severe safety issues for operators or patient results; thus, no design change was required.