Development and assessment of autoverification system for routine coagulation assays in inpatient and outpatient settings of tertiary care hospital: algorithm performance and impact on laboratory efficiency

Setting

The study was conducted in the central laboratory managed by the Department of Laboratory Medicine, located within a 1,500-bed tertiary referral hospital. Annually, the hospital serves over 1.5 million patients, and the laboratory performs more than 120,000 tests per month. The laboratory is responsible for analyzing routine blood biochemistry, hematology, coagulation, urinalysis, and microscopic examination. The requested tests are performed for inpatients, outpatients, and check-ups with a guaranteed turn-around time of 90 min (1:30 h) for all routine analyses, 90 % of tests needed to meet this criterion as a quality target.

The LIS operated in the facility is the HCLAB system (Sysmex Corporation, Kobe, Japan), which is used for the storage and management of laboratory results and transfer of the data to hospital information systems (HIS). All coagulation tests were measured in plasma collected in 3.2 % sodium citrate (Greiner Bio-One, Vacuette, Kremsmünster, Austria) using CS-2500 analyzer (Sysmex Corporation, Kobe, Japan), automated coagulation analyzer that uses the photo-optical reaction method equipped with five wavelengths of optical fiber (340, 405, 575, 550, and 800 nm) [8], 9]. The reagents were Thromborel S and Actin FS for PT for APTT assays, respectively.

This study has been approved by the Institutional Review Board, Faculty of Medicine, Chulalongkorn University, Thailand (IRB No. 0519/67).

AV algorithm development

We developed the AV model by adhering to the CLSI AUTO10-A:2006 [16] and CLSI AUTO15-ED1:2019 [17] guidelines. An interdisciplinary team of clinical pathologists, medical technologists, instrument specialists, and information technology (IT) specialists collaborated to design the AV rules. We created rules based on a combination of pre-analysis, analysis, and post-analysis evaluation to minimize potential errors. The descriptions of each rule are explained in Table 1.

The steps of the AV process include.

1. Quality control (QC) status: QC status of the instruments was first verified before proceeding with AV process.

2. Sample serum indices: the analyzer evaluated the hemolysis, lipemia, and icterus indices to ensure they were within our limit settings.

3. Instrument and result flags or alarms: the flags indicated the need for re-evaluating the sample or the test, e.g. defective volume, clotting curve errors, and results formats.

4. Rerun/repeat result: all rerun or repeat results indicated a problem that should be verified manually.

5. Critical values: the test results exceeded the critical limits, indicating an urgent need for management.

6. Delta check: two consecutive test results from the same patient were assessed based on the laboratory’s established criteria for acceptable changes over a specified timeframe.

7. Verification decision limits (VDL): the VDL was calculated using population-based percentiles and the Tukey method to exclude outliers [7]. Then, the widest range from the percentiles and the lab’s reference values were selected to establish VDL range. Evalution for proper VDL range based on the simulation analysis passing rate.

8. Consistency check: served as a logical rule to identify potential addition errors, including heparin contamination and hemoconcentration (Hct>55 %).

All these processes had been custom-built within our LIS. The AV process is illustrated in Figure 1.

Figure 1:

The coagulation autoverification workflow. The flowchart shows the coagulation AV process. The process started with a quality control check, sample quality check, check for numerical results, analyzer error flags, critical values, delta checks, and consistency checks. If any issues were detected, results were blocked for manual review; otherwise, they were autoverified and reported. QC, quality control.

Validation of AV rules

The AV algorithms were evaluated sequentially during the preliminary phase, pilot phase, and go-live phase.

Data collection

Coagulation data were collected from both outpatients and inpatients at our hospital using the LIS. A total of 70,865 historical coagulation results were analyzed from January 2021 to December 2021 to establish the verification decision range for PT and APTT and implement AV system. Between October and December 2023, a total of 21,364 results were obtained through autoverification in parallel with manual verification to determine the passing rate and assess the performance of the protocols. Following protocol adjustments, an additional 4,509 results were collected in March 2024 to further evaluate the AV algorithm.

Once the protocols were validated, formal implementation started on March 20, 2024. From that date until August 2024, a total of 159,183 tests were collected to monitor the performance and efficiency of the AV system. In October 2024, we collected an extra 12,971 data for re-evaluation following the implementation of error reduction strategies.

Statistical analysis

IBM SPSS Statistics Version 25.0 (IBM Corporation, Chicago, USA) was used for statistical analysis. APTT and PT descriptive statistics, including mean, standard deviation (SD), and 2nd and 98th percentiles, were used to characterize the dataset of coagulation tests. The outlier was calculated by the Tukey Method. The AV rules passing rate and blocking rate were calculated into percentages. The performance of AV rules was analyzed by sensitivity, specificity, PPV, and NPV. The TAT was calculated using mean and SD. A comparison of turnaround times before and after AV implementation was conducted using the Mann-Whitney U test and p-Values < 0.001 was considered indicative of statistical significance. All figures were generated using Microsoft Excel 2021 (Microsoft Corporation, USA)

Define the VDL

From January 2021 to January 2022, a total of 42,149 PT and 28,716 APTT results from patient population were enrolled to calculate the 2nd, 5th, 95th, and 98th percentiles. Upon the preliminary study to set the VDL, we simulated a 4-month patient results (PT of 38,569 and APTT of 34,702). The results, as demonstrated in Table 2, revealed that selecting the 2nd and 98th percentiles combined with reference intervals yielded the highest passing rate with the same error rate.

Validation of AV system

A total of 21,364 tests from October to December 2023, using an AV algorithm were tested in parallel with manual verification. During October 2023, the overall AV passing rate was 62.4 % (6,111 out of 9,792), including 419 true positive cases (sensitivity 98.4 %), 6,104 true negative cases (specificity 65.2 %), 3,262 false positive cases, and seven cases (0.07 %) that were false negative. Similar outcomes were found during December 2023, the overall AV passing rate was 68.1 % (7,884 out of 11,572), including 455 true positive cases (sensitivity 98.1 %), 7,875 true negative cases (specificity 70.8 %), 3,254 false positive cases, and nine cases (0.08 %) that were false negative.

We thoroughly investigated all discordant results and identified that all false-negative cases, which should not have been auto-released, were released due to the instrument failing to detect partial clots. These clots were later identified by paired samples sent for complete blood count (CBC) analysis. However, these specimens exhibited only minimal reductions in the actual results, and most of them fell within the verification decision limit range. The changes in results ranged from −2.5 to 1.8 s for PT tests (mean −0.77 s±0.98) and −6.4 to 3.9 s for APTT tests (mean −3.34 s±2.5). The mean difference in partial clot cases was within the RCV, and TEa (Table 3).

Rules refinement and re-evaluation

After discovering 16 false-negative cases, we adjusted our protocol to minimize these errors by raising the low critical value of APTT from 20s to 21s and choosing specific instruments’ clotting curve error alarms to identify any abnormal clotting patterns (6.1–6.7 in Table 1). This adjustment was expected to prevent partial clots from going undetected by the instrument settings, and algorithm rules.

In March 2024, after rules adjustments, data was recollected before the go-live phase. The AV passing rate was 63.4 % (2,858 out of 4,509). The tests that were autoverified included APTT (2.3 %), PT (9.2 %), and APTT with PT (51.9 %). The true-positive were 88 cases, and the true-negative were 2,854 cases with 95.7 % sensitivity and 64.6 % specificity. However, three false negatives out of 4,509 cases (0.07 %) were caused by partial clots still occurring.

Post-implementation: assessment of coagulation AV performance

From May to June 2024 and Oct 2024, after full implementation, 7,588 out of 12,041 (63.0 %) results were autoverified, including APTT (3.5 %), PT (9.1 %), and APTT with PT (50.5 %). The sensitivity was 94.2 %, while the specificity was 64.1 %, resulting in an overall accuracy of 79.2 %. The NPV was 94.9 % and the PPV was 60.7 %. All 14 cases of false negatives (0.1 %) were identified due to the presence of partial clots, which are similar to patterns found in the earlier phases.

Due to the limitations of testing instruments that could not detect partial clots and the minimal, non-specific changes observed in APTT and PT results, strategies to reduce false negatives were further explored. We found that almost half of the APTT and PT tests had been submitted alongside CBC testing, which typically had a shorter processing time. When clots were detected during the CBC analysis, an alert message was triggered for the coagulation testing team to withhold the autoverification of the results.

After applying this strategy, an additional 12,971 data collected in October 2024 revealed false negative cases by partial clots could be reduced by 48 % (from 12 out of 23). Other instruments’ undetected partial clot specimens were promptly re-collected upon the identification of clots by the laboratory staff. Follow-up evaluations of the patients indicated no adverse effects on clinical outcomes or treatment changes. The AV rules demonstrated a sensitivity of 93.0 %, specificity of 65.0 %, NPV of 94.4 %, and PPV of 59.7 %. We found that most cases of AV failure resulted from defective sample volume at 21.5 %, icteric samples at 8.4 %, and APTT delta check issues at 7.7 %. The overall and individual rules’ blocking rate and false positive rate are shown in Table 4.

Impact of AV system on laboratory process

After completing the implementation phase, a total of 159,183 tests were collected from the LIS (March 20, 2024, to August 2024). The AV passing rate was 100,688 out of 159,183 (63.3 %), indicating that 36.7 % required manual verification. Furthermore, 61.6 % of the overall results had abnormal coagulation values.

The TAT data revealed a significant trend toward decreasing overall average turnaround times (refer to Table 5 and Figure 2) from 56.2 (±24.6) to 53.35 (±25.2) minutes (p<0.001). The median of TAT was shortening from 51.5 to 46.4 min. This trend was accompanied by 2.4 % increase (from 89.7 to 92.1 %) in delivered results within the laboratory’s guaranteed turnaround time [1:30 h]. The overall analysis time was also shortened by 4.32 min (50.5±24.2 and 47.2±23.1, p<0.001) (Table 5).

Figure 2:

The impact of laboratory workflow on TAT and the percentage of results achieved within the guaranteed TAT of 90 minutes for Overall, OPD, and IPD settings. TAT and percentage of results within the guaranteed TAT of 90 minutes for overall, OPD, and IPD. This graph shows the monthly trends in turnaround times, represented by the median TAT (P50) and 90th percentile TAT (P90), along with the percentage of results reported within 90 minutes (green line). Linear trend lines are included for the median, P90, and the percentage within 90 minutes. The post-implementation period is indicated during Mar 2024. (A) Overall (OPD and IPD), (B) OPD, (C) IPD TAT and percentage of results within the guaranteed TAT of 90 minutes. TAT, turnaround time; P50, 50th percentile of TAT; P90, 90th percentile of TAT; OPD, outpatient department; IPD, inpatient department.

A comparison between outpatient and inpatient laboratories revealed the outpatient group TAT was significantly shortened by 7.0 min, decreasing from 53.1 (±20.5) to 46.1 (±19.2) minutes. In contrast, the TAT in inpatient settings showed only a minor reduction of 1.1 min, from 57.4 (±25.6) to 56.3 (±26.4) minutes. The AV passing rates were higher in the outpatient group compared to the inpatient group (69.2 vs. 56.3 %) (Table 5). The trend lines for outpatient samples in Figure 2B indicated a more substantial improvement in the laboratory process, in which 96.8 % of all tests can be released within our laboratory’s time criteria. When comparing the laboratory’s working hours in OPD by evaluating the TAT across different time intervals, marked improvements can be observed before and after the implementation during the same month in different years (August 2023 vs. August 2024) (Figure 3). The data show that post-implementation, the average TAT decreased across all time slots, with particularly noticeable reductions during peak hours (7–11 AM), where the volume of specimens processed was comparable.

Figure 3:

Average turnaround time for OPD before and after AV implementation during August 2023 compared to August 2024. Average turnaround time for the OPD before and after AV implementation, compared over each working hour in one month, analyzing August 2023 (pre-implementation with a total of 3,362 samples) and August 2024 (post-implementation with a total of 3,472 samples).

In terms of patient safety, during the go-live implementation, no incident reports occurred by autoverified results.