Home Using machine learning to develop an autoverification system in a clinical biochemistry laboratory
Article
Licensed
Unlicensed Requires Authentication

Using machine learning to develop an autoverification system in a clinical biochemistry laboratory

  • Hongchun Wang , Huayang Wang , Jian Zhang , Xiaoli Li , Chengxi Sun and Yi Zhang EMAIL logo
Published/Copyright: November 26, 2020
Become an author with De Gruyter Brill
Submit Manuscript Author Information
Clinical Chemistry and Laboratory Medicine (CCLM)
From the journal Clinical Chemistry and Laboratory Medicine (CCLM) Volume 59 Issue 5

Abstract

Objectives

Autoverification systems have greatly improved laboratory efficiency. However, the long-developed rule-based autoverfication models have limitations. The machine learning (ML) algorithm possesses unique advantages in the evaluation of large datasets. We investigated the utility of ML algorithms for developing an artificial intelligence (AI) autoverification system to support laboratory testing. The accuracy and efficiency of the algorithm model were also validated.

Methods

Testing data, including 52 testing items with demographic information, were extracted from the laboratory information system and Roche Cobas® IT 3000 from June 1, 2018 to August 30, 2019. Two rounds of modeling were conducted to train different ML algorithms and test their abilities to distinguish invalid reports. Algorithms with the top three best performances were selected to form the finalized ensemble model. Double-blind testing between experienced laboratory personnel and the AI autoverification system was conducted, and the passing rate and false-negative rate (FNR) were documented. The working efficiency and workload reduction were also analyzed.

Results

The final AI system showed a 89.60% passing rate and 0.95 per mille FNR, in double-blind testing. The AI system lowered the number of invalid reports by approximately 80% compared to those evaluated by a rule-based engine, and therefore enhanced the working efficiency and reduced the workload in the biochemistry laboratory.

Conclusions

We confirmed the feasibility of the ML algorithm for autoverification with high accuracy and efficiency.

Keywords: autoverification; biochemistry; clinical laboratory; machine learning
Corresponding author: Yi Zhang, Professor, Department of Clinical Laboratory, Qilu Hospital of Shandong University, #107 Wenhuaxi Road, Jinan, 250012, Shandong, P.R. China, Phone: +86-531-82166801, Fax: +86-531-86927544, E-mail:

Hongchun Wang and Huayang Wang contributed equally to this article.

Funding source: Natural Science Foundation of Shandong Province

Award Identifier / Grant number: ZR2017MH044

Funding source: Key Technology Research and Development Program of Shandong, China

Award Identifier / Grant number: 2019GSF108247

Funding source: National Natural Science Foundation of China

Award Identifier / Grant number: 81702815, 81972005

Funding source: Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan

Award Identifier / Grant number: 201805061

Acknowledgments

We would like to acknowledge Roche Diagnostics (Shanghai) Limited, Digital Solution team from CI and Solution Integration team from CPS & MD, for their digital expertise and technical supports.

  1. Research funding: National Natural Science Foundation of China (No. 81972005, 81702815), Natural Science Foundation of Shandong Province (No. ZR2017MH044), Key Technology Research and Development Program of Shandong, China (No. 2019GSF108247), Jinan Science and Technology Plan & Clinical Medical Technology Innovation Plan (No. 201805061).

  2. Author contributions: All authors have accepted responsibility for the entire content of this submitted manuscript and approved its submission.

  3. Conflicts of Interest: Authors state no conflict of interest.

  4. Ethical approval: This study has been approved by the Human Research Ethics Committee of Qilu Hospital of Shandong University (KYLL-2019-2-045).

References

1. Randell, EW, Yenice, S, Khine Wamono, AA, Orth, M. Autoverification of test results in the core clinical laboratory. Clin Biochem 2019;73:11–25. https://doi.org/10.1016/j.clinbiochem.2019.08.002.Search in Google Scholar PubMed

2. Autoverification of Clinical Laboratory Test Results, 1st ed. CLSI guideline AUTO10-A; 2006. [Online]. Available from: https://clsi.org/standards/products/automation-and-informatics/documents/auto10/.Search in Google Scholar

3. Autoverification of Medical Laboratory Results for Specific Disciplines, 1st ed. CLSI guideline AUTO15; 2019. [Online]. Available from: https://clsi.org/standards/products/automation-and-informatics/documents/auto15/.Search in Google Scholar

4. Valdiguie, PM, Rogari, E, Philippe, H. VALAB: expert system for validation of biochemical data. Clin Chem 1992;38:83–7. https://doi.org/10.1093/clinchem/38.1.83.Search in Google Scholar

5. Cabitza, F, Banfi, G. Machine learning in laboratory medicine: waiting for the flood? Clin Chem Lab Med 2018;56:516–24. https://doi.org/10.1515/cclm-2017-0287.Search in Google Scholar PubMed

6. Obermeyer, Z, Emanuel, EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med 2016;375:1216–9. https://doi.org/10.1056/nejmp1606181.Search in Google Scholar PubMed PubMed Central

7. Wilkes, EH, Rumsby, G, Woodward, GM. Using machine learning to aid the interpretation of urine steroid profiles. Clin Chem 2018;64:1586–95. https://doi.org/10.1373/clinchem.2018.292201.Search in Google Scholar PubMed

8. Luo, Y, Szolovits, P, Dighe, AS, Baron, JM. Using machine learning to predict laboratory test results. Am J Clin Pathol 2016;145:778–88. https://doi.org/10.1093/ajcp/aqw064.Search in Google Scholar PubMed

9. Somnay, YR, Craven, M, McCoy, KL, Carty, SE, Wang, TS, Greenberg, CC, et al.. Improving diagnostic recognition of primary hyperparathyroidism with machine learning. Surgery 2017;161:1113–21. https://doi.org/10.1016/j.surg.2016.09.044.Search in Google Scholar PubMed PubMed Central

10. Krasowski, MD, Davis, SR, Drees, D, Morris, C, Kulhavy, J, Crone, C, et al.. Autoverification in a core clinical chemistry laboratory at an academic medical center. J Pathol Inf 2014;5:13. https://doi.org/10.4103/2153-3539.129450.Search in Google Scholar PubMed PubMed Central

11. Demirci, F, Akan, P, Kume, T, Sisman, AR, Erbayraktar, Z, Sevinc, S. Artificial neural network approach in laboratory test reporting: learning algorithms. Am J Clin Pathol 2016;146:227–37. https://doi.org/10.1093/ajcp/aqw104.Search in Google Scholar PubMed

12. Zhu, J, Wang, B, Guo, W, Pan, B. Establishment and optimization of an autoverification system for clinical biochemistry test results [in Chinese]. Chin J Clin Lab Sci 2018;36:704–8.Search in Google Scholar

13. Dabbura, I. K-means clustering: algorithm, applications, evaluation methods, and drawbacks; 2018. Available from: https://imaddabbura.github.io/post/kmeans-clustering/.Search in Google Scholar

14. Autoverification of clinical laboratory quantitative test results [in Chinese], National Health Commission of PRC standard WS/T 616-2018; 2018. [Online]. Available from: http://www.nhc.gov.cn/ewebeditor/uploadfile/2018/09/20180925121506686.pdf.Search in Google Scholar

15. Chawla, N, Bowyer, K, Hall, L, Kegelmeyer, W. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321–57. https://doi.org/10.1613/jair.953.Search in Google Scholar

16. He, H, Bai, Y, Garcia, E, Li, S. ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence). China: Hong Kong; 2008.Search in Google Scholar

17. Barber, D. Bayesian reasoning and machine learning. New York, NY: Cambridge University Press; 2012:697 p.10.1017/CBO9780511804779Search in Google Scholar

18. Guo, G, Wang, H, Bell, D, Bi, Y, Greer, K. KNN model-based approach in classification. In: Meersman, R, Tari, Z, Schmidt, DC, editors. On the move to meaningful internet systems. German: Springer; 2003.10.1007/978-3-540-39964-3_62Search in Google Scholar

19. Deng, H. An introduction to random forest; 2018. Available from: https://towardsdatascience.com/random-forest-3a55c3aca46d.Search in Google Scholar

20. Chen, T. Introduction to boosted trees; 2014. Available from: https://homes.cs.washington.edu/∼tqchen/pdf/BoostedTree.pdf.Search in Google Scholar

21. Hastie, T, Tibshirani, R, Friedman, JH. The elements of statistical learning: data mining, inference, and prediction, 2nd ed. New York, NY: Springer; 2009:745 p.10.1007/978-0-387-84858-7Search in Google Scholar

22. Murdoch, WJ, Singh, C, Kumbier, K, Abbasi-Asl, R, Yu, B. Definitions, methods, and applications in interpretable machine learning. Proc Natl Acad Sci U S A 2019;116:22071–80. https://doi.org/10.1073/pnas.1900654116.Search in Google Scholar PubMed PubMed Central

23. Richardson, A, Signor, BM, Lidbury, BA, Badrick, T. Clinical chemistry in higher dimensions: machine-learning and enhanced prediction from routine clinical chemistry data. Clin Biochem 2016;49:1213–20. https://doi.org/10.1016/j.clinbiochem.2016.07.013.Search in Google Scholar PubMed

24. Allen, RH. The evaluation and choice of laboratory equipment and reagents. Can Med Assoc J 1965;93:760–1.Search in Google Scholar

25. Doshi-Velez, F, Kim, B. Towards A rigorous science of interpretable machine learning. arXiv; 2017. Available from: https://arxiv.org/abs/1702.08608v2.Search in Google Scholar

26. Torke, N, Boral, L, Nguyen, T, Perri, A, Chakrin, A. Process improvement and operational efficiency through test result autoverification. Clin Chem 2005;51:2406–8. https://doi.org/10.1373/clinchem.2005.054395.Search in Google Scholar PubMed

27. Chauhan, KP, Trivedi, AP, Patel, D, Gami, B, Haridas, N. Monitoring and root cause analysis of clinical biochemistry turn around time at an academic hospital. Indian J Clin Biochem 2014;29:505–9. https://doi.org/10.1007/s12291-013-0397-x.Search in Google Scholar PubMed PubMed Central

28. Wang, Z, Peng, C, Kang, H, Fan, X, Mu, R, Zhou, L, et al.. Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory. BMC Med Inf Decis Making 2019;19:123. https://doi.org/10.1186/s12911-019-0848-2.Search in Google Scholar PubMed PubMed Central

29. Kumar, Y, Sahoo, G. Prediction of different types of liver diseases using rule based classification model. Technol Health Care 2013;21:417–32. https://doi.org/10.3233/thc-130742.Search in Google Scholar PubMed

30. Ruthotto, L, Osher, SJ, Li, W, Nurbekyan, L, Fung, SW. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proc Natl Acad Sci U S A 2020;117:9183–93. https://doi.org/10.1073/pnas.1922204117.Search in Google Scholar PubMed PubMed Central

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2020-0716).

Received: 2020-05-14
Accepted: 2020-11-12
Published Online: 2020-11-26
Published in Print: 2021-04-27

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Home pregnancy tests: quality first
  4. Review
  5. Non-invasive determination of uric acid in human saliva in the diagnosis of serious disorders
  6. Opinion Papers
  7. Basophil counting in hematology analyzers: time to discontinue?
  8. The role of laboratory hematology between technology and professionalism: the paradigm of basophil counting
  9. Recommendations for validation testing of home pregnancy tests (HPTs) in Europe
  10. General Clinical Chemistry and Laboratory Medicine
  11. The use of preanalytical quality indicators: a Turkish preliminary survey study
  12. The Italian External Quality Assessment (EQA) program on urinary sediment by microscopy examination: a 20 years journey
  13. Non-HDL-C/TG ratio indicates significant underestimation of calculated low-density lipoprotein cholesterol (LDL-C) better than TG level: a study on the reliability of mathematical formulas used for LDL-C estimation
  14. Evaluation of the protein gap for detection of abnormal serum gammaglobulin level: an imperfect predictor
  15. Impact of routine S100B protein assay on CT scan use in children with mild traumatic brain injury
  16. Using machine learning to develop an autoverification system in a clinical biochemistry laboratory
  17. Effect of collection matrix, platelet depletion, and storage conditions on plasma extracellular vesicles and extracellular vesicle-associated miRNAs measurements
  18. Pneumatic tube transportation of urine samples
  19. Evaluation of the first immunosuppressive drug assay available on a fully automated LC-MS/MS-based clinical analyzer suggests a new era in laboratory medicine
  20. A validated LC-MS/MS method for the simultaneous quantification of the novel combination antibiotic, ceftolozane–tazobactam, in plasma (total and unbound), CSF, urine and renal replacement therapy effluent: application to pilot pharmacokinetic studies
  21. Immunosuppressant quantification in intravenous microdialysate – towards novel quasi-continuous therapeutic drug monitoring in transplanted patients
  22. Reference Values and Biological Variations
  23. Reference intervals for venous blood gas measurement in adults
  24. Cardiovascular Diseases
  25. Detection and functional characterization of a novel MEF2A variation responsible for familial dilated cardiomyopathy
  26. Diabetes
  27. Evaluation of the ARKRAY HA-8190V instrument for HbA1c
  28. Infectious Diseases
  29. An original multiplex method to assess five different SARS-CoV-2 antibodies
  30. Evaluation of dried blood spots as alternative sampling material for serological detection of anti-SARS-CoV-2 antibodies using established ELISAs
  31. Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR
  32. The vasoactive peptide MR-pro-adrenomedullin in COVID-19 patients: an observational study
  33. Corrigenda
  34. Corrigendum to: Understanding and managing interferences in clinical laboratory assays: the role of laboratory professionals
  35. Corrigendum to: Age appropriate reference intervals for eight kidney function and injury markers in infants, children and adolescents
  36. Letters to the Editor
  37. A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data on Sysmex XN-series analysers
  38. Letter in reply to the letter to the editor of Harte JV and Mykytiv V with the title “A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data”
  39. SARS-CoV-2 serologic tests: do not forget the good laboratory practice
  40. Long-term kinetics of anti-SARS-CoV-2 antibodies in a cohort of 197 hospitalized and non-hospitalized COVID-19 patients
  41. Self-sampling at home using volumetric absorptive microsampling: coupling analytical evaluation to volunteers’ perception in the context of a large scale study
  42. Vortex mixing to alleviate pseudothrombocytopenia in a blood specimen with platelet satellitism and platelet clumps
  43. Comparative evaluation of the fully automated HemosIL® AcuStar ADAMTS13 activity assay vs. ELISA: possible interference by autoantibodies different from anti ADAMTS-13
  44. Significant interference on specific point-of-care glucose measurements due to high dose of intravenous vitamin C therapy in critically ill patients
  45. As time goes by, on that you can rely … preservation of urine samples for morphological analysis of erythrocytes and casts
  46. Stability of control materials for α-thalassemia immunochromatographic strip test
  47. Reformulated Architect® cyclosporine CMIA assay: improved imprecision, worse comparability between methods
  48. Urine-to-plasma contamination mimicking acute kidney injury: small drops with major consequences
  49. Automated Mindray CL-1200i chemiluminescent assays of renin and aldosterone for the diagnosis of primary aldosteronism
  50. Use of common reference intervals does not necessarily allow inter-method numerical result trending
  51. Reply to Dr Hawkins regarding comparability of results for monitoring
https://doi.org/10.1515/cclm-2020-0716
Keywords for this article
autoverification; biochemistry; clinical laboratory; machine learning

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Home pregnancy tests: quality first
  4. Review
  5. Non-invasive determination of uric acid in human saliva in the diagnosis of serious disorders
  6. Opinion Papers
  7. Basophil counting in hematology analyzers: time to discontinue?
  8. The role of laboratory hematology between technology and professionalism: the paradigm of basophil counting
  9. Recommendations for validation testing of home pregnancy tests (HPTs) in Europe
  10. General Clinical Chemistry and Laboratory Medicine
  11. The use of preanalytical quality indicators: a Turkish preliminary survey study
  12. The Italian External Quality Assessment (EQA) program on urinary sediment by microscopy examination: a 20 years journey
  13. Non-HDL-C/TG ratio indicates significant underestimation of calculated low-density lipoprotein cholesterol (LDL-C) better than TG level: a study on the reliability of mathematical formulas used for LDL-C estimation
  14. Evaluation of the protein gap for detection of abnormal serum gammaglobulin level: an imperfect predictor
  15. Impact of routine S100B protein assay on CT scan use in children with mild traumatic brain injury
  16. Using machine learning to develop an autoverification system in a clinical biochemistry laboratory
  17. Effect of collection matrix, platelet depletion, and storage conditions on plasma extracellular vesicles and extracellular vesicle-associated miRNAs measurements
  18. Pneumatic tube transportation of urine samples
  19. Evaluation of the first immunosuppressive drug assay available on a fully automated LC-MS/MS-based clinical analyzer suggests a new era in laboratory medicine
  20. A validated LC-MS/MS method for the simultaneous quantification of the novel combination antibiotic, ceftolozane–tazobactam, in plasma (total and unbound), CSF, urine and renal replacement therapy effluent: application to pilot pharmacokinetic studies
  21. Immunosuppressant quantification in intravenous microdialysate – towards novel quasi-continuous therapeutic drug monitoring in transplanted patients
  22. Reference Values and Biological Variations
  23. Reference intervals for venous blood gas measurement in adults
  24. Cardiovascular Diseases
  25. Detection and functional characterization of a novel MEF2A variation responsible for familial dilated cardiomyopathy
  26. Diabetes
  27. Evaluation of the ARKRAY HA-8190V instrument for HbA1c
  28. Infectious Diseases
  29. An original multiplex method to assess five different SARS-CoV-2 antibodies
  30. Evaluation of dried blood spots as alternative sampling material for serological detection of anti-SARS-CoV-2 antibodies using established ELISAs
  31. Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR
  32. The vasoactive peptide MR-pro-adrenomedullin in COVID-19 patients: an observational study
  33. Corrigenda
  34. Corrigendum to: Understanding and managing interferences in clinical laboratory assays: the role of laboratory professionals
  35. Corrigendum to: Age appropriate reference intervals for eight kidney function and injury markers in infants, children and adolescents
  36. Letters to the Editor
  37. A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data on Sysmex XN-series analysers
  38. Letter in reply to the letter to the editor of Harte JV and Mykytiv V with the title “A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data”
  39. SARS-CoV-2 serologic tests: do not forget the good laboratory practice
  40. Long-term kinetics of anti-SARS-CoV-2 antibodies in a cohort of 197 hospitalized and non-hospitalized COVID-19 patients
  41. Self-sampling at home using volumetric absorptive microsampling: coupling analytical evaluation to volunteers’ perception in the context of a large scale study
  42. Vortex mixing to alleviate pseudothrombocytopenia in a blood specimen with platelet satellitism and platelet clumps
  43. Comparative evaluation of the fully automated HemosIL® AcuStar ADAMTS13 activity assay vs. ELISA: possible interference by autoantibodies different from anti ADAMTS-13
  44. Significant interference on specific point-of-care glucose measurements due to high dose of intravenous vitamin C therapy in critically ill patients
  45. As time goes by, on that you can rely … preservation of urine samples for morphological analysis of erythrocytes and casts
  46. Stability of control materials for α-thalassemia immunochromatographic strip test
  47. Reformulated Architect® cyclosporine CMIA assay: improved imprecision, worse comparability between methods
  48. Urine-to-plasma contamination mimicking acute kidney injury: small drops with major consequences
  49. Automated Mindray CL-1200i chemiluminescent assays of renin and aldosterone for the diagnosis of primary aldosteronism
  50. Use of common reference intervals does not necessarily allow inter-method numerical result trending
  51. Reply to Dr Hawkins regarding comparability of results for monitoring
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 9.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cclm-2020-0716/pdf
Scroll to top button