Home Diagnostic performance of machine learning models using cell population data for the detection of sepsis: a comparative study
Article
Licensed
Unlicensed Requires Authentication

Diagnostic performance of machine learning models using cell population data for the detection of sepsis: a comparative study

  • Urko Aguirre ORCID logo EMAIL logo and Eloísa Urrechaga ORCID logo
Published/Copyright: November 11, 2022
Become an author with De Gruyter Brill
Submit Manuscript Author Information
Clinical Chemistry and Laboratory Medicine (CCLM)
From the journal Clinical Chemistry and Laboratory Medicine (CCLM) Volume 61 Issue 2

Abstract

Objectives

To compare the artificial intelligence algorithms as powerful machine learning methods for evaluating patients with suspected sepsis using data from routinely available blood tests performed on arrival at the hospital. Results were compared with those obtained from the classical logistic regression method.

Methods

The study group consisted of consecutive patients with fever and suspected infection admitted to the Emergency Department. The complete blood counts (CBC) were acquired using the Mindray BC-6800 Plus analyser (Mindray Diagnostics, Shenzhen, China). Cell Population Data (CPD) were also recorded. The ML and artificial intelligence (AI) models were developed; their performance was evaluated using several indicators, such as the area under the receiver operating curve (AUC), calibration plots and decision curve analysis (DCA).

Results

Overall, all the tested approaches obtained an AUC>0.90. The logistic regression (LR) performed well compared to the ML/AI models. The naïve Bayes and the K-nearest neighbour (KNN) methods did not show good calibration properties. The multi-layer perceptron (MLP) model was the best in terms of discrimination, calibration and clinical usefulness.

Conclusions

The best performance in the early detection of sepsis was achieved using the ML and AI models. However, external validation studies are needed to strengthen model derivation and procedure updating.

Keywords: artificial intelligence; cell population data; leukocytes; machine learning; Mindray BC 6800 Plus; sepsis
Corresponding author: Urko Aguirre, Research Unit, Osakidetza Basque Health Service, Barrualde-Galdakao Integrated Health Organisation, Galdakao-Usansolo Hospital, 48960, Galdakao, Spain; Kronikgune Institute for Health Services Research, 48902 Barakaldo, Spain; Research Network in Health Services in Chronic Diseases (Red de Investigación en Servicios de Salud en Enfermedades Crónicas, REDISSEC), 48960, Galdakao, Spain; and Network for Research on Chronicity, Primary Care, and Health Promotion (RICAPPS), Galdakao, Spain, E-mail:

  1. Research funding: None declared.

  2. Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Informed consent was obtained from all individuals included in this study.

  5. Ethical approval: Approval from the Ethics Committee Board of our hospital was obtained.

References

1. Angus, DC, van der Poll, T. Severe sepsis and septic shock. N Engl J Med 2013;369:840–51. https://doi.org/10.1056/nejmra1208623.Search in Google Scholar

2. Singer, M, Deutschman, CS, Seymour, CW, Shankar-Hari, M, Annane, D, Bauer, M, et al.. The Third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:801–10. https://doi.org/10.1001/jama.2016.0287.Search in Google Scholar PubMed PubMed Central

3. Gyawali, B, Ramakrishna, K, Dhamoon, AS. Sepsis: the evolution in definition, pathophysiology, and management. SAGE Open Med 2019;7:2050312119835043. https://doi.org/10.1177/2050312119835043.Search in Google Scholar PubMed PubMed Central

4. Reinhart, K, Daniels, R, Kissoon, N, Machado, FR, Schachter, RD, Finfer, S. Recognizing sepsis as a global health priority — a WHO resolution. N Engl J Med 2017;377:414–7. https://doi.org/10.1056/nejmp1707170.Search in Google Scholar

5. Brun-Buisson, C. The epidemiology of the systemic inflammatory response. Intensive Care Med 2000;26:S064–74. https://doi.org/10.1007/s001340051121.Search in Google Scholar PubMed PubMed Central

6. Graber, ML, Patel, M, Claypool, S. Sepsis as a model for improving diagnosis. Diagnosis 2018;5:3–10. https://doi.org/10.1515/dx-2017-0036.Search in Google Scholar PubMed

7. Faix, JD. Biomarkers of sepsis. Crit Rev Clin Lab Sci 2013;50:23–36. https://doi.org/10.3109/10408363.2013.764490.Search in Google Scholar PubMed PubMed Central

8. Lippi, G. Sepsis biomarkers: past, present and future. Clin Chem Lab Med 2019;57:1281–3. https://doi.org/10.1515/cclm-2018-1347.Search in Google Scholar PubMed

9. Schuetz, P, Plebani, M. Can biomarkers help us to better diagnose and manage sepsis? Diagnosis 2015;2:81–7. https://doi.org/10.1515/dx-2014-0073.Search in Google Scholar PubMed

10. Wang, X, Zhu, T, Xia, M, Liu, Y, Wang, Y, Wang, X, et al.. Predicting the prognosis of patients in the coronary care unit: a novel multi-category machine learning model using XGBoost. Front Cardiovasc Med 2022;9:764629. https://doi.org/10.3389/fcvm.2022.764629.Search in Google Scholar PubMed PubMed Central

11. Lu, J, Bu, P, Xia, X, Lu, N, Yao, L, Jiang, H. Feasibility of machine learning methods for predicting hospital emergency room visits for respiratory diseases. Environ Sci Pollut Control Ser 2021;28:29701–9. https://doi.org/10.1007/s11356-021-12658-7.Search in Google Scholar PubMed

12. Layeghian Javan, S, Sepehri, MM, Layeghian Javan, M, Khatibi, T. An intelligent warning model for early prediction of cardiac arrest in sepsis patients. Comput Methods Progr Biomed 2019;178:47–58. https://doi.org/10.1016/j.cmpb.2019.06.010.Search in Google Scholar PubMed

13. Singh, YV, Singh, P, Khan, S, Singh, RS, Gupta, SK. A machine learning model for early prediction and detection of sepsis in intensive care unit patients. J Healthc Eng 2022;2022:1–11. https://doi.org/10.1155/2022/9263391.Search in Google Scholar PubMed PubMed Central

14. Wang, D, Li, J, Sun, Y, Ding, X, Zhang, X, Liu, S, et al.. A machine learning model for accurate prediction of sepsis in ICU patients. Front Public Health 2021;9:754348. https://doi.org/10.3389/fpubh.2021.754348.Search in Google Scholar PubMed PubMed Central

15. Urrechaga, E, Bóveda, O, Aguirre, U. Role of leucocytes cell population data in the early detection of sepsis. J Clin Pathol 2018;71:259–66. https://doi.org/10.1136/jclinpath-2017-204524.Search in Google Scholar PubMed

16. Urrechaga, E, Bóveda, O, Aguirre, U. Improvement in detecting sepsis using leukocyte cell population data (CPD). Clin Chem Lab Med 2019;57:918–26. https://doi.org/10.1515/cclm-2018-0979.Search in Google Scholar PubMed

17. Martens, RJH, van Adrichem, AJ, Mattheij, NJA, Brouwer, CG, van Twist, DJL, Broerse, JJCR, et al.. Hemocytometric characteristics of COVID-19 patients with and without cytokine storm syndrome on the sysmex XN-10 hematology analyzer. Clin Chem Lab Med 2021;59:783–93. https://doi.org/10.1515/cclm-2020-1529.Search in Google Scholar PubMed

18. Urrechaga, E, Aguirre, U, España, PP, García de Guadiana, L. Complete blood counts and cell population data from Sysmex XN analyser in the detection of SARS-CoV-2 infection. Clin Chem Lab Med 2021;59:e57–60. https://doi.org/10.1515/cclm-2020-1416.Search in Google Scholar PubMed

19. Harte, JV, Mykytiv, V. A panhaemocytometric approach to COVID-19: a retrospective study on the importance of monocyte and neutrophil population data on Sysmex XN-series analysers. Clin Chem Lab Med 2021;59:e169–72. https://doi.org/10.1515/cclm-2021-0096.Search in Google Scholar PubMed

20. Urrechaga, E. Morphometric parameters of leukocytes in the management of sepsis running title: cell population data in sepsis. Highl Med Med Res 2021;9:44–58.10.9734/bpi/hmmr/v9/8560DSearch in Google Scholar

21. Webb, GI. Naïve Bayes. In Sammut C, Webb GI, editors. Encyclopedia of machine learning. Boston, MA: Springer; 2011: 713–4 pp.10.1007/978-0-387-30164-8_576Search in Google Scholar

22. Hastie, TT, Robert, Friedman, J. The elements of statistical learning. New York, NY: Springer; 2009.10.1007/978-0-387-84858-7Search in Google Scholar

23. Chen, T, Guestrin, C. XGBoost. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. New York, NY: Association for Computing Machinery; 2016:785–94 pp.10.1145/2939672.2939785Search in Google Scholar

24. LaValle, SM, Branicky, MS, Lindemann, SR. On the relationship between classical grid search and probabilistic roadmaps. Int J Robot Res 2016;23:673–92. https://doi.org/10.1177/0278364904045481.Search in Google Scholar

25. Kohavi, R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International joint conference on artificial intelligence IJCA; 1995, vol. 2:9 p.Search in Google Scholar

26. Ruopp, MD, Perkins, NJ, Whitcomb, BW, Schisterman, EF. Youden index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J 2008;50:419–30. https://doi.org/10.1002/bimj.200710415.Search in Google Scholar PubMed PubMed Central

27. Rufibach, K. Use of brier score to assess binary predictions. J Clin Epidemiol 2010;63:938–9. https://doi.org/10.1016/j.jclinepi.2009.11.009.Search in Google Scholar PubMed

28. Huang, Y, Li, W, Macheret, F, Gabriel, RA, Ohno-Machado, L. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Inf Assoc 2020;27:621–33. https://doi.org/10.1093/jamia/ocz228.Search in Google Scholar PubMed PubMed Central

29. Appenzeller, C, Liniger, MA, Weigel, AP. The discrete brier and ranked probability skill scores. Mon Weather Rev 2007;135:118–24. https://doi.org/10.1175/mwr3280.1.Search in Google Scholar

30. Lundberg, SM, Erion, G, Chen, H, DeGrave, A, Prutkin, JM, Nair, B, et al.. From local explanations to global understanding with explainable AI for trees. Nat Mach Intell 2020;2:56–67. https://doi.org/10.1038/s42256-019-0138-9.Search in Google Scholar PubMed PubMed Central

31. Pedregosa, F, Varoquaux, G, Gramfort, A, Michel, V, Thirion, B, Grisel, O, et al.. Scikit-learn: machine learning in Python. J Mach Learn Res 2011;12:2825–30.Search in Google Scholar

32. Portell, RIM, Alarcón, RR, Benayas, BMP, Avivar, C. Analysis of leukocyte cell population data (CPD) as biomarkers in the diagnosis of sepsis. Hematol Transfus Int J 2022;2:33–7.10.15406/htij.2022.10.00278Search in Google Scholar

33. Buoro, S, Manenti, B, Seghezzi, M, Dominoni, P, Barbui, T, Ghirardi, A, et al.. Innovative haematological parameters for early diagnosis of sepsis in adult patients admitted in intensive care unit. J Clin Pathol 2018;71:330–5. https://doi.org/10.1136/jclinpath-2017-204643.Search in Google Scholar PubMed

34. Zhang, W, Zhang, Z, Pan, S, Li, J, Yang, Y, Qi, H, et al.. The clinical value of hematological neutrophil and monocyte parameters in the diagnosis and identification of sepsis. Ann Transl Med 2021;9:1680. https://doi.org/10.21037/atm-21-5639.Search in Google Scholar PubMed PubMed Central

35. Buoro, S, Carobene, A, Seghezzi, M, Manenti, B, Pacioni, A, Ceriotti, F, et al.. Short- and medium-term biological variation estimates of leukocytes extended to differential count and morphology-structural parameters (cell population data) in blood samples obtained from healthy people. Clin Chim Acta 2017;473:147–56. https://doi.org/10.1016/j.cca.2017.07.009.Search in Google Scholar PubMed

36. Tang, H, Jing, J, Bo, D, Xu, D. Biological variations of leukocyte numerical and morphologic parameters determined by UniCel DxH 800 hematology analyzer. Arch Pathol Lab Med 2012;136:1392–6. https://doi.org/10.5858/arpa.2011-0679-oa.Search in Google Scholar PubMed

37. Hoffmann, J. New hematology analyzer parameters and their clinical relevance. EFLM Newslett 2018;1:3.Search in Google Scholar

38. Seghezzi, M, Buoro, S, Previtali, G, Moioli, V, Manenti, B, Simon-Lopez, R, et al.. A preliminary proposal for quality control assessment and harmonization of leukocytes morphology-structural parameters (cell population data parameters). J Med Biochem 2018;37:486–98. https://doi.org/10.2478/jomb-2018-0005.Search in Google Scholar PubMed PubMed Central

39. Rajula, HSR, Verlato, G, Manchia, M, Antonucci, N, Fanos, V. Comparison of conventional statistical methods with machine learning in medicine: diagnosis, drug development, and treatment. Medicina 2020;56:455. https://doi.org/10.3390/medicina56090455.Search in Google Scholar PubMed PubMed Central

40. Zhang, Z, Hong, Y. Development of a novel score for the prediction of hospital mortality in patients with severe sepsis: the use of electronic healthcare records with LASSO regression. Oncotarget 2017;8:49637–45. https://doi.org/10.18632/oncotarget.17870.Search in Google Scholar PubMed PubMed Central

41. Rawson, TM, Hernandez, B, Moore, LSP, Blandy, O, Herrero, P, Gilchrist, M, et al.. Supervised machine learning for the prediction of infection on admission to hospital: a prospective observational cohort study. J Antimicrob Chemother 2019;74:1108–15. https://doi.org/10.1093/jac/dky514.Search in Google Scholar PubMed

42. Hu, C, Li, L, Huang, W, Wu, T, Xu, Q, Liu, J, et al.. Interpretable machine learning for early prediction of prognosis in sepsis: a discovery and validation study. Infect Dis Ther 2022;11:1117–32. https://doi.org/10.1007/s40121-022-00628-6.Search in Google Scholar PubMed PubMed Central

43. Kijpaisalratana, N, Sanglertsinlapachai, D, Techaratsami, S, Musikatavorn, K, Saoraya, J. Machine learning algorithms for early sepsis detection in the emergency department: a retrospective study. Int J Med Inf 2022;160:104689. https://doi.org/10.1016/j.ijmedinf.2022.104689.Search in Google Scholar PubMed

44. Ghias, N, Haq, SU, Arshad, H, Sultan, H, Bashir, F, Ghaznavi, SA, et al.. Using machine learning algorithms to predict sepsis and its stages in ICU patients. medRxiv 2022:2022.03.15.22271655. https://doi.org/10.1101/2022.03.15.22271655.Search in Google Scholar

45. Zeng, Z, Yao, S, Zheng, J, Gong, X. Development and validation of a novel blending machine learning model for hospital mortality prediction in ICU patients with Sepsis. BioData Min 2021;14. https://doi.org/10.1186/s13040-021-00276-5.Search in Google Scholar PubMed PubMed Central

46. Kuhn, M, Johnson, K. Applied predictive modeling. New York, NY: Springer; 2013.10.1007/978-1-4614-6849-3Search in Google Scholar

47. Dankowski, T, Ziegler, A. Calibrating Random Forests for probability estimation. Stat Med 2016;35:3949–60. https://doi.org/10.1002/sim.6959.Search in Google Scholar PubMed PubMed Central

48. Cabitza, F, Rasoini, R, Gensini, GF. Unintended consequences of machine learning in medicine. JAMA 2017;318:517–8. https://doi.org/10.1001/jama.2017.7797.Search in Google Scholar PubMed

49. Martins, EC, Silveira, LdF, Viegas, K, Beck, AD, Fioravantti Júnior, G, Cremonese, RV, et al.. Neutrophil-lymphocyte ratio in the early diagnosis of sepsis in an intensive care unit: a case-control study. Revista Brasileira de Terapia Intensiva 2019;31:64–70. https://doi.org/10.5935/0103-507x.20190010.Search in Google Scholar PubMed PubMed Central

50. Drăgoescu, AN, Pădureanu, V, Stănculescu, AD, Chiuțu, LC, Tomescu, P, Geormăneanu, C, et al.. Neutrophil to lymphocyte ratio (NLR)—a useful tool for the prognosis of sepsis in the ICU. Biomedicines 2021;10:75. https://doi.org/10.3390/biomedicines10010075.Search in Google Scholar PubMed PubMed Central

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2022-0713).

Received: 2022-07-24
Accepted: 2022-10-18
Published Online: 2022-11-11
Published in Print: 2023-01-27

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Standardization and harmonization in laboratory medicine: not only for clinical chemistry measurands
  4. Mini Review
  5. The effect of the immunoassay curve fitting routine on bias in troponin
  6. Opinion Papers
  7. An overview of the most important preanalytical factors influencing the clinical performance of SARS-CoV-2 antigen rapid diagnostic tests (Ag-RDTs)
  8. Time to address quality control processes applied to antibody testing for infectious diseases
  9. Perspectives
  10. Definition and application of performance specifications for measurement uncertainty of 23 common laboratory tests: linking theory to daily practice
  11. The gaps between the new EU legislation on in vitro diagnostics and the on-the-ground reality
  12. General Clinical Chemistry and Laboratory Medicine
  13. Second-generation Elecsys cerebrospinal fluid immunoassays aid diagnosis of early Alzheimer’s disease
  14. Evaluation of the interchangeability between the new fully-automated affinity chrome-mediated immunoassay (ACMIA) and the Quantitative Microsphere System (QMS) with a CE-IVD-certified LC-MS/MS assay for therapeutic drug monitoring of everolimus after solid organ transplantation
  15. Quantitative detection of anti-PLA2R antibodies targeting different epitopes and its clinical application in primary membranous nephropathy
  16. Reference Values and Biological Variations
  17. A zlog-based algorithm and tool for plausibility checks of reference intervals
  18. Harmonization of indirect reference intervals calculation by the Bhattacharya method
  19. Red blood cell parameters in early childhood: a prospective cohort study
  20. Cancer Diagnostics
  21. Association of circulating free and total oxysterols in breast cancer patients
  22. Effect of short-term storage of blood samples on gene expression in lung cancer patients
  23. Infectious Diseases
  24. Operation Moonshot: rapid translation of a SARS-CoV-2 targeted peptide immunoaffinity liquid chromatography-tandem mass spectrometry test from research into routine clinical use
  25. Seroprevalence of SARS-CoV-2 antibodies in Italy in newborn dried blood spots
  26. Positivization time of a COVID-19 rapid antigen self-test predicts SARS-CoV-2 viral load: a proof of concept
  27. New insights into SARS-CoV-2 Lumipulse G salivary antigen testing: accuracy, safety and short TAT enhance surveillance
  28. Preanalytical stability of SARS-CoV-2 anti-nucleocapsid antibodies
  29. Prognostic value of procalcitonin in cancer patients with coronavirus disease 2019
  30. 24/7 workflow for bloodstream infection diagnostics in microbiology laboratories: the first step to improve clinical management
  31. Diagnostic performance of machine learning models using cell population data for the detection of sepsis: a comparative study
  32. Diagnostic value of procalcitonin, hypersensitive C-reactive protein and neutrophil-to-lymphocyte ratio for bloodstream infections in pediatric tumor patients
  33. Letters to the Editor
  34. Pseudohyponatremia: interference of hyperglycemia on indirect potentiometry
  35. Ligand binding assay-related underestimation of 25-hydroxyvitamin D in pregnant women exaggerates the prevalence of vitamin D insufficiency
  36. Urokinase-type plasminogen activator soluble receptor (suPAR) assay in clinical routine: evaluation one year after its introduction in the high automation corelab of the A. Gemelli hospital
  37. Is this a true lambda free light chain?
  38. Effect of various blood collection tubes on serum lithium and other electrolytes: our perspective from a tertiary healthcare institute
https://doi.org/10.1515/cclm-2022-0713
Keywords for this article
artificial intelligence; cell population data; leukocytes; machine learning; Mindray BC 6800 Plus; sepsis

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Standardization and harmonization in laboratory medicine: not only for clinical chemistry measurands
  4. Mini Review
  5. The effect of the immunoassay curve fitting routine on bias in troponin
  6. Opinion Papers
  7. An overview of the most important preanalytical factors influencing the clinical performance of SARS-CoV-2 antigen rapid diagnostic tests (Ag-RDTs)
  8. Time to address quality control processes applied to antibody testing for infectious diseases
  9. Perspectives
  10. Definition and application of performance specifications for measurement uncertainty of 23 common laboratory tests: linking theory to daily practice
  11. The gaps between the new EU legislation on in vitro diagnostics and the on-the-ground reality
  12. General Clinical Chemistry and Laboratory Medicine
  13. Second-generation Elecsys cerebrospinal fluid immunoassays aid diagnosis of early Alzheimer’s disease
  14. Evaluation of the interchangeability between the new fully-automated affinity chrome-mediated immunoassay (ACMIA) and the Quantitative Microsphere System (QMS) with a CE-IVD-certified LC-MS/MS assay for therapeutic drug monitoring of everolimus after solid organ transplantation
  15. Quantitative detection of anti-PLA2R antibodies targeting different epitopes and its clinical application in primary membranous nephropathy
  16. Reference Values and Biological Variations
  17. A zlog-based algorithm and tool for plausibility checks of reference intervals
  18. Harmonization of indirect reference intervals calculation by the Bhattacharya method
  19. Red blood cell parameters in early childhood: a prospective cohort study
  20. Cancer Diagnostics
  21. Association of circulating free and total oxysterols in breast cancer patients
  22. Effect of short-term storage of blood samples on gene expression in lung cancer patients
  23. Infectious Diseases
  24. Operation Moonshot: rapid translation of a SARS-CoV-2 targeted peptide immunoaffinity liquid chromatography-tandem mass spectrometry test from research into routine clinical use
  25. Seroprevalence of SARS-CoV-2 antibodies in Italy in newborn dried blood spots
  26. Positivization time of a COVID-19 rapid antigen self-test predicts SARS-CoV-2 viral load: a proof of concept
  27. New insights into SARS-CoV-2 Lumipulse G salivary antigen testing: accuracy, safety and short TAT enhance surveillance
  28. Preanalytical stability of SARS-CoV-2 anti-nucleocapsid antibodies
  29. Prognostic value of procalcitonin in cancer patients with coronavirus disease 2019
  30. 24/7 workflow for bloodstream infection diagnostics in microbiology laboratories: the first step to improve clinical management
  31. Diagnostic performance of machine learning models using cell population data for the detection of sepsis: a comparative study
  32. Diagnostic value of procalcitonin, hypersensitive C-reactive protein and neutrophil-to-lymphocyte ratio for bloodstream infections in pediatric tumor patients
  33. Letters to the Editor
  34. Pseudohyponatremia: interference of hyperglycemia on indirect potentiometry
  35. Ligand binding assay-related underestimation of 25-hydroxyvitamin D in pregnant women exaggerates the prevalence of vitamin D insufficiency
  36. Urokinase-type plasminogen activator soluble receptor (suPAR) assay in clinical routine: evaluation one year after its introduction in the high automation corelab of the A. Gemelli hospital
  37. Is this a true lambda free light chain?
  38. Effect of various blood collection tubes on serum lithium and other electrolytes: our perspective from a tertiary healthcare institute
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 16.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cclm-2022-0713/html
Scroll to top button