Home Comparison of three chatbots as an assistant for problem-solving in clinical laboratory
Article
Licensed
Unlicensed Requires Authentication

Comparison of three chatbots as an assistant for problem-solving in clinical laboratory

  • Sedat Abusoglu ORCID logo EMAIL logo , Muhittin Serdar ORCID logo , Ali Unlu ORCID logo and Gulsum Abusoglu ORCID logo
Published/Copyright: December 14, 2023
Become an author with De Gruyter Brill
Submit Manuscript Author Information
Clinical Chemistry and Laboratory Medicine (CCLM)
From the journal Clinical Chemistry and Laboratory Medicine (CCLM) Volume 62 Issue 7

Abstract

Objectives

Data generation in clinical settings is ongoing and perpetually increasing. Artificial intelligence (AI) software may help detect data-related errors or facilitate process management. The aim of the present study was to test the extent to which the frequently encountered pre-analytical, analytical, and postanalytical errors in clinical laboratories, and likely clinical diagnoses can be detected through the use of a chatbot.

Methods

A total of 20 case scenarios, 20 multiple-choice, and 20 direct questions related to errors observed in pre-analytical, analytical, and postanalytical processes were developed in English. Difficulty assessment was performed for the 60 questions. Responses by 4 chatbots to the questions were scored in a blinded manner by 3 independent laboratory experts for accuracy, usefulness, and completeness.

Results

According to Chi-squared test, accuracy score of ChatGPT-3.5 (54.4 %) was significantly lower than CopyAI (86.7 %) (p=0.0269) and ChatGPT v4.0. (88.9 %) (p=0.0168), respectively in cases. In direct questions, there was no significant difference between ChatGPT-3.5 (67.8 %) and WriteSonic (69.4 %), ChatGPT v4.0. (78.9 %) and CopyAI (73.9 %) (p=0.914, p=0.433 and p=0.675, respectively) accuracy scores. CopyAI (90.6 %) presented significantly better performance compared to ChatGPT-3.5 (62.2 %) (p=0.036) in multiple choice questions.

Conclusions

These applications presented considerable performance to find out the cases and reply to questions. In the future, the use of AI applications is likely to increase in clinical settings if trained and validated by technical and medical experts within a structural framework.

Keywords: artificial intelligence; clinical laboratory; assistant; machine learning
Corresponding author: Prof. Sedat Abusoglu, PhD, Department of Biochemistry, Selcuk University Faculty of Medicine, Alaaddin Keykubat Campus, Postal Code: 42075 Selcuklu, Konya, Türkiye, Phone: +905370212647, E-mail:
Sedat Abusoglu and Muhittin Serdar contributed equally to this work.

  1. Research ethics: Not applicable.

  2. Informed consent: Not applicable.

  3. Author contributions: The authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  4. Competing interests: The authors state no conflict of interest.

  5. Research funding: None declared.

  6. Data availability: Not applicable.

References

1. Sallam, M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare 2023;11:1–20. https://doi.org/10.3390/healthcare11060887.Search in Google Scholar PubMed PubMed Central

2. OpenAI. Chatbot generative pre-trained transformer, ChatGPT. https://openai.com/blog/chatgpt [Accessed 6 May 2023].Search in Google Scholar

3. Cascella, M, Montomoli, J, Bellini, V, Bignami, E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst 2023;47:33. https://doi.org/10.1007/s10916-023-01925-4.Search in Google Scholar PubMed PubMed Central

4. Swathi, K. GPT 4 used in Khan academy and Duolingo to enhance their AI learning platforms. https://www.analyticsinsight.net/gpt-4-is-being-used-by-khan-academy-and-duolingo/ [Accessed 18 Apr 2023].Search in Google Scholar

5. Chen, TJ. ChatGPT and other artificial intelligence applications speed up scientific writing. J Chin Med Assoc 2023;86:351–3. https://doi.org/10.1097/jcma.0000000000000900.Search in Google Scholar PubMed

6. Sciacovelli, L, Lippi, G, Sumarac, Z, West, J, Del Pino Castro, IG, Vieira, KF, et al.. Quality indicators in laboratory medicine: the status of the progress of IFCC working group “laboratory errors and patient safety” project. Clin Chem Lab Med 2017;55:348–57. https://doi.org/10.1515/cclm-2016-0929.Search in Google Scholar PubMed

7. Rifai, N, Horvath, AR, Wittwer, C. Tietz textbook of clinical chemistry and molecular diagnostics, 6th ed. St. Louis, Missouri, USA: Elsevier; 2018.Search in Google Scholar

8. Oosterhuis, WP, Verboeket-van de Venne, WPHG. Reflective testing in primary care. http://www.reflectivetesting.com/uk/index.htm [Accessed 25 Apr 2023].Search in Google Scholar

9. Allen, LC, Dominiczak, MH, Pulkki, K, Pazzagli, M. Clinical case material for teaching clinical chemistry and laboratory medicine. Clin Chem Lab Med 2001;39:875–89. https://doi.org/10.1515/cclm.2001.145.Search in Google Scholar PubMed

10. Wang, H, Wu, W, Dou, Z, He, L, Yang, L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inf 2023;177:105173. https://doi.org/10.1016/j.ijmedinf.2023.105173.Search in Google Scholar PubMed

11. Cadamuro, J, Cabitza, F, Debeljak, Z, Bruyne, SD, Frans, G, Perez, SM, et al.. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European federation of clinical chemistry and laboratory medicine (EFLM) working group on artificial intelligence (WG-AI). Clin Chem Lab Med 2023;61:1158–66. https://doi.org/10.1515/cclm-2023-0355.Search in Google Scholar PubMed

12. Takagi, S, Watari, T, Erabi, A, Sakaguchi, K. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study. JMIR Med Educ 2023;9:e48002. https://doi.org/10.2196/48002.Search in Google Scholar PubMed PubMed Central

13. Lim, ZW, Pushpanathan, K, Yew, SME, Lai, Y, Sun, CH, Lam, JSH, et al.. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine 2023;95:104770. https://doi.org/10.1016/j.ebiom.2023.104770.Search in Google Scholar PubMed PubMed Central

14. Munoz-Zuluaga, C, Zhao, Z, Wang, F, Greenblatt, MB, Yang, HS. Assessing the accuracy and clinical utility of ChatGPT in laboratory medicine. Clin Chem 2023;69:939–40. https://doi.org/10.1093/clinchem/hvad058.Search in Google Scholar PubMed

15. World Health Organization. WHO guideline for the clinical management of exposure to lead. Geneva: CC BY-NC-SA 3.0 IGO; 2021.Search in Google Scholar

16. Temsah, O, Khan, SA, Chaiah, Y, Senjab, A, Alhasan, K, Jamal, A, et al.. Overview of early ChatGPT’s presence in medical literature: insights from a hybrid literature review by ChatGPT and human experts. Cureus 2023;15:e37281. https://doi.org/10.7759/cureus.37281.Search in Google Scholar PubMed PubMed Central

17. Padoan, A, Plebani, M. Artificial intelligence: is it the right time for clinical laboratories? Clin Chem Lab Med 2022;60:1859–61. https://doi.org/10.1515/cclm-2022-1015.Search in Google Scholar PubMed

18. Mosqueira-Rey, E, Hernández-Pereira, E, Alonso-Ríos, D, Bobes-Bascarán, J, Fernández-Leal, Á. Human-in-the-loop machine learning: a state of the art. Artif Intell Rev 2023;56:3005–54. https://doi.org/10.1007/s10462-022-10246-w.Search in Google Scholar

19. Plebani, M. Artificial intelligence in laboratory medicine: lights and shadows. Biochim Clin 2023;47:217–9.Search in Google Scholar

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/cclm-2023-1058).

Received: 2023-09-22
Accepted: 2023-12-05
Published Online: 2023-12-14
Published in Print: 2024-06-25

© 2023 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. LC-MS/MS random access automation – a game changer for the 24/7 clinical laboratory
  4. Reviews
  5. Neurofilament light protein as a biomarker for spinal muscular atrophy: a review and reference ranges
  6. Differential diagnosis of ascites: etiologies, ascitic fluid analysis, diagnostic algorithm
  7. Opinion Papers
  8. Clinical Decision Support System in laboratory medicine
  9. Blood over-testing: impact, ethical issues and mitigating actions
  10. General Clinical Chemistry and Laboratory Medicine
  11. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of zonisamide in human serum and plasma
  12. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of carbamazepine in human serum and plasma
  13. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure (RMP) for the quantification of phenobarbital in human serum and plasma
  14. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure (RMP) for the quantification of primidone in human serum and plasma
  15. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of carbamazepine-10,11-epoxide in human serum and plasma
  16. Should we depend on reference intervals from manufacturer package inserts? Comparing TSH and FT4 reference intervals from four manufacturers with results from modern indirect methods and the direct method
  17. Comparison of three chatbots as an assistant for problem-solving in clinical laboratory
  18. Evidence-based cutoffs for total and adjusted calcium: a major factor in detecting severe hypo- and hypercalcemia
  19. Minor head injury in anticoagulated patients: performance of biomarkers S100B, NSE, GFAP, UCH-L1 and Alinity TBI in the detection of intracranial injury. A prospective observational study
  20. A comparative evaluation of the analytical performances of premier resolution-high-performance liquid chromatography (PR-HPLC) with capillary zone electrophoresis (CZE) assays for the detection of hemoglobin variants and the quantitation of HbA0, A2, E, and F
  21. Get reliable laboratory findings – how to recognize the deceptive effects of angiotensin-converting enzyme inhibitor therapy in the laboratory diagnostics of sarcoidosis?
  22. Reference Values and Biological Variations
  23. Vitamin D and vitamin K status in postmenopausal women with normal and low bone mineral density
  24. Hematology and Coagulation
  25. An automatic analysis and quality assurance method for lymphocyte subset identification
  26. Cancer Diagnostics
  27. Machine learning-based delta check method for detecting misidentification errors in tumor marker tests
  28. Cardiovascular Diseases
  29. Analytical evaluation of the novel Mindray high sensitivity cardiac troponin I immunoassay on CL-1200i
  30. Infectious Diseases
  31. A reactive monocyte subset characterized by low expression of CD91 is expanded during sterile and septic inflammation
  32. Letters to the Editor
  33. Inadvertent omission of a specimen integrity comment – an overlooked post-analytical error
  34. Falsely elevated T3 due to interference of anti-T3 autoantibodies: a case report
  35. Validation of the Siemens Atellica cortisol immunoassay compared to liquid chromatography mass spectrometry in adrenal venous sampling for primary hyperaldosteronism
  36. Lessons learned from site-specific sampling and biological half-life of IGFII and IIE(68-88) peptide: a case study
  37. The added value of automated HPC count: detecting clinically important interferences on the flow cytometric CD34+ cell count
  38. Clinical pilot study on microfluidic automation of IGH-VJ library preparation for next generation sequencing
  39. Long-term effects of interventions applied to optimize the use of 25-OH vitamin D tests in primary health care
https://doi.org/10.1515/cclm-2023-1058
Keywords for this article
artificial intelligence; clinical laboratory; assistant; machine learning

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. LC-MS/MS random access automation – a game changer for the 24/7 clinical laboratory
  4. Reviews
  5. Neurofilament light protein as a biomarker for spinal muscular atrophy: a review and reference ranges
  6. Differential diagnosis of ascites: etiologies, ascitic fluid analysis, diagnostic algorithm
  7. Opinion Papers
  8. Clinical Decision Support System in laboratory medicine
  9. Blood over-testing: impact, ethical issues and mitigating actions
  10. General Clinical Chemistry and Laboratory Medicine
  11. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of zonisamide in human serum and plasma
  12. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of carbamazepine in human serum and plasma
  13. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure (RMP) for the quantification of phenobarbital in human serum and plasma
  14. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure (RMP) for the quantification of primidone in human serum and plasma
  15. An isotope dilution-liquid chromatography-tandem mass spectrometry (ID-LC-MS/MS)-based candidate reference measurement procedure for the quantification of carbamazepine-10,11-epoxide in human serum and plasma
  16. Should we depend on reference intervals from manufacturer package inserts? Comparing TSH and FT4 reference intervals from four manufacturers with results from modern indirect methods and the direct method
  17. Comparison of three chatbots as an assistant for problem-solving in clinical laboratory
  18. Evidence-based cutoffs for total and adjusted calcium: a major factor in detecting severe hypo- and hypercalcemia
  19. Minor head injury in anticoagulated patients: performance of biomarkers S100B, NSE, GFAP, UCH-L1 and Alinity TBI in the detection of intracranial injury. A prospective observational study
  20. A comparative evaluation of the analytical performances of premier resolution-high-performance liquid chromatography (PR-HPLC) with capillary zone electrophoresis (CZE) assays for the detection of hemoglobin variants and the quantitation of HbA0, A2, E, and F
  21. Get reliable laboratory findings – how to recognize the deceptive effects of angiotensin-converting enzyme inhibitor therapy in the laboratory diagnostics of sarcoidosis?
  22. Reference Values and Biological Variations
  23. Vitamin D and vitamin K status in postmenopausal women with normal and low bone mineral density
  24. Hematology and Coagulation
  25. An automatic analysis and quality assurance method for lymphocyte subset identification
  26. Cancer Diagnostics
  27. Machine learning-based delta check method for detecting misidentification errors in tumor marker tests
  28. Cardiovascular Diseases
  29. Analytical evaluation of the novel Mindray high sensitivity cardiac troponin I immunoassay on CL-1200i
  30. Infectious Diseases
  31. A reactive monocyte subset characterized by low expression of CD91 is expanded during sterile and septic inflammation
  32. Letters to the Editor
  33. Inadvertent omission of a specimen integrity comment – an overlooked post-analytical error
  34. Falsely elevated T3 due to interference of anti-T3 autoantibodies: a case report
  35. Validation of the Siemens Atellica cortisol immunoassay compared to liquid chromatography mass spectrometry in adrenal venous sampling for primary hyperaldosteronism
  36. Lessons learned from site-specific sampling and biological half-life of IGFII and IIE(68-88) peptide: a case study
  37. The added value of automated HPC count: detecting clinically important interferences on the flow cytometric CD34+ cell count
  38. Clinical pilot study on microfluidic automation of IGH-VJ library preparation for next generation sequencing
  39. Long-term effects of interventions applied to optimize the use of 25-OH vitamin D tests in primary health care
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 4.11.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cclm-2023-1058/html
Scroll to top button