Benchmarking AI chatbots: assessing their accuracy in identifying hijacked medical journals
-
Mihály Hegedűs
,
Mehdi Dadkhah
und
Lóránt Dénes Dávid
Abstract
Objectives
The challenges posed by questionable journals to academia are very real, and being able to detect hijacked journals would be valuable to the research community. Using an artificial intelligence (AI) chatbot may be a promising approach to early detection. The purpose of this research is to analyze and benchmark the performance of different AI chatbots in identifying hijacked medical journals.
Methods
This study utilized a dataset comprising 21 previously identified hijacked journals and 10 newly detected hijacked journals, alongside their respective legitimate versions. ChatGPT, Gemini, Copilot, DeepSeek, Qwen, Perplexity, and Claude were selected for benchmarking. Three question types were developed to assess AI chatbots’ performance in providing information about hijacked journals, identifying hijacked websites, and verifying legitimate ones.
Results
The results show that current AI chatbots can provide general information about hijacked journals, but cannot reliably identify either real or hijacked journal titles. While Copilot performed better than others, it was not error-free.
Conclusions
Current AI chatbots are not yet reliable for detecting hijacked journals and may inadvertently promote them.
Funding source: Hungarian University of Agriculture and Life Sciences (MATE)
Acknowledgments
The authors would like to thank the Flagship Research Groups Programme of the Hungarian University of Agriculture and Life Sciences (MATE).
-
Research ethics: Not applicable.
-
Informed consent: Not applicable.
-
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
-
Use of Large Language Models, AI and Machine Learning Tools: AI has been used to improve the readability of the paper. The other AI usage has been clarified in the methodology section.
-
Conflict of interest: The authors state no conflict of interest.
-
Research funding: This work was supported by the Flagship Research Groups Programme of the Hungarian University of Agriculture and Life Sciences (MATE).
-
Data availability: The authors confirm that the data supporting the findings of this study are available within the article.
References
1. Albert, MA, Lalu, MM, Grudniewicz, A. Investigating the trustworthiness of research evidence used to inform public health policy: a qualitative interview study on the use of predatory journal citations in policy documents. Health Res Pol Syst 2025;23:7. https://doi.org/10.1186/s12961-024-01282-9.Suche in Google Scholar PubMed PubMed Central
2. Jalalian, M. Hijacked journals are attacking the reliability and validity of medical research. Electron Physician 2014;6. https://doi.org/10.14661/2014.925-926.Suche in Google Scholar PubMed PubMed Central
3. Jalalian, M, Mahboobi, H. Hijacked journals and predatory publishers: is there a need to re-think how to assess the quality of academic research? Walailak J Sci Technol WJST 2014;11:389–94.Suche in Google Scholar
4. Laine, C, Babski, D, Bachelet, VC, Bärnighausen, TW, Baethge, C, Bibbins-Domingo, K, et al.. Predatory journals: what can we due to protect their prey? Lancet 2025;405:362–4. https://doi.org/10.1016/s0140-6736(24)02863-0.Suche in Google Scholar PubMed
5. Sahni, V. Journal hijacking. Br Dent J 2025;238:76. https://doi.org/10.1038/s41415-025-8343-x.Suche in Google Scholar PubMed
6. Yoo, JH. How to cope with predatory journals. Jkms 2025;40:e78–0. https://doi.org/10.3346/jkms.2025.40.e78.Suche in Google Scholar PubMed PubMed Central
7. Dadkhah, M, Rahimnia, F, Rafati Niya, S, Borchardt, G. Jourchain: using blockchain to avoid questionable journals. Ir J Med Sci 1971 – 2022;191:1435–9. https://doi.org/10.1007/s11845-021-02697-x.Suche in Google Scholar PubMed
8. Beall, J. Dangerous predatory publishers threaten medical research. J Kor Med Sci 2016;31:1511. https://doi.org/10.3346/jkms.2016.31.10.1511.Suche in Google Scholar PubMed PubMed Central
9. Teixeira da Silva, JA. The ethical and academic implications of the Jeffrey Beall (www.scholarlyoa.com) blog shutdown. Sci Eng Ethics 2020;26:3465–7. https://doi.org/10.1007/s11948-017-9905-3.Suche in Google Scholar PubMed
10. Beall, J. J acad librariansh. 2013;39(6):588, https://doi.org/10.1016/j.acalib.2013.03.005.Suche in Google Scholar
11. Bisaccio, M. Cabells’ journal whitelist and blacklist: intelligent data for informed journal evaluations. Learn Publ 2018;31:243–8. https://doi.org/10.1002/leap.1164.Suche in Google Scholar
12. Kakamad, FH, Mohammed, SH, Najar, KA, Qadr, GA, Ahmed, JO, Mohammed, KK, et al.. Kscien’s list; a new strategy to hoist predatory journals and publishers. Int J Surg Open 2019;17. https://doi.org/10.1016/j.ijso.2019.01.002. [Internet] Available from: https://journals.lww.com/ijsopen/fulltext/2019/17000/kscien_s_list__a_new_strategy_to_hoist_predatory.2.aspx.Suche in Google Scholar
13. Adnan, A, Anwar, S, Zia, T, Razzaq, S, Maqbool, F, Rehman, MZU. Beyond Beall’s blacklist: automatic detection of open access predatory research journals. IEEE 2018:1692–7.10.1109/HPCC/SmartCity/DSS.2018.00274Suche in Google Scholar
14. Chen, LX, Su, SW, Liao, CH, Wong, KS, Yuan, SM. An open automation system for predatory journal detection. Sci Rep 2023;13:2976. https://doi.org/10.1038/s41598-023-30176-z.Suche in Google Scholar PubMed PubMed Central
15. Jalalian, M, Dadkhah, M. The full story of 90 hijacked journals from August 2011 to June 2015. Geogr Pannonica 2015;19:73–87. https://doi.org/10.5937/geopan1502073j.Suche in Google Scholar
16. Abalkina, A. Detecting a network of hijacked journals by its archive. Scientometrics 2021;126:7123–48. https://doi.org/10.1007/s11192-021-04056-0.Suche in Google Scholar
17. Graber, ML, Plebani, M. The growing threat of hijacked journals. Diagnosis 2024;11:219. https://doi.org/10.1515/dx-2024-0103.Suche in Google Scholar PubMed
18. Abalkina, A. Challenges posed by hijacked journals in Scopus. J Assoc Inf Sci Technol 2024;75:395–422. https://doi.org/10.1002/asi.24855.Suche in Google Scholar
19. Müller, SD, Sæbø, JI. The ‘hijacking’of the scandinavian journal of information systems: implications for the information systems community. Inf Syst J 2024;34:364–83. https://doi.org/10.1111/isj.12481.Suche in Google Scholar
20. Dadkhah, M, Maliszewski, T. Hijacked journals-threats and challenges to countries’ scientific ranking. Int J Technol Enhanc Learn (IJTEL) 2015;7:281–8. https://doi.org/10.1504/ijtel.2015.072819.Suche in Google Scholar
21. Hegedűs, M, Dadkhah, M, Dávid, LD. Unmasking greenwashing: mapping hijacked medicine journals to the sustainable development goals. Adv Pharmaceut Bull 2024;14:729–36. https://doi.org/10.34172/apb.43763.Suche in Google Scholar PubMed PubMed Central
22. Mishchuk, H, Czarkowski, JJ, Neverkovets, A, Lukács, E. Ensuring sustainable development in light of pandemic “new normal” influence. Sustainability 2023;15. https://doi.org/10.3390/su151813979.Suche in Google Scholar
23. The retraction Watch hijacked journal checker [Internet]. [cited 2025 Feb 22]. Available from: https://retractionwatch.com/the-retraction-watch-hijacked-journal-checker/.Suche in Google Scholar
24. Dadkhah, M, Oermann, MH, Hegedüs, M, Raman, R, Dávid, LD. Diagnosis unreliability of ChatGPT for journal evaluation. Adv Pharmaceut Bull 2024;14:1–4. https://doi.org/10.34172/apb.2024.020.Suche in Google Scholar PubMed PubMed Central
25. Carlini, N, Tramer, F, Wallace, E, Jagielski, M, Herbert-Voss, A, Lee, K, et al.. Extracting training data from large language models. In: 30th USENIX security symposium; 2021:2633–50 pp. [Internet] Available from: https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting.Suche in Google Scholar
26. Humphreys, D, Koay, A, Desmond, D, Mealy, E. AI hype as a cyber security risk: the moral responsibility of implementing generative AI in business. AI Ethics 2024;4:791–804. https://doi.org/10.1007/s43681-024-00443-4.Suche in Google Scholar
27. Welsby, P, Cheung, BMY. ChatGPT. Postgrad Med J 2023;99:1047–8. https://doi.org/10.1093/postmj/qgad056.Suche in Google Scholar PubMed
28. Heston, TF, Khun, C. Prompt engineering in medical education. Int Med Educ 2023;2:198–205. https://doi.org/10.3390/ime2030019.Suche in Google Scholar
29. Martínez Puertas, S, Illescas Manzano, MD, Segovia López, C, Ribeiro-Cardoso, P. Purchase intentions in a chatbot environment: an examination of the effects of customer experience. Oecon Copern 2024;15:145–94. https://doi.org/10.24136/oc.2914.Suche in Google Scholar
30. Semeraro, F, Cascella, M, Montomoli, J, Bellini, V, Bignami, EG. Comparative analysis of AI tools for disseminating CPR guidelines: implications for cardiac arrest education. Resuscitation 2025;208:110528. https://doi.org/10.1016/j.resuscitation.2025.110528.Suche in Google Scholar PubMed
31. Wu, DJ, Bibault, JE. A comparative study of Large Language models for generating summaries of breast cancer patient-reported treatment toxicities. Int J Radiat Oncol Biol Phys 2024;120:e666. https://doi.org/10.1016/j.ijrobp.2024.07.1461.Suche in Google Scholar
32. Yau, JYS, Saadat, S, Hsu, E, Murphy, LSL, Roh, JS, Suchard, J, et al.. Accuracy of prospective assessments of 4 Large Language model chatbot responses to patient questions about emergency care: experimental comparative study. J Med Internet Res 2024;26:e60291. https://doi.org/10.2196/60291.Suche in Google Scholar PubMed PubMed Central
33. Imran, M, Almusharraf, N. Google Gemini as a next generation AI educational tool: a review of emerging educational technology. Smart Learn Environ 2024;11:22. https://doi.org/10.1186/s40561-024-00310-z.Suche in Google Scholar
34. Alhur, A. Redefining healthcare with artificial intelligence (AI): the contributions of ChatGPT, Gemini, and Co-pilot. Cureus 2024;16. https://doi.org/10.7759/cureus.57795.Suche in Google Scholar PubMed PubMed Central
35. Peng, Y, Malin, BA, Rousseau, JF, Wang, Y, Xu, Z, Xu, X, et al.. From GPT to DeepSeek: significant gaps remain in realizing AI in healthcare. J Biomed Inf 2025:104791. https://doi.org/10.1016/j.jbi.2025.104791.Suche in Google Scholar PubMed
36. Tu, X, He, Z, Huang, Y, Zhang, ZH, Yang, M, Zhao, J. An overview of large AI models and their applications. Vis Intell 2024;2:34. https://doi.org/10.1007/s44267-024-00065-8.Suche in Google Scholar
37. Deike, M. Evaluating the performance of ChatGPT and perplexity AI in business reference. J Bus Finance Librarian 2024;29:125–54. https://doi.org/10.1080/08963568.2024.2317534.Suche in Google Scholar
38. Adetayo, AJ, Aborisade, MO, Sanni, BA. Microsoft Copilot and anthropic Claude AI in education and library service. Libr Hi Tech News 2024 Feb 22. https://doi.org/10.1108/LHTN-01-2024-0002 [Epub ahead of print].Suche in Google Scholar
39. Al-Moghrabi, D, Abu Arqub, S, Maroulakos, MP, Pandis, N, Fleming, PS. Can ChatGPT identify predatory biomedical and dental journals? A cross-sectional content analysis. J Dent 2024:142. [Internet] Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85183152939&doi=10.1016%2fj.jdent.2024.104840&partnerID=40&md5=87fa3adec0fc45b8e14b1cad2c55977c.Suche in Google Scholar
40. Tsigaris, P, Kendall, G, Teixeira da Silva, JA. What does ChatGPT advise about predatory publishing? J Prof Nurs 2023;49:188–9. https://doi.org/10.1016/j.profnurs.2023.08.002.Suche in Google Scholar PubMed
41. Nebl, PJ, Teixeira da Silva, JA, McCutcheon, LE. Can an artificial intelligence tool accurately classify top-ranked psychology journals as predatory or non-predatory? N Am J Psychol 2024;26:717–28.Suche in Google Scholar
42. Teixeira da Silva, JA, Kendall, G. (Mis-)Classification of 17,721 journals by an artificial intelligence predatory journal detector. Publ Res Q 2023;39:263–79. https://doi.org/10.1007/s12109-023-09956-y.Suche in Google Scholar
43. Teixeira da Silva, JA, Scelles, N. An artificial intelligence tool misclassifies sport science journals as predatory. J Sci Med Sport 2024;27:266–9. https://doi.org/10.1016/j.jsams.2023.12.006.Suche in Google Scholar PubMed
44. Dadkhah, M, Oermann, MH, Hegedűs, M, Dénes Dávid, L, Raman, R. Detecting new hijacked journals by using a list of known hijacked journals and the diagnosis of web domain data. Ser Rev 2024;50:91–6.10.1080/00987913.2024.2411664Suche in Google Scholar
© 2025 Walter de Gruyter GmbH, Berlin/Boston
