Could generative artificial intelligence replace fieldwork in pain research?

Suzana Bojic; Nemanja Radovanovic; Milica Radovic; Dusica Stamenkovic

doi:10.1515/sjpain-2023-0136

Article Open Access

Could generative artificial intelligence replace fieldwork in pain research?

Suzana Bojic , Nemanja Radovanovic , Milica Radovic and Dusica Stamenkovic

Published/Copyright: March 7, 2024

Published by

Become an author with De Gruyter Brill

Author Information Explore this Subject

From the journal Scandinavian Journal of Pain Volume 24 Issue 1

Abstract

Background

Generative artificial intelligence (AI) models offer potential assistance in pain research data acquisition, yet concerns persist regarding data accuracy and reliability. In a comparative study, we evaluated open generative AI models’ capacity to acquire data on acute pain in rock climbers comparable to field research.

Methods

Fifty-two rock climbers (33 m/19 f; age 29.0 [24.0–35.75] years) were asked to report pain location and intensity during a single climbing session. Five generative pretrained transformer models were tasked with responses to the same questions.

Results

Climbers identified the back of the forearm (19.2%) and toes (17.3%) as primary pain sites, with reported median pain intensity at 4 [3–5] and median maximum pain intensity at 7 [5–8]. Conversely, AI models yielded divergent findings, indicating fingers, hands, shoulders, legs, and feet as primary pain localizations with average and maximum pain intensity ranging from 3 to 4.4 and 5 to 10, respectively. Only two AI models provided references that were untraceable in PubMed and Google searches.

Conclusion

Our findings reveal that, currently, open generative AI models cannot match the quality of field-collected data on acute pain in rock climbers. Moreover, the models generated nonexistent references, raising concerns about their reliability.

Keywords: acute pain; artificial intelligence; climbing; generative pretrained transformer

1 Introduction

Since their introduction a year ago, open generative artificial intelligence (AI) models have had a tremendous but controversial impact on science [1]. They are the first tool available to the general public, including the scientific community, which were trained through crowdsourcing, web crawling and scraping, public datasets, user-generated content, and customer data [2], giving them access to unprecedented amount of diverse data. Although generative AI models may not be able to access information from the internet in real time, they have had access to it within a relatively close time limit up until now, during their training.

Given this fact, we pondered if generative AI could be used for data acquisition in pain research. Data acquisition is often the most time-consuming, labor-intensive, and costly aspect of any research. While using generative AI to access data holds the potential to save significant resources, concerns persist regarding the accuracy and reliability of the data obtained in this manner.

In this study, we aimed to assess if currently available open generative AI models are able to produce data on the intensity and location of acute pain in rock climbers of similar quality to data collected through standard field research.

2 Methods

This comparative study was conducted as a two-step process.

First, we performed a small cross-sectional study involving all consenting rock climbers in the field researchers’ area of residence. This part of the research was approved by an Institutional Ethics Board (decision No. 01-49/1-2022). A written informed consent was obtained from all subjects. Climbers were asked to provide detailed reports on the primary location and maximum and average intensity of pain they experienced during a single indoor training session. Average and maximum pain levels were assessed using an 11-point numeric rating scale.

Second, we employed five open AI deep learning models developed using the generative pretrained transformer (GPT) architecture. The authors used all AI models that were available within their country of residence at the time of writing. These models were instructed to provide responses to the same questions as those presented to rock climbers, so that data on localization should be presented as percentages and data on pain intensity as mean ± SD or median and interquartile range. AI models were also instructed to disclose the source of the data.

2.1 Statistical analysis

Statistical analysis was performed in SPSS Statistics Version 25 software (SPSS Inc., Chicago, IL, USA). The normality of the data was assessed with the Kolmogorov–Smirnov test. The data are presented as median and 25th–75th quartiles or frequencies. AI models were asked to report results on pain intensity as median and 25th–75th quartiles or frequencies or mean and standard deviation.

3 Results

The study included 52 rock climbers (33 males and 19 females, with an average age of 29.0 [24.0–35.75] years) with 2 [1–6] years of climbing experience. They identified the back of the forearm (19.2%), toes (17.3%), fingertips (15.4%), volar forearm (9.6%), base of the thumb (5.8%), calves, low back, anterior, lateral and posterior shoulder (3.8%, each), and foot arch, hamstrings, and lateral and medial elbow (1.9%, each) as the primary sites where they experienced pain. The reported median pain intensity was 4 [3–5], while the median maximum pain intensity was 7 [5–8].

In contrast, the AI models presented divergent findings: three models indicated fingers and hands as the primary locations of pain, one model pointed to the shoulder, and another to legs and feet. The assessed average pain intensity by the AI models varied, with three models providing ratings between 3 and 4.4. The maximum pain intensity was assessed by four of the AI models, yielding values within a range of 5–10. Notably, only two of the AI models provided references for their results, which were untraceable in both PubMed and Google searches (Table 1).

Table 1

AI-generated outcomes for acute pain intensity and localization among rock climbers

	ChatGPT (https://openai.com/blog/chatgpt)	BARD AI (https://bard.google.com/chat)	ChatSonic (https://chatsonic.pro/)	CopyAI (https://www.copy.ai/)	AiryChat (https://airychat.com/)
Average pain intensity (NRS)^a	3–4	3.7 ± 1.8^b	38%	4.4	Cannot produce number
Maximum pain intensity (NRS)	7–10	8	5	6	Cannot produce number
Localization of pain (%)	Finger joints and tendons 60–70%	Fingers and hands 40–60%	Legs and feet 50%	Shoulders	Fingers
					Hands
					Wrists
		Elbows 10–20%		Knee	Forearms
	Forearms – pumps 20–30%	Shoulders 10–20%		Feet – calcaneus, talus	Shoulders
	Elbows 5–10%	Knees 5–10%		Cannot give %	Elbow
	Shoulders 1–5%	Ankles 5–10%		Cannot give %	Cannot give %
References (Yes/No)	no	yes (4)	yes (4)	no	no
Do references exist? (Yes/No)	no	no	no	NA	NA

Values of average and maximum pain intensity, as well as pain localization, in rock climbers were generated by open AI deep learning models based on GPT architecture.

^aNRS – numeric rating scale; ^bmean ± SD; AI: artificial intelligence; GPT: generative pretrained transformer; NA – not applicable.

4 Discussion

To test the assumption that open generative AI models could be used for data acquisition in pain research, we asked them to provide the answers to the simplest pain-related questions – pain intensity and localization – and compared the answers to those acquired by field researchers. Based on our results, current iterations of open generative AI models were not able to deliver data on the intensity and location of acute pain in rock climbers that could match the quality of the data collected by field researchers.

Average pain intensity values reported by AI models were fairly similar to those found by field researchers, while maximum pain intensity levels and pain localization varied greatly. Given that a Google search using “rock climbing” and “pain” results in 62 million hits, more uniform answers regarding pain intensity between different generative AI models could have been expected.

More concerning, however, is that these AI models provided incorrect information in the form of references that did not exist, which, if not double-checked, could have a detrimental effect on research credibility. GPT models actually have 11 identified categories of failures, including reasoning, factual errors, math, coding, and bias [3]. They are also known to produce compelling disinformation [4] and even hallucinations [5], as was the case in our research.

We argue that current versions of generative AI that are available to the general public could not be reliably used for data acquisition in pain research. Although web crawling and scraping, public datasets, user-generated content, and customer data are already used in pain research, a comprehensive and free-of-charge tool, similar to open GPT models, that would be conducive to these techniques in real time is not yet available. Beta versions of AI models that facilitate reference search or identify research questions, however, are already emerging.

Acknowledgement

The study employed ChatGPT, BARD AI, ChatSonic, CopyAI, and AiryChat, as detailed in the Methods section.

Research ethics: Part of the research involving human subjects complied with all relevant national regulations and institutional policies, is in accordance with the tenets of the Helsinki Declaration (as amended in 2013), and has been approved by the authors Institutional Research Ethical Committee (No: 01-49/1-2022).
Informed consent: Informed consent has been obtained from all individuals included in this study.
Author contributions: S.B. conceived and designed the study and drafted the manuscript. M.R. and N.R. performed data acquisition, including field research and interaction with the generative AI models. All authors participated in data processing, discussed the results, and provided meaningful contribution to the manuscript writing. All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Competing interests: The authors state that there is no conflict of interest.
Research funding: The authors state that no funding is involved.
Data availability: Not applicable.

References

[1] Stokel-Walker C, Van Noorden R. ChatGPT and generative AI’s impact on science. Nature. 2023;614(7947):214–6. 10.1038/d41586-023-00340-6.Search in Google Scholar PubMed

[2] Schedlbauer J, Raptis G, Ludwig B. Web crawling, web scraping, and text mining for medical informatics labor market analysis. Int J Med Inform. 2021;150:104453. 10.1016/j.ijmedinf.2021.104453.Search in Google Scholar PubMed

[3] Borji A. A categorical archive of ChatGPT failures. ArXiv; 2023. 10.48550/arXiv.2302.03494.Search in Google Scholar

[4] Spitale G, Biller-Andorno N, Germani F. GPT-3 AI model (dis)informs better than humans. Sci Adv. 2023;9(26):eadh1850. 10.1126/sciadv.adh1850.Search in Google Scholar PubMed PubMed Central

[5] Moskatel LS, Zhang N. An observational, qualitative study on the utility of ChatGPT in assessing literature on migraine prevention. Front Neurol. 2023;14:1225223. 10.3389/fneur.2023.1225223.Search in Google Scholar PubMed PubMed Central

Received: 2023-11-22

Revised: 2023-12-08

Accepted: 2023-12-11

Published Online: 2024-03-07

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/sjpain-2023-0136

Keywords for this article

acute pain; artificial intelligence; climbing; generative pretrained transformer

Creative Commons

BY 4.0