Understanding ironic utterances: A comprehensive examination of ChatGPT-4o
-
Xu Wen
Xu Wen is Professor at Southwest University, China. His research interests are cognitive linguistics and pragmatics. His publications include
The Cognitive Foundation of Language ,The Routledge Handbook of Cognitive Linguistics , etc. He is editors-in-chief ofCognitive Linguistic Studies , andAsian-Pacific Journal of Second and Foreign Language Education , and co-editors of the book series “Bloomsbury Studies in Cognitive Linguistics”. He is President of China Cognitive Translation Society.
und
Yaling Tian
Yaling Tian is a master student at the College of International Studies, Southwest University, China. His research interests focus on cognitive linguistics, pragmatics and the integration of LLMs with foreign language teaching and research.
Abstract
As large language models (LLMs) increasingly permeate various domains of human life, their ability to accurately comprehend and appropriately respond to irony has become a critical challenge. Irony, as a linguistic phenomenon heavily reliant on contextual, cultural, and cognitive factors, places elevated demands on LLMs’ comprehension. Guided by a systematic theoretical framework, this study integrates qualitative and quantitative methods to construct a comprehensive test set. By analyzing the responses of ChatGPT-4o and comparing them with those of human participants, the study examines the model’s accuracy in understanding and responding to different types of irony. The findings reveal that both human participants and ChatGPT-4o achieved perfect accuracy in comprehension tasks involving situational irony, visual irony, and multimodal irony. However, significant difficulties were observed in the comprehension of verbal irony. Furthermore, the study explores key factors affecting ChatGPT-4o′s performance and identifies the primary mechanisms the model tends to rely on when processing ironic utterances. The results indicate that verbal irony, which requires a more sophisticated grasp of emotional tone and complex cognitive abilities, constitutes the primary factor affecting the performance of LLMs in understanding irony. Meanwhile, Grice’s maxim of quality and inferences about interpersonal relationships are the main mechanisms that LLMs tend to rely on when processing ironic utterances. The findings provide empirical support and developmental pathways for enhancing the capacity of AI systems to handle complex pragmatic phenomena. Furthermore, the study offers important insights into the integration of linguistic theory with artificial intelligence, highlighting new directions for future interdisciplinary research.
About the authors
Xu Wen is Professor at Southwest University, China. His research interests are cognitive linguistics and pragmatics. His publications include The Cognitive Foundation of Language, The Routledge Handbook of Cognitive Linguistics, etc. He is editors-in-chief of Cognitive Linguistic Studies, and Asian-Pacific Journal of Second and Foreign Language Education, and co-editors of the book series “Bloomsbury Studies in Cognitive Linguistics”. He is President of China Cognitive Translation Society.
Yaling Tian is a master student at the College of International Studies, Southwest University, China. His research interests focus on cognitive linguistics, pragmatics and the integration of LLMs with foreign language teaching and research.
-
Research funding: This research was supported by the Collaborative Innovation Team for Modern Cognitive Science and Language-Culture Studies, Chongqing Municipality (Document No. Yu Jiaoxuanfa [2020] No. 3); the Graduate Advisor Team Project of Chongqing Municipality “Language and Cognitive Science Research Team”([2019] No. 292); and the Chongqing Municipal Postgraduate Scientific Research and Innovation Project (Project No. CYS240118).
References
Choi, Minje, Jiaxin Pei, Sagar Kumar, Chang Shu & David Jurgens. 2023. Do LLMs understand social knowledge? Evaluating the sociability of large language models with socket benchmark. https://arxiv.org/abs/2305.14938 (accessed 31 August 2024).10.18653/v1/2023.emnlp-main.699Suche in Google Scholar
Chomsky, Noam. 1956. Three models for the description of language. IRE Transactions on Information Theory 2(3). 113–124.10.1109/TIT.1956.1056813Suche in Google Scholar
Chomsky, Noam. 1964. Aspects of the theory of syntax. Cambridge, MA: MIT Press.10.21236/AD0616323Suche in Google Scholar
Clark, Herbert H. & Richard J. Gerrig. 1984. On the pretense theory of irony. Journal of Experimental Psychology 113(1). 121–126. https://doi.org/10.1037/0096-3445.113.1.121.Suche in Google Scholar
Colston, Herbert L. 1997. Salting a wound or sugaring a pill: The pragmatic functions of ironic criticism. Discourse Processes 23(1). 25–45. https://doi.org/10.1080/01638539709544980.Suche in Google Scholar
de Mendoza Ibáñez, Francisco José Ruiz. 2017. Cognitive modeling and irony. In Herbert L. Colston & Angeliki Athanasiadou (eds.), Irony in language use and communication, 179–200. Amsterdam: John Benjamins.10.1075/ftl.1.09demSuche in Google Scholar
de Mendoza Ibáñez, Francisco José Ruiz & Alicia Galera Masegosa. 2014. Cognitive modeling: A linguistic perspective. Amsterdam: John Benjamins.10.1075/hcp.45Suche in Google Scholar
Floyd, Sammy. 2022. PragMega materials. Open Science Framework. https://osf.io/6abgk/ (accessed 31 August 2024).Suche in Google Scholar
Gandhi, Kanishk, Jan-Philipp Fraenken, Tobias Gerstenberg & Noah Goodman. 2023. Understanding social reasoning in language models with language models. https://arxiv.org/abs/2306.15448 (accessed 31 August 2024).Suche in Google Scholar
Gorenz, Drew & Norbert Schwarz. 2024. How funny is ChatGPT? A comparison of human-and AI-produced jokes. PLoS One 19(7). e0305364. https://doi.org/10.1371/journal.pone.0305364.Suche in Google Scholar
Grice, H. Paul. 1975. Logic and conversation. In Peter Cole & Jerry L. Morgan (eds.), Syntax and semantics, volume 3: Speech acts, 41–58. New York: Academic Press.10.1163/9789004368811_003Suche in Google Scholar
Gucman, Magda. 2016. The role of individual differences and situational factors in perception of verbal irony. Psychology of Language and Communication 20(3). 255–277. https://doi.org/10.1515/plc-2016-0016.Suche in Google Scholar
Harari, Yuval Noah. 2014. Sapiens: A brief history of humankind. New York: Random House.Suche in Google Scholar
Kierkegaard, Søren. 1965. The concept of irony. Bloomington: Indiana University Press.Suche in Google Scholar
Köder, Franziska & Ingrid Lossius Falkum. 2021. Irony and perspective-taking in children: The roles of norm violations and tone of voice. Frontiers in Psychology 12. 624604. https://doi.org/10.3389/fpsyg.2021.624604.Suche in Google Scholar
Levinson, Stephen C. 2000. Presumptive meanings: The theory of generalized conversational implicature. Cambridge, MA: The MIT Press.10.7551/mitpress/5526.001.0001Suche in Google Scholar
Lozano-Palacio, Inés & Francisco José Ruiz de Mendoza Ibáñez. 2022. Modeling irony: A cognitive-pragmatic account. Amsterdam: John Benjamins.10.1075/ftl.12Suche in Google Scholar
Lu, Jianming. 2024. The artificial intelligence language model of ChatGPT will not affect human rhetoric knowledge production paradigms. Contemporary Rhetoric(2). 18–22.Suche in Google Scholar
Lucariello, Joan. 1994. Situational irony: A concept of events gone awry. Journal of Experimental Psychology: General 123(2). 129–145. https://doi.org/10.1037//0096-3445.123.2.129.Suche in Google Scholar
Mahowald, Kyle, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum & Evelina Fedorenko. 2023. Dissociating language and thought in large language models: A cognitive perspective. Trends in Cognitive Sciences 27(7). 520–532.Suche in Google Scholar
Shapira, Natalie, Guy Zwirn & Yoav Goldberg. 2023. How well do large language models perform on faux pas tests? In Proceedings of the 61st annual Meeting of the Association for computational linguistics, 10438–10451. Toronto, Canada: Association for Computational Linguistics.10.18653/v1/2023.findings-acl.663Suche in Google Scholar
Shelley, Cameron. 2001. The bicoherence theory of situational irony. Cognitive Science 25(5). 775–818. https://doi.org/10.1207/s15516709cog2505-7.Suche in Google Scholar
Sperber, Dan & Ira A. Noveck. 2004. Introduction. In Ira A. Noveck & Dan Sperber (eds.), Experimental pragmatics, 1–22. Basingstoke: Palgrave Macmillan.10.1057/9780230524125_1Suche in Google Scholar
Sperber, Dan & Deirdre Wilson. 1986. Relevance: Communication and cognition. Cambridge: Harvard University Press.Suche in Google Scholar
Strachan, James W. A., Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, Michael S. A. Graziano & Cristina Becchio. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour 8(10). 1285–1295. https://doi.org/10.1038/s41562-024-01882-z.Suche in Google Scholar
Trott, Sean, Jones Cameron, Chang Tyler, James Michaelov & Bergen Benjamin. 2023. Do large language models know what humans know? Cognitive Science 47(7). https://doi.org/10.1111/cogs.13309.Suche in Google Scholar
van Dis, Eva A. M., Bollen Johan, Zuidema Willem, Robert van Rooij & Claudi L. Bockting. 2023. ChatGPT: Five priorities for research. Nature 614(7947). 224–226. https://doi.org/10.1038/d41586-023-00288-7.Suche in Google Scholar
van Dorst, Isolde, Mathew Gillings & Jonathan Culpeper. 2024. Sociopragmatic variation in Britain: A corpus-based study of politeness. Journal of Pragmatics 227. 37–56. https://doi.org/10.1016/j.pragma.2024.04.009.Suche in Google Scholar
Verma, Palak, Neha Shukla & A. P. Shukla. 2021. Techniques of sarcasm detection: A review. In 2021 international Conference on advance Computing and innovative Technologies in engineering (ICACITE), 968–972. Greater Noida, India: IEEE.10.1109/ICACITE51222.2021.9404585Suche in Google Scholar
Waldrop, M. Mitchell. 2024. Can ChatGPT help researchers understand how the human brain handles language? Proceedings of the National Academy of Sciences USA – PNAS 121(25). https://doi.org/10.1073/pnas.2410196121.Suche in Google Scholar
Weng, Xin, Xiaoming Jiang & Qiaoyun Liao. 2023. Using information of relationship closeness in the comprehension of Chinese ironic criticism: Evidence from behavioral experiments. Journal of Pragmatics 218. 55–69. https://doi.org/10.1016/j.pragma.2023.07.005.Suche in Google Scholar
Wu, Zixuan & Yuxia Wang. 2022. The roles of social status information in irony comprehension: An eye-tracking study. Frontiers in Psychology 13. 959397. https://doi.org/10.3389/fpsyg.2022.959397.Suche in Google Scholar
Zou, Bin & Mingjie Wang. 2021. Artificial intelligence technology and English language teaching and learning: Present and future. Foreign Language and Literature 37(3). 124–130.Suche in Google Scholar
© 2025 Walter de Gruyter GmbH, Berlin/Boston
Artikel in diesem Heft
- Frontmatter
- Research Articles
- Language acquisition in vector space
- Divergent strategies in text simplification: A comparative analysis of AI and human approaches in language processing
- AI, be less ‘stereotypical’: ChatGPT’s speech is conventional but never unique
- Understanding ironic utterances: A comprehensive examination of ChatGPT-4o
- Common ground as (inter)cultural in-betweenness in human-machine communication: A literary pragmatic perspective
- Common ground in artificial intelligence applications
- Seeking common ground with a conversational chatbot
- “ChatGPT for intercultural pragmatic learning? potentially, but not yet” – The question of using AI to develop students’ intercultural pragmatic competence
- What does “I will, yeah!” mean? Can chatbots do pragmatics?
- Improving chatbot design and intent recognition: An approach through the methods of intercultural pragmatics
Artikel in diesem Heft
- Frontmatter
- Research Articles
- Language acquisition in vector space
- Divergent strategies in text simplification: A comparative analysis of AI and human approaches in language processing
- AI, be less ‘stereotypical’: ChatGPT’s speech is conventional but never unique
- Understanding ironic utterances: A comprehensive examination of ChatGPT-4o
- Common ground as (inter)cultural in-betweenness in human-machine communication: A literary pragmatic perspective
- Common ground in artificial intelligence applications
- Seeking common ground with a conversational chatbot
- “ChatGPT for intercultural pragmatic learning? potentially, but not yet” – The question of using AI to develop students’ intercultural pragmatic competence
- What does “I will, yeah!” mean? Can chatbots do pragmatics?
- Improving chatbot design and intent recognition: An approach through the methods of intercultural pragmatics
