Skip to main content
Article
Licensed
Unlicensed Requires Authentication

Multimodal language processing in school-aged Mandarin-speaking children: the role of beat gesture in enhancing memory for discourse information

  • ORCID logo EMAIL logo and
Published/Copyright: January 23, 2025
Linguistics Vanguard
From the journal Linguistics Vanguard Volume 11 Issue 1

Abstract

Communication is typically multimodal, meaning that communication involves the utilization of various cues from different modalities. While spoken words in the auditory modality primarily convey semantic information, gestures from the visual modality complement and enhance the communication process. However, the role of visual cues, specifically beat gestures, a type of non-verbal co-speech gesture used to emphasize certain information, in language processing remains largely underexplored. The present study, using a memory task where 90 Mandarin-speaking schoolchildren aged 6–9 viewed stories individually and were later asked about what happened in the stories, investigates the role of beat gestures in enhancing the memory for discourse information in Mandarin. The results show that words with beat gestures were generally recalled better for children of all grades, indicating that by age 6, Mandarin children have already acquired the ability to utilize beat gestures to encode a discourse. This study contributes significantly to our limited understanding of the significance of visual cues in multimodal language processing.

摘要

在日常交流中,人们通常依赖多种模态的线索进行有效沟通。这些模态包括听觉和视觉,其中听觉模态通过语音传递具体的信息,而视觉模态中的手势则发挥着辅助和强化信息传达的作用。然而,关于视觉线索,特别是节拍手势(即通过强调语句中的重点信息来增强语言表达效果)对语言理解的作用,现有的研究尚不充分。本研究旨在探讨节拍手势对儿童记忆语篇信息的影响。研究对象为90名年龄在6至9岁之间、以汉语为母语的普通小学生。实验设计通过让儿童观看故事并回答相关问题的方式,评估其对语篇信息的记忆能力。研究结果表明,带有节拍手势的单词相比没有节拍手势的单词更容易被儿童记住,且这一效应在不同年级的儿童中普遍存在。这表明,大约在6岁时,儿童已经具备了将节拍手势作为辅助工具来帮助编码和记忆语篇信息的能力。本研究的结果揭示了多模态语言加工中视觉线索的重要性。

Keywords: beat gesture; memory; discourse information; Mandarin; children
关键词: 节拍手势; 记忆; 语篇信息; 普通话; 儿童
Corresponding author: Mengzhu Yan, Huazhong University of Science and Technology, Wuhan, China; and HUST-GYENNO CNS Intelligent Digital Medicine Technology Center, Wuhan, China, E-mail:

Funding source: National Office for Philosophy and Social Sciences

Award Identifier / Grant number: 21CYY014

Acknowledgements

Thanks to two reviewers, and the editor, for helpful comments on earlier versions of the paper. Thanks also to all the participants who took part.

  1. Research funding: This work is supported by “National Office for Philosophy and Social Sciences (21CYY014)”, awarded to Mengzhu Yan.

References

Akker, Evelien & Anne Cutler. 2003. Prosodic cues to semantic structure in native and nonnative listening. Bilingualism: Language and Cognition 6(2). 81–96. https://doi.org/10.1017/S1366728903001056.Search in Google Scholar

Austin, E. Elizabeth & Naomi Sweller. 2014. Presentation and production: The role of gesture in spatial communication. Journal of Experimental Child Psychology 122. 92–103. https://doi.org/10.1016/j.jecp.2013.12.008.Search in Google Scholar

Baills, Florence, Nerea Suárez-González, Santiago González-Fuente & Pilar Prieto. 2019. Observing and producing pitch gestures facilitates the learning of Mandarin Chinese tones and words. Studies in Second Language Acquisition 41(1). 33–58. https://doi.org/10.1017/S0272263118000074.Search in Google Scholar

Bates, Douglas, Reinhold Kliegl, Shravan Vasishth & Harald Baayen. 2015. Parsimonious mixed models. arXiv. https://doi.org/10.48550/arXiv.1506.04967.Search in Google Scholar

Beier, Eleonora J. & Fernanda Ferreira 2022. Replication of Cutler, A. & Fodor, J. A. (1979). Semantic focus and sentence comprehension. Cognition 7(1). 49–59. Journal of Memory and Language 126. 104339. https://doi.org/10.1016/j.jml.2022.104339.Search in Google Scholar

Bosker, Hans R. & David Peeters. 2021. Beat gestures influence which speech sounds you hear. Proceedings of the Royal Society B 288(1943). https://doi.org/10.1098/rspb.2020.2419.Search in Google Scholar

Chen, Hui-Ching, Krista Szendrői, Stephen Crain & Barbara Höhle. 2019. Understanding prosodic focus marking in Mandarin Chinese: Data from children and adults. Journal of Psycholinguistic Research 48(1). 19–32. https://doi.org/10.1007/s10936-018-9580-9.Search in Google Scholar

Cruttenden, Alan. 1985. Intonation comprehension in ten-year-olds. Journal of Child Language 12(3). 643–661. https://doi.org/10.1017/S030500090000670X.Search in Google Scholar

Cutler, Anne. 1976. Phoneme-monitoring reaction time as a function of preceding intonation contour. Attention, Perception & Psychophysics 20(1). 55–60. https://doi.org/10.3758/BF03198706.Search in Google Scholar

Cutler, Anne & Jerry A. Fodor. 1979. Semantic focus and sentence comprehension. Cognition 7(1). 49–59. https://doi.org/10.1016/0010-0277(79)90010-6.Search in Google Scholar

Cutler, Anne, Delphine Dahan & Wilma van Donselaar. 1997. Prosody in the comprehension of spoken language: A literature review. Language and Speech 40(2). 141–201. https://doi.org/10.1177/002383099704000203.Search in Google Scholar

Esteve-Gibert, Núria & Pilar Prieto. 2013. Prosodic structure shapes the temporal of intonation and manual gesture movements. Journal of Speech, Language, and Hearing Research 56. 850–864. https://doi.org/10.1044/1092-4388(2012/12-0049.Search in Google Scholar

Esteve-Gibert, Núria, Hélène Lœvenbruck, Marion Dohen & Mariapaola D’ Imperio. 2022. Preschoolers use head gestures rather than prosodic cues to highlight important information in speech. Developmental Science 25(1). e13154. https://doi.org/10.1111/desc.13154.Search in Google Scholar

Igualada, Alfonso, Núria Esteve-Gibert & Pilar Prieto. 2017. Beat gestures improve word recall in 3- to 5-year-old children. Journal of Experimental Child Psychology 156. 99–112. https://doi.org/10.1016/j.jecp.2016.11.017.Search in Google Scholar

Im, Suyeon & Stefan Baumann. 2020. Probabilistic relation between co-speech gestures, pitch accents and information status. Proceedings of the Linguistic Society of America 5, 685. https://doi.org/10.3765/plsa.v5i1.4755.Search in Google Scholar

Ito, Kiwako, Nobuyuki Jincho, Utako Minai, Naoto Yamane & Reiko Mazuka. 2012. Intonation facilitates contrast resolution: Evidence from Japanese adults and 6-year-olds. Journal of Memory and Language 66(1). 265–284. https://doi.org/10.1016/j.jml.2011.09.002.Search in Google Scholar

Ito, Kiwako, Sarah A. Bibyk, Laura Wagner & Shari R. Speer. 2014. Interpretation of contrastive pitch accent in six- to eleven-year-old English-speaking children (and adults). Journal of Child Language 41. 84–110. https://doi.org/10.1017/s0305000912000554.Search in Google Scholar

Lee, Eun-Kyung & Jesse Snedeker. 2016. Effects of contrastive accents on children’s discourse comprehension. Psychonomic Bulletin & Review 23(5). 1589–1595. https://doi.org/10.3758/s13423-016-1069-7.Search in Google Scholar

Lenth, Russell V., Henrik Singmann, Jonathon Love, Buerkner Paul & Maxime Herve. 2019. Emmeans: Estimated marginal means, aka least-squares means. version 1.4.2 [R package] https://rdocumentation.org/packages/emmeans/versions/1.4.2.Search in Google Scholar

Levantinou, Eleni I. & Costanza Navarretta. 2016. An investigation of the effect of beat and iconic gestures on memory recall in L2 speakers. In Proceedings from the 3rd European symposium on Multimodal communication, 32–37. Linköping, Sweden: Linköping University Electronic Press.Search in Google Scholar

Llanes-Coromina, Judith, Ingrid Vilà-Giménez, Olga Kushch, Joan Borràs-Comes & Pilar Prieto. 2018. Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology 172. 168–188. https://doi.org/10.1016/j.jecp.2018.02.004.Search in Google Scholar

McNeill, David.. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Search in Google Scholar

McNeill, David.. 2006. Gesture and communication. In Keith Brown, Anne H. Anderson, Laurie Bauer, Margie Berns, Graeme Hirst & Jim Miller (eds.), Encyclopedia of Language and linguistics, 2nd edn., 58–66. Boston: Elsevier.10.1016/B0-08-044854-2/00798-7Search in Google Scholar

Morett, Laura M. & Li-Yun Chang. 2015. Emphasising sound and meaning: Pitch gestures enhance Mandarin lexical tone acquisition. Language, Cognition and Neuroscience 30. 347–353. https://doi.org/10.1080/23273798.2014.923105.Search in Google Scholar

Morett, Laura M. & Scott H. Fraundorf. 2019. Listeners consider alternative speaker productions in discourse comprehension and memory: Evidence from beat gesture and pitch accenting. Memory & Cognition 47(8). 1515–1530. https://doi.org/10.3758/s13421-019-00945-1.Search in Google Scholar

Morett, Laura M., Scott H. Fraundorf & James C. McPartland. 2021. Eye see what you’re saying: Contrastive use of beat gesture and pitch accent affects online interpretation of spoken discourse. Journal of Experimental Psychology: Learning, Memory, and Cognition 47(9). 1494–1526. https://doi.org/10.1037/xlm0000986.Search in Google Scholar

Pi, Zhongling, Fangfang Zhu, Yi Zhang & Jiumin Yang. 2024. An instructor’s beat gestures facilitate second language vocabulary learning from instructional videos: Behavioral and neural evidence. Language Teaching Research 28(5). https://doi.org/10.1177/13621688211039023.Search in Google Scholar

R Core Team. 2021. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.R-project.org/.Search in Google Scholar

Rohrer, Patrick L., Elisabeth Delais-Roussarie & Pilar Prieto. 2020. Beat gestures for comprehension and recall: Differential effects of language learners and native listeners. Frontiers in Psychology 11. 575929. https://doi.org/10.3389/fpsyg.2020.575929.Search in Google Scholar

So, Wing C., Colin S. Chen-Hui & Julie L. Wei-Shan. 2012. Mnemonic effect of iconic gesture and beat gesture in adults and children: Is meaning in gesture important for memory recall? Language and Cognitive Processes 27. 665–681. https://doi.org/10.1080/01690965.2011.573220.Search in Google Scholar

Sweller, John. 2011. Cognitive load theory. Psychology of Learning and Motivation 55. 37–76. https://doi.org/10.1016/b978-0-12-387691-1.00002-8.Search in Google Scholar

Swerts, Marc & Emiel Krahmer. 2008. Facial expression and prosodic prominence: Effects of modality and facial area. Journal of Phonetics 36(2). 219–238. https://doi.org/10.1016/j.wocn.2007.05.001.Search in Google Scholar

Szendrői, Kriszta, Carline Bernard, Frauke Berger, Judit Gervain & Barbara Höhle. 2018. Acquisition of prosodic focus marking by English, French, and German three-four-five- and six-year-olds. Journal of Child Language 45(1). 219–241. https://doi.org/10.1017/S0305000917000071.Search in Google Scholar

Tang, Ping, Ivan Yuen, Katherine Demuth & Nan Xu Rattanasone. 2023. The acquisition of contrastive focus during online sentence-comprehension by children learning Mandarin Chinese. Developmental Psychology 59(5). 845–861. https://doi.org/10.1037/dev0001498.Search in Google Scholar

Wang, Lin & Mingyuan Chu. 2013. The role of beat gesture and pitch accent in semantic processing: An ERP study. Neuropsychologia 51(13). 2847–2855. https://doi.org/10.1016/j.neuropsychologia.2013.09.027.Search in Google Scholar

Wells, Bill, Sue Peppé & Nata Goulandris. 2004. Intonation development from five to thirteen. Journal of Child Language 31(4). 749–778. https://doi.org/10.1017/S030500090400652X.Search in Google Scholar

Yan, Mengzhu & Sasha Calhoun. 2020. Rejecting false alternatives in Chinese and English: The interaction of prosody, syntax and default focus position. Laboratory Phonology 11(1). https://doi.org/10.5334/labphon.255.Search in Google Scholar

Yan, Mengzhu & Sasha Calhoun. 2023. The role of prosody and beat gesture in enhancing memory for discourse information in Mandarin. In Radek Skarnitzl & Jan Volín (eds.), Proceedings of the 20th international congress of phonetic Sciences, Prague, Czech republic 2023, 1330–1334. Prague: Guarant International.Search in Google Scholar
Received: 2023-10-05
Accepted: 2024-12-10
Published Online: 2025-01-23

© 2025 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Editorial 2025
  4. Research Articles
  5. Vowel formant track normalization using discrete cosine transform coefficients
  6. Asymmetry in French speech-in-noise perception: the effects of native dialect and cross-dialectal exposure
  7. Direct pseudo-partitives in US English
  8. A baseline for object clitic climbing in Italian
  9. Semantic granularity in derivation
  10. Shared processing strategies as a mechanism for contact-induced change in flexible constituent order
  11. The (non)canonical status of the ka- passive in Balinese
  12. A comparative study of 时 si 2 /shi 2 in Meixian Hakka and Ancient Chinese using the Minimalist Program
  13. A quantitative method for syntactic gradience: words, phrases, and the constructions in between
  14. Yeah, but how? Operationalizing the functions of the discourse-pragmatic marker yeah
  15. Hotspots for acoustic politeness in Korean and Japanese deferential speech
  16. How fast is fast and how slow is slow in mental simulation? Two rating studies on Estonian speed adverbs
  17. Discourse effects in processing Chinese reflexive pronouns
  18. Attitudinal negotiation: the analysis of online commentary videos about an international event on Chinese social media platform bilibili.com
  19. Crosslinguistic constructions and strategies: where do concessive conditionals fit in?
  20. Recurring patterns in tone (chain) shift
  21. Null pronoun interpretation probed via thematic role ambiguity: a case in Korean
  22. Experimental investigation on quantifier scope in Chinese relative clauses
  23. Sensitivity to honorific agreement: a window into predictive processing
  24. The negative concord illusion: an acceptability study with Czech neg-words
  25. Expletive negation in Italian temporal clauses: an acceptability judgement and a self-paced reading study
  26. Effects of information structure on pronoun resolution: the number of pronouns matters
  27. The cognitive processing of nouns and verbs in second language reading: an eye-tracking study
  28. Comprehension of conversational implicatures in L3 Mandarin
  29. Effects of crosslinguistic influence in definiteness acquisition: comparing HL-English and HL-Russian bilingual children acquiring Hebrew
  30. Multimodal language processing in school-aged Mandarin-speaking children: the role of beat gesture in enhancing memory for discourse information
  31. My Memoji, my self: prosodic correlates of online performed code-switching via avatar
  32. Gender effects in Mandarin creaky voice evaluation: a matched-guise study
  33. Narrating the doctoral journey on Chinese social media: chronotopes and scales in user interaction on Xiaohongshu
  34. Salient Language in Context (SLIC): a web app for collecting real-time attention data in response to audio samples
  35. Children’s emerging sociolinguistic expectations around social roles: a triangulated approach
  36. Situating speakers in change: a methodology for quantifying degree and direction of change over the lifespan
  37. Testing the effect of speech separation on vowel formant estimates
  38. Researching dialects with high school students: a citizen science approach
  39. Sociolinguistic research projects as brands
  40. Do readers perceive various types of knowledge expressed through evidentials in news reports with different degrees of certainty?
  41. Quantitative relationship between distribution of sentence length and dependency distance in Spanish
  42. Large corpora and large language models: a replicable method for automating grammatical annotation
  43. Using ATLAS.ti for constructing and analysing multimodal social media corpora
  44. Exploring the effect of semantic diversity on boundary permeability in verb/noun heterosemy using deep contextualized word embedding
  45. Communicative pressures influence the use of adverbs as well as adjectives: evidence from a crosslinguistic investigation
  46. Non-signers favor two-handed gestures when expressing inherently plural meanings
  47. Encoding Chinese metaphorical motion: a typological perspective
  48. Frequency does not predict the processing speed of multi-morpheme sequences in Japanese
  49. Did he lead monologues or did he talk to himself? How typological distance between source and target language influences the preservation of metaphorical mappings in translation
  50. How long is too long? Production-internal and communicative constraints in the coding of conditionality in Spanish
  51. Long English objects and short Chinese objects: language diversity shaped by cognitive universality
  52. Corrigendum
  53. Corrigendum to: Sign recognition: the effect of parameters and features in sign mispronunciations
https://doi.org/10.1515/lingvan-2023-0150
Keywords for this article
beat gesture; memory; discourse information; Mandarin; children

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Editorial 2025
  4. Research Articles
  5. Vowel formant track normalization using discrete cosine transform coefficients
  6. Asymmetry in French speech-in-noise perception: the effects of native dialect and cross-dialectal exposure
  7. Direct pseudo-partitives in US English
  8. A baseline for object clitic climbing in Italian
  9. Semantic granularity in derivation
  10. Shared processing strategies as a mechanism for contact-induced change in flexible constituent order
  11. The (non)canonical status of the ka- passive in Balinese
  12. A comparative study of 时 si 2 /shi 2 in Meixian Hakka and Ancient Chinese using the Minimalist Program
  13. A quantitative method for syntactic gradience: words, phrases, and the constructions in between
  14. Yeah, but how? Operationalizing the functions of the discourse-pragmatic marker yeah
  15. Hotspots for acoustic politeness in Korean and Japanese deferential speech
  16. How fast is fast and how slow is slow in mental simulation? Two rating studies on Estonian speed adverbs
  17. Discourse effects in processing Chinese reflexive pronouns
  18. Attitudinal negotiation: the analysis of online commentary videos about an international event on Chinese social media platform bilibili.com
  19. Crosslinguistic constructions and strategies: where do concessive conditionals fit in?
  20. Recurring patterns in tone (chain) shift
  21. Null pronoun interpretation probed via thematic role ambiguity: a case in Korean
  22. Experimental investigation on quantifier scope in Chinese relative clauses
  23. Sensitivity to honorific agreement: a window into predictive processing
  24. The negative concord illusion: an acceptability study with Czech neg-words
  25. Expletive negation in Italian temporal clauses: an acceptability judgement and a self-paced reading study
  26. Effects of information structure on pronoun resolution: the number of pronouns matters
  27. The cognitive processing of nouns and verbs in second language reading: an eye-tracking study
  28. Comprehension of conversational implicatures in L3 Mandarin
  29. Effects of crosslinguistic influence in definiteness acquisition: comparing HL-English and HL-Russian bilingual children acquiring Hebrew
  30. Multimodal language processing in school-aged Mandarin-speaking children: the role of beat gesture in enhancing memory for discourse information
  31. My Memoji, my self: prosodic correlates of online performed code-switching via avatar
  32. Gender effects in Mandarin creaky voice evaluation: a matched-guise study
  33. Narrating the doctoral journey on Chinese social media: chronotopes and scales in user interaction on Xiaohongshu
  34. Salient Language in Context (SLIC): a web app for collecting real-time attention data in response to audio samples
  35. Children’s emerging sociolinguistic expectations around social roles: a triangulated approach
  36. Situating speakers in change: a methodology for quantifying degree and direction of change over the lifespan
  37. Testing the effect of speech separation on vowel formant estimates
  38. Researching dialects with high school students: a citizen science approach
  39. Sociolinguistic research projects as brands
  40. Do readers perceive various types of knowledge expressed through evidentials in news reports with different degrees of certainty?
  41. Quantitative relationship between distribution of sentence length and dependency distance in Spanish
  42. Large corpora and large language models: a replicable method for automating grammatical annotation
  43. Using ATLAS.ti for constructing and analysing multimodal social media corpora
  44. Exploring the effect of semantic diversity on boundary permeability in verb/noun heterosemy using deep contextualized word embedding
  45. Communicative pressures influence the use of adverbs as well as adjectives: evidence from a crosslinguistic investigation
  46. Non-signers favor two-handed gestures when expressing inherently plural meanings
  47. Encoding Chinese metaphorical motion: a typological perspective
  48. Frequency does not predict the processing speed of multi-morpheme sequences in Japanese
  49. Did he lead monologues or did he talk to himself? How typological distance between source and target language influences the preservation of metaphorical mappings in translation
  50. How long is too long? Production-internal and communicative constraints in the coding of conditionality in Spanish
  51. Long English objects and short Chinese objects: language diversity shaped by cognitive universality
  52. Corrigendum
  53. Corrigendum to: Sign recognition: the effect of parameters and features in sign mispronunciations
Downloaded on 28.4.2026 from https://www.degruyterbrill.com/document/doi/10.1515/lingvan-2023-0150/html?lang=en
Scroll to top button