Home Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese
Article
Licensed
Unlicensed Requires Authentication

Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese

  • Feier Gao ORCID logo EMAIL logo
Published/Copyright: July 11, 2024

Abstract

Crossmodal correspondence refers to the phenomenon in which individuals match stimulus features (e.g., auditory pitch) with different sensory modalities (e.g., visual size). While studies on correspondences exhibited by suprasegmentals have mostly focused on pitch-size and pitch-shape associations, audiospatial binding observed in the production and perception of Mandarin tones, where pitch of the syllable distinguishes word meanings, sheds light on the symbolic potential of auditory pitch. In the present study, a forced-choice mapping task was conducted in the form of a word guessing game, where native Mandarin listeners select the meaning of an auditory “alien” word from two visual motions. The results showed that: (1) listeners reliably match auditory tones with visual motions in the way that pitch trajectories are congruent with spatial movements, (2) vowel category impacts tone-motion correspondence when syllables are articulated in non-contour tones, and (3) the capacities in driving the tone-motion correspondence are different across tonal categories. These findings further contribute to our understanding of the sound symbolic potential of lexical tones and expand the boundary of crossmodal correspondence that can be demonstrated by pitch.


Corresponding author: Feier Gao, School of Foreign Languages, Southeast University, Nanjing, China, E-mail:

Funding source: Jiangsu Provincial Social Science Foundation Grant of China

Award Identifier / Grant number: 23YYC009

Funding source: Start-up Research Fund of Southeast University

Award Identifier / Grant number: RF1028623034

Acknowledgments

Many thanks to Stuart Davis, Xiangkun Wang, Chun Hau Ngai, and Yu-Fu Chien for their help and feedback, as well as to the two anonymous reviewers for their comments that helped improve the manuscript.

  1. Research funding: This work was suppored by Jiangsu Provincial Social Science Foundation Grant of China (23YYC009) and Start-up Research Fund of Southeast University (RF1028623034).

  2. Data availability: The stimuli, data, and R scripts for this paper are available at https://osf.io/y46bj.

References

Bates, Douglas, Martin Mächler, Ben Bolker & Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67(1). 1–48. https://doi.org/10.18637/jss.v067.i01.Search in Google Scholar

Boersma, Paul & David Weenink. 2018. Praat: Doing phonetics by computer, version 6.0.37 [Computer program]. Available at: http://www.praat.org/.Search in Google Scholar

Chang, Yen-Han, Mingxue Zhao, Yi-Chuan Chen & Pi-Chun Huang. 2021. The effects of Mandarin Chinese lexical tones in sound–shape and sound–size correspondences. Multisensory Research 35(3). 243–257. https://doi.org/10.1163/22134808-bja10068.Search in Google Scholar

Chao, Yuen Ren. 1968. A grammar of spoken Chinese. Berkeley: University of California Press.Search in Google Scholar

Chen, Trevor H. & Dominic W. Massaro. 2008. Seeing pitch: Visual information for lexical tones of Mandarin-Chinese. Journal of the Acoustical Society of America 123(4). 2356–2366. https://doi.org/10.1121/1.2839004.Search in Google Scholar

Childs, G. Tucker. 1994. African ideophones. In Leanne Hinton, Johanna Nichols & John J. Ohala (eds.), Sound symbolism, 178–204. Cambridge: Cambridge University Press.10.1017/CBO9780511751806.013Search in Google Scholar

Connell, Louise, Zhenguang G. Cai & Judith Holler. 2013. Do you see what I’m singing? Visuospatial movement biases pitch perception. Brain & Cognition 81(1). 124–130. https://doi.org/10.1016/j.bandc.2012.09.005.Search in Google Scholar

Cuskley, Christine. 2013. Mappings between linguistic sound and motion. Public Journal of Semiotics 5(1). 39–62. https://doi.org/10.37693/pjos.2013.5.9651.Search in Google Scholar

Davis, R. 1961. The fitness of names to drawings: A cross-cultural study in Tanganyika. British Journal of Psychology 52(3). 259–268. https://doi.org/10.1111/j.2044-8295.1961.tb00788.x.Search in Google Scholar

Dolscheid, Sarah, Shakila Shayan, Asifa Majid & Daniel Casasanto. 2013. The thickness of musical pitch: Psychophysical evidence for linguistic relativity. Psychological Science 24(5). 613–621. https://doi.org/10.1177/0956797612457374.Search in Google Scholar

D’Onofrio, Annette. 2014. Phonetic detail and dimensionality in sound-shape correspondences: Refining the Bouba-Kiki paradigm. Language & Speech 57(3). 367–393. https://doi.org/10.1177/0023830913507694.Search in Google Scholar

Gallace, Alberto & Charles Spence. 2006. Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics 68(7). 1191–1203. https://doi.org/10.3758/BF03193720.Search in Google Scholar

Garg, Saurabh, Ghassan Hamarneh, Allard Jongman, Joan A. Sereno & Yue Wang. 2019. Computer-vision analysis reveals facial movements made during Mandarin tone production align with pitch trajectories. Speech Communication 113. 47–62. https://doi.org/10.1016/j.specom.2019.08.003.Search in Google Scholar

Hannah, Beverly, Yue Wang, Allard Jongman, Joan A. Sereno, Jiguo Cao & Yunlong Nie. 2017. Cross-modal association between auditory and visuospatial information in Mandarin tone perception in noise by native and non-native perceivers. Frontiers in Psychology 8. 2051. https://doi.org/10.3389/fpsyg.2017.02051.Search in Google Scholar

Hinton, Leanne, Johanna Nichols & John J. Ohala (eds.). 1994. Sound symbolism. Cambridge: Cambridge University Press.10.1017/CBO9780511751806Search in Google Scholar

Holler, Judith, Linda Drijvers, Afrooz Rafiee & Asifa Majid. 2022. Embodied space-pitch associations are shaped by language. Cognitive Science 46(2). e13083. https://doi.org/10.1111/cogs.13083.Search in Google Scholar

Imai, Mutsumi, Sotaro Kita, Miho Nagumo & Hiroyuki Okada. 2008. Sound symbolism facilitates early verb learning. Cognition 109(1). 54–65. https://doi.org/10.1016/j.cognition.2008.07.015.Search in Google Scholar

Köhler, Wolfgang. 1929. Gestalt psychology. New York: Liveright.Search in Google Scholar

Kuznetsova, Alexandra, Per B. Brockhoff & Rune H. B. Christensen. 2017. lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software 82(1). 1–26. https://doi.org/10.18637/jss.v082.i13.Search in Google Scholar

Lenth, Russell V., Paul Buerkner, Maxime Herve, Jonathon Love, Hannes Riebl & Henrik Singmann. 2021. Emmeans: Estimated marginal means, aka least-squares means. https://CRAN.R-project.org/package=emmeans (accessed 8 September 2021).Search in Google Scholar

Lockwood, Gwilym & Mark Dingemanse. 2015. Iconicity in the lab: A review of behavioral, developmental, and neuroimaging research into sound-symbolism. Frontiers in Psychology 6. 1246. https://doi.org/10.3389/fpsyg.2015.01246.Search in Google Scholar

Marks, Lawrence. 1987. On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception & Performance 13. 384–394. https://doi.org/10.1037/0096-1523.13.3.384.Search in Google Scholar

McCormick, Kelly, Jee Young Kim, Sara List & Lynne Nygaard. 2015. Sound to meaning mappings in the Bouba-Kiki effect. In Proceedings of the 37th Annual Conference of the Cognitive Science Society, 1565–1570. Austin, TX: Cognitive Science Society. https://osf.io/derkz (accessed 6 June 2024).Search in Google Scholar

Morett, Laura M. & Li-Yun Chang. 2015. Emphasizing sound and meaning: Pitch gestures enhance Mandarin lexical tone acquisition. Language, Cognition & Neuroscience 30(3). 347–353. https://doi.org/10.1080/23273798.2014.923105.Search in Google Scholar

Morett, Laura M., Jacob B. Feiler & Laura M. Getz. 2022. Elucidating the influences of embodiment and conceptual metaphor on lexical and non-speech tone learning. Cognition 222. 105014. https://doi.org/10.1016/j.cognition.2022.105014.Search in Google Scholar

Nielsen, Alan & Drew Rendall. 2011. The sound of round: Evaluating the role of consonants in the classic Takete–Maluma phenomenon. Canadian Journal of Experimental Psychology 65(2). 115–124. https://doi.org/10.1037/a0022268.Search in Google Scholar

O’Boyle, Michael W. & Robert D. Tarte. 1980. Implications for phonetic symbolism: The relationship between pure tones and geometric figures. Journal of Psycholinguistic Research 9(6). 535–544. https://doi.org/10.1007/BF01068115.Search in Google Scholar

Ohala, John J. 1984. An ethological perspective on common cross-language utilization of F0 of voice. Phonetica 41(1). 1–16. https://doi.org/10.1159/000261706.Search in Google Scholar

Ohala, John J. 1995. The frequency code underlies the sound-symbolic use of voice pitch. In Johanna Nichols, John J. Ohala & Leanne Hinton (eds.), Sound symbolism, 325–347. Cambridge: Cambridge University Press.10.1017/CBO9780511751806.022Search in Google Scholar

Parise, Cesare V., Katharina Knorre & Marc O. Ernst. 2014. Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences 111(16). 6104–6108. https://doi.org/10.1073/pnas.1322705111.Search in Google Scholar

Pratt, C. C. 1930. The spatial character of high and low tones. Journal of Experimental Psychology 13. 278–285. https://doi.org/10.1037/h0072651.Search in Google Scholar

R Core Team. 2020. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.r-project.org/ (accessed 8 September 2021).Search in Google Scholar

Sadaghiani, Sepideh, Joost Maier & Uta Noppeney. 2009. Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing. Journal of Neuroscience 29(20). 6490–6499. https://doi.org/10.1523/JNEUROSCI.5437-08.2009.Search in Google Scholar

Shang, Nan & Suzy Styles. 2017. Is a high tone pointy? Speakers of different languages match Mandarin Chinese tones to visual shapes differently. Frontiers in Psychology 8. 2139 https://doi.org/10.3389/fpsyg.2017.02139.Search in Google Scholar

Shang, Nan & Suzy Styles. 2023. Implicit Association Test (IAT) studies investigating pitch-shape audiovisual cross-modal associations across language groups. Cognitive Science 47. 13221.https://doi.org/10.1111/cogs.13221.Search in Google Scholar

Shaw, Jason A., Wei-Rong Chen, Michael I. Proctor & Donald Derrick. 2016. Influences of tone on vowel articulation in Mandarin Chinese. Journal of Speech, Language, & Hearing Research 59(6). S1566–S1574. https://doi.org/10.1044/2015_JSLHR-S-15-0031.Search in Google Scholar

Shinohara, Kazuko & Shigeto Kawahara. 2010. A cross-linguistic study of sound symbolism: The images of size. Annual Meeting of the Berkeley Linguistics Society 36(1). 396–410. https://doi.org/10.3765/bls.v36i1.3926.Search in Google Scholar

Spence, Charles. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4). 971–995. https://doi.org/10.3758/s13414-010-0073-7.Search in Google Scholar

Spence, Charles & Ophelia Deroy. 2013. How automatic are crossmodal correspondences? Consciousness & Cognition 22(1). 245–260. https://doi.org/10.1016/j.concog.2012.12.006.Search in Google Scholar

Stevens, Kenneth, Samuel Jay Keyser & Haruko Kawasaki. 1986. Toward a phonetic and phonological theory of redundant features. In Joseph S. Perkell & Dennis H. Klatt (eds.), Invariance and variability in speech processes, 426–449. Hillsdale, New Jersey: Lawrence Erlbaum.Search in Google Scholar

Sun, Ching Chu, Peter Hendrix, Jianqiang Ma & Rolf Harald Baayen. 2018. Chinese lexical database (CLD). Behavior Research Methods 50(6). 2606–2629. https://doi.org/10.3758/s13428-018-1038-3.Search in Google Scholar

Thompson, Arthur Lewis. 2018. Are tones in the expressive lexicon iconic? Evidence from three Chinese languages. PLoS One 13(12). e0204270. https://doi.org/10.1371/journal.pone.0204270.Search in Google Scholar

Van Hoey, Thomas. 2024. Onomatopoeia in Mandarin Chinese. In Lívia Körtvélyessy & Pavol Štekauer (eds.), Onomatopoeia in the world’s languages, 563–575. Berlin: Mouton de Gruyter.10.1515/9783111053226-047Search in Google Scholar

Van Hoey, Thomas & Arthur Lewis Thompson. 2020. The Chinese ideophone database (CHIDEOD). Cahiers de Linguistique Asie Orientale 49(2). 136–167. https://doi.org/10.1163/19606028-bja10006.Search in Google Scholar

Xu, Yi & Q. Emily Wang. 2001. Pitch targets and their realization: Evidence from Mandarin Chinese. Speech Communication 33(4). 319–337. https://doi.org/10.1016/S0167-6393(00)00063-7.Search in Google Scholar

Yip, Moira. 2002. Tone. Cambridge: Cambridge University Press.Search in Google Scholar

Zhen, Anna, Stephen Van Hedger, Shannon Heald, Susan Goldin-Meadow & Xing Tian. 2019. Manual directional gestures facilitate cross-modal perceptual learning. Cognition 187. 178–187. https://doi.org/10.1016/j.cognition.2019.03.004.Search in Google Scholar

Received: 2023-10-10
Accepted: 2024-03-19
Published Online: 2024-07-11

© 2024 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Editorial
  3. Editorial 2024
  4. Phonetics & Phonology
  5. The role of recoverability in the implementation of non-phonemic glottalization in Hawaiian
  6. Epenthetic vowel quality crosslinguistically, with focus on Modern Hebrew
  7. Japanese speakers can infer specific sub-lexicons using phonotactic cues
  8. Articulatory phonetics in the market: combining public engagement with ultrasound data collection
  9. Investigating the acoustic fidelity of vowels across remote recording methods
  10. The role of coarticulatory tonal information in Cantonese spoken word recognition: an eye-tracking study
  11. Tracking phonological regularities: exploring the influence of learning mode and regularity locus in adult phonological learning
  12. Morphology & Syntax
  13. #AreHashtagsWords? Structure, position, and syntactic integration of hashtags in (English) tweets
  14. The meaning of morphomes: distributional semantics of Spanish stem alternations
  15. A refinement of the analysis of the resultative V-de construction in Mandarin Chinese
  16. L2 cognitive construal and morphosyntactic acquisition of pseudo-passive constructions
  17. Semantics & Pragmatics
  18. “All women are like that”: an overview of linguistic deindividualization and dehumanization of women in the incelosphere
  19. Counterfactual language, emotion, and perspective: a sentence completion study during the COVID-19 pandemic
  20. Constructing elderly patients’ agency through conversational storytelling
  21. Language Documentation & Typology
  22. Conative animal calls in Macha Oromo: function and form
  23. The syntax of African American English borrowings in the Louisiana Creole tense-mood-aspect system
  24. Syntactic pausing? Re-examining the associations
  25. Bibliographic bias and information-density sampling
  26. Historical & Comparative Linguistics
  27. Revisiting the hypothesis of ideophones as windows to language evolution
  28. Verifying the morpho-semantics of aspect via typological homogeneity
  29. Psycholinguistics & Neurolinguistics
  30. Sign recognition: the effect of parameters and features in sign mispronunciations
  31. Influence of translation on perceived metaphor features: quality, aptness, metaphoricity, and familiarity
  32. Effects of grammatical gender on gender inferences: Evidence from French hybrid nouns
  33. Processing reflexives in adjunct control: an exploration of attraction effects
  34. Language Acquisition & Language Learning
  35. How do L1 glosses affect EFL learners’ reading comprehension performance? An eye-tracking study
  36. Modeling L2 motivation change and its predictive effects on learning behaviors in the extramural digital context: a quantitative investigation in China
  37. Ongoing exposure to an ambient language continues to build implicit knowledge across the lifespan
  38. On the relationship between complexity of primary occupation and L2 varietal behavior in adult migrants in Austria
  39. The acquisition of speaking fundamental frequency (F0) features in Cantonese and English by simultaneous bilingual children
  40. Sociolinguistics & Anthropological Linguistics
  41. A computational approach to detecting the envelope of variation
  42. Attitudes toward code-switching among bilingual Jordanians: a comparative study
  43. “Let’s ride this out together”: unpacking multilingual top-down and bottom-up pandemic communication evidenced in Singapore’s coronavirus-related linguistic and semiotic landscape
  44. Across time, space, and genres: measuring probabilistic grammar distances between varieties of Mandarin
  45. Navigating linguistic ideologies and market dynamics within China’s English language teaching landscape
  46. Streetscapes and memories of real socialist anti-fascism in south-eastern Europe: between dystopianism and utopianism
  47. What can NLP do for linguistics? Towards using grammatical error analysis to document non-standard English features
  48. From sociolinguistic perception to strategic action in the study of social meaning
  49. Minority genders in quantitative survey research: a data-driven approach to clear, inclusive, and accurate gender questions
  50. Variation is the way to perfection: imperfect rhyming in Chinese hip hop
  51. Shifts in digital media usage before and after the pandemic by Rusyns in Ukraine
  52. Computational & Corpus Linguistics
  53. Revisiting the automatic prediction of lexical errors in Mandarin
  54. Finding continuers in Swedish Sign Language
  55. Conversational priming in repetitional responses as a mechanism in language change: evidence from agent-based modelling
  56. Construction grammar and procedural semantics for human-interpretable grounded language processing
  57. Through the compression glass: language complexity and the linguistic structure of compressed strings
  58. Could this be next for corpus linguistics? Methods of semi-automatic data annotation with contextualized word embeddings
  59. The Red Hen Audio Tagger
  60. Code-switching in computer-mediated communication by Gen Z Japanese Americans
  61. Supervised prediction of production patterns using machine learning algorithms
  62. Introducing Bed Word: a new automated speech recognition tool for sociolinguistic interview transcription
  63. Decoding French equivalents of the English present perfect: evidence from parallel corpora of parliamentary documents
  64. Enhancing automated essay scoring with GCNs and multi-level features for robust multidimensional assessments
  65. Sociolinguistic auto-coding has fairness problems too: measuring and mitigating bias
  66. The role of syntax in hashtag popularity
  67. Language practices of Chinese doctoral students studying abroad on social media: a translanguaging perspective
  68. Cognitive Linguistics
  69. Metaphor and gender: are words associated with source domains perceived in a gendered way?
  70. Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese
Downloaded on 17.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/lingvan-2023-0151/html
Scroll to top button