Abstract
Crossmodal correspondence refers to the phenomenon in which individuals match stimulus features (e.g., auditory pitch) with different sensory modalities (e.g., visual size). While studies on correspondences exhibited by suprasegmentals have mostly focused on pitch-size and pitch-shape associations, audiospatial binding observed in the production and perception of Mandarin tones, where pitch of the syllable distinguishes word meanings, sheds light on the symbolic potential of auditory pitch. In the present study, a forced-choice mapping task was conducted in the form of a word guessing game, where native Mandarin listeners select the meaning of an auditory “alien” word from two visual motions. The results showed that: (1) listeners reliably match auditory tones with visual motions in the way that pitch trajectories are congruent with spatial movements, (2) vowel category impacts tone-motion correspondence when syllables are articulated in non-contour tones, and (3) the capacities in driving the tone-motion correspondence are different across tonal categories. These findings further contribute to our understanding of the sound symbolic potential of lexical tones and expand the boundary of crossmodal correspondence that can be demonstrated by pitch.
Funding source: Jiangsu Provincial Social Science Foundation Grant of China
Award Identifier / Grant number: 23YYC009
Funding source: Start-up Research Fund of Southeast University
Award Identifier / Grant number: RF1028623034
Acknowledgments
Many thanks to Stuart Davis, Xiangkun Wang, Chun Hau Ngai, and Yu-Fu Chien for their help and feedback, as well as to the two anonymous reviewers for their comments that helped improve the manuscript.
-
Research funding: This work was suppored by Jiangsu Provincial Social Science Foundation Grant of China (23YYC009) and Start-up Research Fund of Southeast University (RF1028623034).
-
Data availability: The stimuli, data, and R scripts for this paper are available at https://osf.io/y46bj.
References
Bates, Douglas, Martin Mächler, Ben Bolker & Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67(1). 1–48. https://doi.org/10.18637/jss.v067.i01.Search in Google Scholar
Boersma, Paul & David Weenink. 2018. Praat: Doing phonetics by computer, version 6.0.37 [Computer program]. Available at: http://www.praat.org/.Search in Google Scholar
Chang, Yen-Han, Mingxue Zhao, Yi-Chuan Chen & Pi-Chun Huang. 2021. The effects of Mandarin Chinese lexical tones in sound–shape and sound–size correspondences. Multisensory Research 35(3). 243–257. https://doi.org/10.1163/22134808-bja10068.Search in Google Scholar
Chao, Yuen Ren. 1968. A grammar of spoken Chinese. Berkeley: University of California Press.Search in Google Scholar
Chen, Trevor H. & Dominic W. Massaro. 2008. Seeing pitch: Visual information for lexical tones of Mandarin-Chinese. Journal of the Acoustical Society of America 123(4). 2356–2366. https://doi.org/10.1121/1.2839004.Search in Google Scholar
Childs, G. Tucker. 1994. African ideophones. In Leanne Hinton, Johanna Nichols & John J. Ohala (eds.), Sound symbolism, 178–204. Cambridge: Cambridge University Press.10.1017/CBO9780511751806.013Search in Google Scholar
Connell, Louise, Zhenguang G. Cai & Judith Holler. 2013. Do you see what I’m singing? Visuospatial movement biases pitch perception. Brain & Cognition 81(1). 124–130. https://doi.org/10.1016/j.bandc.2012.09.005.Search in Google Scholar
Cuskley, Christine. 2013. Mappings between linguistic sound and motion. Public Journal of Semiotics 5(1). 39–62. https://doi.org/10.37693/pjos.2013.5.9651.Search in Google Scholar
Davis, R. 1961. The fitness of names to drawings: A cross-cultural study in Tanganyika. British Journal of Psychology 52(3). 259–268. https://doi.org/10.1111/j.2044-8295.1961.tb00788.x.Search in Google Scholar
Dolscheid, Sarah, Shakila Shayan, Asifa Majid & Daniel Casasanto. 2013. The thickness of musical pitch: Psychophysical evidence for linguistic relativity. Psychological Science 24(5). 613–621. https://doi.org/10.1177/0956797612457374.Search in Google Scholar
D’Onofrio, Annette. 2014. Phonetic detail and dimensionality in sound-shape correspondences: Refining the Bouba-Kiki paradigm. Language & Speech 57(3). 367–393. https://doi.org/10.1177/0023830913507694.Search in Google Scholar
Gallace, Alberto & Charles Spence. 2006. Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics 68(7). 1191–1203. https://doi.org/10.3758/BF03193720.Search in Google Scholar
Garg, Saurabh, Ghassan Hamarneh, Allard Jongman, Joan A. Sereno & Yue Wang. 2019. Computer-vision analysis reveals facial movements made during Mandarin tone production align with pitch trajectories. Speech Communication 113. 47–62. https://doi.org/10.1016/j.specom.2019.08.003.Search in Google Scholar
Hannah, Beverly, Yue Wang, Allard Jongman, Joan A. Sereno, Jiguo Cao & Yunlong Nie. 2017. Cross-modal association between auditory and visuospatial information in Mandarin tone perception in noise by native and non-native perceivers. Frontiers in Psychology 8. 2051. https://doi.org/10.3389/fpsyg.2017.02051.Search in Google Scholar
Hinton, Leanne, Johanna Nichols & John J. Ohala (eds.). 1994. Sound symbolism. Cambridge: Cambridge University Press.10.1017/CBO9780511751806Search in Google Scholar
Holler, Judith, Linda Drijvers, Afrooz Rafiee & Asifa Majid. 2022. Embodied space-pitch associations are shaped by language. Cognitive Science 46(2). e13083. https://doi.org/10.1111/cogs.13083.Search in Google Scholar
Imai, Mutsumi, Sotaro Kita, Miho Nagumo & Hiroyuki Okada. 2008. Sound symbolism facilitates early verb learning. Cognition 109(1). 54–65. https://doi.org/10.1016/j.cognition.2008.07.015.Search in Google Scholar
Köhler, Wolfgang. 1929. Gestalt psychology. New York: Liveright.Search in Google Scholar
Kuznetsova, Alexandra, Per B. Brockhoff & Rune H. B. Christensen. 2017. lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software 82(1). 1–26. https://doi.org/10.18637/jss.v082.i13.Search in Google Scholar
Lenth, Russell V., Paul Buerkner, Maxime Herve, Jonathon Love, Hannes Riebl & Henrik Singmann. 2021. Emmeans: Estimated marginal means, aka least-squares means. https://CRAN.R-project.org/package=emmeans (accessed 8 September 2021).Search in Google Scholar
Lockwood, Gwilym & Mark Dingemanse. 2015. Iconicity in the lab: A review of behavioral, developmental, and neuroimaging research into sound-symbolism. Frontiers in Psychology 6. 1246. https://doi.org/10.3389/fpsyg.2015.01246.Search in Google Scholar
Marks, Lawrence. 1987. On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception & Performance 13. 384–394. https://doi.org/10.1037/0096-1523.13.3.384.Search in Google Scholar
McCormick, Kelly, Jee Young Kim, Sara List & Lynne Nygaard. 2015. Sound to meaning mappings in the Bouba-Kiki effect. In Proceedings of the 37th Annual Conference of the Cognitive Science Society, 1565–1570. Austin, TX: Cognitive Science Society. https://osf.io/derkz (accessed 6 June 2024).Search in Google Scholar
Morett, Laura M. & Li-Yun Chang. 2015. Emphasizing sound and meaning: Pitch gestures enhance Mandarin lexical tone acquisition. Language, Cognition & Neuroscience 30(3). 347–353. https://doi.org/10.1080/23273798.2014.923105.Search in Google Scholar
Morett, Laura M., Jacob B. Feiler & Laura M. Getz. 2022. Elucidating the influences of embodiment and conceptual metaphor on lexical and non-speech tone learning. Cognition 222. 105014. https://doi.org/10.1016/j.cognition.2022.105014.Search in Google Scholar
Nielsen, Alan & Drew Rendall. 2011. The sound of round: Evaluating the role of consonants in the classic Takete–Maluma phenomenon. Canadian Journal of Experimental Psychology 65(2). 115–124. https://doi.org/10.1037/a0022268.Search in Google Scholar
O’Boyle, Michael W. & Robert D. Tarte. 1980. Implications for phonetic symbolism: The relationship between pure tones and geometric figures. Journal of Psycholinguistic Research 9(6). 535–544. https://doi.org/10.1007/BF01068115.Search in Google Scholar
Ohala, John J. 1984. An ethological perspective on common cross-language utilization of F0 of voice. Phonetica 41(1). 1–16. https://doi.org/10.1159/000261706.Search in Google Scholar
Ohala, John J. 1995. The frequency code underlies the sound-symbolic use of voice pitch. In Johanna Nichols, John J. Ohala & Leanne Hinton (eds.), Sound symbolism, 325–347. Cambridge: Cambridge University Press.10.1017/CBO9780511751806.022Search in Google Scholar
Parise, Cesare V., Katharina Knorre & Marc O. Ernst. 2014. Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences 111(16). 6104–6108. https://doi.org/10.1073/pnas.1322705111.Search in Google Scholar
Pratt, C. C. 1930. The spatial character of high and low tones. Journal of Experimental Psychology 13. 278–285. https://doi.org/10.1037/h0072651.Search in Google Scholar
R Core Team. 2020. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. https://www.r-project.org/ (accessed 8 September 2021).Search in Google Scholar
Sadaghiani, Sepideh, Joost Maier & Uta Noppeney. 2009. Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing. Journal of Neuroscience 29(20). 6490–6499. https://doi.org/10.1523/JNEUROSCI.5437-08.2009.Search in Google Scholar
Shang, Nan & Suzy Styles. 2017. Is a high tone pointy? Speakers of different languages match Mandarin Chinese tones to visual shapes differently. Frontiers in Psychology 8. 2139 https://doi.org/10.3389/fpsyg.2017.02139.Search in Google Scholar
Shang, Nan & Suzy Styles. 2023. Implicit Association Test (IAT) studies investigating pitch-shape audiovisual cross-modal associations across language groups. Cognitive Science 47. 13221.https://doi.org/10.1111/cogs.13221.Search in Google Scholar
Shaw, Jason A., Wei-Rong Chen, Michael I. Proctor & Donald Derrick. 2016. Influences of tone on vowel articulation in Mandarin Chinese. Journal of Speech, Language, & Hearing Research 59(6). S1566–S1574. https://doi.org/10.1044/2015_JSLHR-S-15-0031.Search in Google Scholar
Shinohara, Kazuko & Shigeto Kawahara. 2010. A cross-linguistic study of sound symbolism: The images of size. Annual Meeting of the Berkeley Linguistics Society 36(1). 396–410. https://doi.org/10.3765/bls.v36i1.3926.Search in Google Scholar
Spence, Charles. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4). 971–995. https://doi.org/10.3758/s13414-010-0073-7.Search in Google Scholar
Spence, Charles & Ophelia Deroy. 2013. How automatic are crossmodal correspondences? Consciousness & Cognition 22(1). 245–260. https://doi.org/10.1016/j.concog.2012.12.006.Search in Google Scholar
Stevens, Kenneth, Samuel Jay Keyser & Haruko Kawasaki. 1986. Toward a phonetic and phonological theory of redundant features. In Joseph S. Perkell & Dennis H. Klatt (eds.), Invariance and variability in speech processes, 426–449. Hillsdale, New Jersey: Lawrence Erlbaum.Search in Google Scholar
Sun, Ching Chu, Peter Hendrix, Jianqiang Ma & Rolf Harald Baayen. 2018. Chinese lexical database (CLD). Behavior Research Methods 50(6). 2606–2629. https://doi.org/10.3758/s13428-018-1038-3.Search in Google Scholar
Thompson, Arthur Lewis. 2018. Are tones in the expressive lexicon iconic? Evidence from three Chinese languages. PLoS One 13(12). e0204270. https://doi.org/10.1371/journal.pone.0204270.Search in Google Scholar
Van Hoey, Thomas. 2024. Onomatopoeia in Mandarin Chinese. In Lívia Körtvélyessy & Pavol Štekauer (eds.), Onomatopoeia in the world’s languages, 563–575. Berlin: Mouton de Gruyter.10.1515/9783111053226-047Search in Google Scholar
Van Hoey, Thomas & Arthur Lewis Thompson. 2020. The Chinese ideophone database (CHIDEOD). Cahiers de Linguistique Asie Orientale 49(2). 136–167. https://doi.org/10.1163/19606028-bja10006.Search in Google Scholar
Xu, Yi & Q. Emily Wang. 2001. Pitch targets and their realization: Evidence from Mandarin Chinese. Speech Communication 33(4). 319–337. https://doi.org/10.1016/S0167-6393(00)00063-7.Search in Google Scholar
Yip, Moira. 2002. Tone. Cambridge: Cambridge University Press.Search in Google Scholar
Zhen, Anna, Stephen Van Hedger, Shannon Heald, Susan Goldin-Meadow & Xing Tian. 2019. Manual directional gestures facilitate cross-modal perceptual learning. Cognition 187. 178–187. https://doi.org/10.1016/j.cognition.2019.03.004.Search in Google Scholar
© 2024 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Editorial
- Editorial 2024
- Phonetics & Phonology
- The role of recoverability in the implementation of non-phonemic glottalization in Hawaiian
- Epenthetic vowel quality crosslinguistically, with focus on Modern Hebrew
- Japanese speakers can infer specific sub-lexicons using phonotactic cues
- Articulatory phonetics in the market: combining public engagement with ultrasound data collection
- Investigating the acoustic fidelity of vowels across remote recording methods
- The role of coarticulatory tonal information in Cantonese spoken word recognition: an eye-tracking study
- Tracking phonological regularities: exploring the influence of learning mode and regularity locus in adult phonological learning
- Morphology & Syntax
- #AreHashtagsWords? Structure, position, and syntactic integration of hashtags in (English) tweets
- The meaning of morphomes: distributional semantics of Spanish stem alternations
- A refinement of the analysis of the resultative V-de construction in Mandarin Chinese
- L2 cognitive construal and morphosyntactic acquisition of pseudo-passive constructions
- Semantics & Pragmatics
- “All women are like that”: an overview of linguistic deindividualization and dehumanization of women in the incelosphere
- Counterfactual language, emotion, and perspective: a sentence completion study during the COVID-19 pandemic
- Constructing elderly patients’ agency through conversational storytelling
- Language Documentation & Typology
- Conative animal calls in Macha Oromo: function and form
- The syntax of African American English borrowings in the Louisiana Creole tense-mood-aspect system
- Syntactic pausing? Re-examining the associations
- Bibliographic bias and information-density sampling
- Historical & Comparative Linguistics
- Revisiting the hypothesis of ideophones as windows to language evolution
- Verifying the morpho-semantics of aspect via typological homogeneity
- Psycholinguistics & Neurolinguistics
- Sign recognition: the effect of parameters and features in sign mispronunciations
- Influence of translation on perceived metaphor features: quality, aptness, metaphoricity, and familiarity
- Effects of grammatical gender on gender inferences: Evidence from French hybrid nouns
- Processing reflexives in adjunct control: an exploration of attraction effects
- Language Acquisition & Language Learning
- How do L1 glosses affect EFL learners’ reading comprehension performance? An eye-tracking study
- Modeling L2 motivation change and its predictive effects on learning behaviors in the extramural digital context: a quantitative investigation in China
- Ongoing exposure to an ambient language continues to build implicit knowledge across the lifespan
- On the relationship between complexity of primary occupation and L2 varietal behavior in adult migrants in Austria
- The acquisition of speaking fundamental frequency (F0) features in Cantonese and English by simultaneous bilingual children
- Sociolinguistics & Anthropological Linguistics
- A computational approach to detecting the envelope of variation
- Attitudes toward code-switching among bilingual Jordanians: a comparative study
- “Let’s ride this out together”: unpacking multilingual top-down and bottom-up pandemic communication evidenced in Singapore’s coronavirus-related linguistic and semiotic landscape
- Across time, space, and genres: measuring probabilistic grammar distances between varieties of Mandarin
- Navigating linguistic ideologies and market dynamics within China’s English language teaching landscape
- Streetscapes and memories of real socialist anti-fascism in south-eastern Europe: between dystopianism and utopianism
- What can NLP do for linguistics? Towards using grammatical error analysis to document non-standard English features
- From sociolinguistic perception to strategic action in the study of social meaning
- Minority genders in quantitative survey research: a data-driven approach to clear, inclusive, and accurate gender questions
- Variation is the way to perfection: imperfect rhyming in Chinese hip hop
- Shifts in digital media usage before and after the pandemic by Rusyns in Ukraine
- Computational & Corpus Linguistics
- Revisiting the automatic prediction of lexical errors in Mandarin
- Finding continuers in Swedish Sign Language
- Conversational priming in repetitional responses as a mechanism in language change: evidence from agent-based modelling
- Construction grammar and procedural semantics for human-interpretable grounded language processing
- Through the compression glass: language complexity and the linguistic structure of compressed strings
- Could this be next for corpus linguistics? Methods of semi-automatic data annotation with contextualized word embeddings
- The Red Hen Audio Tagger
- Code-switching in computer-mediated communication by Gen Z Japanese Americans
- Supervised prediction of production patterns using machine learning algorithms
- Introducing Bed Word: a new automated speech recognition tool for sociolinguistic interview transcription
- Decoding French equivalents of the English present perfect: evidence from parallel corpora of parliamentary documents
- Enhancing automated essay scoring with GCNs and multi-level features for robust multidimensional assessments
- Sociolinguistic auto-coding has fairness problems too: measuring and mitigating bias
- The role of syntax in hashtag popularity
- Language practices of Chinese doctoral students studying abroad on social media: a translanguaging perspective
- Cognitive Linguistics
- Metaphor and gender: are words associated with source domains perceived in a gendered way?
- Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese
Articles in the same Issue
- Frontmatter
- Editorial
- Editorial 2024
- Phonetics & Phonology
- The role of recoverability in the implementation of non-phonemic glottalization in Hawaiian
- Epenthetic vowel quality crosslinguistically, with focus on Modern Hebrew
- Japanese speakers can infer specific sub-lexicons using phonotactic cues
- Articulatory phonetics in the market: combining public engagement with ultrasound data collection
- Investigating the acoustic fidelity of vowels across remote recording methods
- The role of coarticulatory tonal information in Cantonese spoken word recognition: an eye-tracking study
- Tracking phonological regularities: exploring the influence of learning mode and regularity locus in adult phonological learning
- Morphology & Syntax
- #AreHashtagsWords? Structure, position, and syntactic integration of hashtags in (English) tweets
- The meaning of morphomes: distributional semantics of Spanish stem alternations
- A refinement of the analysis of the resultative V-de construction in Mandarin Chinese
- L2 cognitive construal and morphosyntactic acquisition of pseudo-passive constructions
- Semantics & Pragmatics
- “All women are like that”: an overview of linguistic deindividualization and dehumanization of women in the incelosphere
- Counterfactual language, emotion, and perspective: a sentence completion study during the COVID-19 pandemic
- Constructing elderly patients’ agency through conversational storytelling
- Language Documentation & Typology
- Conative animal calls in Macha Oromo: function and form
- The syntax of African American English borrowings in the Louisiana Creole tense-mood-aspect system
- Syntactic pausing? Re-examining the associations
- Bibliographic bias and information-density sampling
- Historical & Comparative Linguistics
- Revisiting the hypothesis of ideophones as windows to language evolution
- Verifying the morpho-semantics of aspect via typological homogeneity
- Psycholinguistics & Neurolinguistics
- Sign recognition: the effect of parameters and features in sign mispronunciations
- Influence of translation on perceived metaphor features: quality, aptness, metaphoricity, and familiarity
- Effects of grammatical gender on gender inferences: Evidence from French hybrid nouns
- Processing reflexives in adjunct control: an exploration of attraction effects
- Language Acquisition & Language Learning
- How do L1 glosses affect EFL learners’ reading comprehension performance? An eye-tracking study
- Modeling L2 motivation change and its predictive effects on learning behaviors in the extramural digital context: a quantitative investigation in China
- Ongoing exposure to an ambient language continues to build implicit knowledge across the lifespan
- On the relationship between complexity of primary occupation and L2 varietal behavior in adult migrants in Austria
- The acquisition of speaking fundamental frequency (F0) features in Cantonese and English by simultaneous bilingual children
- Sociolinguistics & Anthropological Linguistics
- A computational approach to detecting the envelope of variation
- Attitudes toward code-switching among bilingual Jordanians: a comparative study
- “Let’s ride this out together”: unpacking multilingual top-down and bottom-up pandemic communication evidenced in Singapore’s coronavirus-related linguistic and semiotic landscape
- Across time, space, and genres: measuring probabilistic grammar distances between varieties of Mandarin
- Navigating linguistic ideologies and market dynamics within China’s English language teaching landscape
- Streetscapes and memories of real socialist anti-fascism in south-eastern Europe: between dystopianism and utopianism
- What can NLP do for linguistics? Towards using grammatical error analysis to document non-standard English features
- From sociolinguistic perception to strategic action in the study of social meaning
- Minority genders in quantitative survey research: a data-driven approach to clear, inclusive, and accurate gender questions
- Variation is the way to perfection: imperfect rhyming in Chinese hip hop
- Shifts in digital media usage before and after the pandemic by Rusyns in Ukraine
- Computational & Corpus Linguistics
- Revisiting the automatic prediction of lexical errors in Mandarin
- Finding continuers in Swedish Sign Language
- Conversational priming in repetitional responses as a mechanism in language change: evidence from agent-based modelling
- Construction grammar and procedural semantics for human-interpretable grounded language processing
- Through the compression glass: language complexity and the linguistic structure of compressed strings
- Could this be next for corpus linguistics? Methods of semi-automatic data annotation with contextualized word embeddings
- The Red Hen Audio Tagger
- Code-switching in computer-mediated communication by Gen Z Japanese Americans
- Supervised prediction of production patterns using machine learning algorithms
- Introducing Bed Word: a new automated speech recognition tool for sociolinguistic interview transcription
- Decoding French equivalents of the English present perfect: evidence from parallel corpora of parliamentary documents
- Enhancing automated essay scoring with GCNs and multi-level features for robust multidimensional assessments
- Sociolinguistic auto-coding has fairness problems too: measuring and mitigating bias
- The role of syntax in hashtag popularity
- Language practices of Chinese doctoral students studying abroad on social media: a translanguaging perspective
- Cognitive Linguistics
- Metaphor and gender: are words associated with source domains perceived in a gendered way?
- Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese