Emotion recognition ability across different modalities: The role of language status (L1/LX), proficiency and cultural background

Pernelle Lorette; Jean-Marc Dewaele

doi:10.1515/applirev-2017-0015

Article

Emotion recognition ability across different modalities: The role of language status (L1/LX), proficiency and cultural background

Pernelle Lorette and Jean-Marc Dewaele

Published/Copyright: October 24, 2018

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information

From the journal Applied Linguistics Review Volume 11 Issue 1

Abstract

This paper considers individual differences in the Emotion Recognition Ability (ERA) of 1368 participants in different modalities. The sample consisted of 557 first language (L1) and 881 foreign language (LX) users of English from all over the world. This study investigates four independent variables, namely modality of communication, language status (L1 versus LX), proficiency, and cultural background. The dependent variable is a score reflecting ERA. Participants were asked to identify an emotion (happiness, sadness, anger, fear, surprise and disgust) portrayed by a native English-speaking actress in six short recordings – either audiovisual or audio-only – embedded in an online questionnaire. English proficiency was measured through a lexical recognition test. Statistical analyses revealed that participants were better able to recognise emotions when visual cues are available. Overall, there was no difference between L1 and LX users’ ERA. However, L1 users outperformed LX users when visual cues were not available, which suggest that LX users are able to reach L1-like ERA when they can rely on a sufficient amount of cues. Participants with higher proficiency scores had significantly higher ERA scores, particularly in the audio-only condition. Asian LX users were found to score significantly lower than other LX users.

Keywords: emotion recognition; English L1 versus LX; modality; proficiency; cultural background

Appendix

The stimuli can be consulted on the following links. They have been transcribed below, in order of apparition in the survey.

1) Disgust: http://www.youtube.com/embed/rH6evcth9Vs (visual-vocal-verbal) and https://www.youtube.com/watch?v=UUU0vreMgj8 (vocal-verbal)

So, Jerry, you wanted to discuss the proposal that I put together for the two separate groups. You, you, you’ve got something … Kind of … No, no, it’s not … It’s sort of there. No, it’s still there. It’s now dripping down a little bit. Maybe if you use a napkin somewhere that you could wipe it with.

2) Anger: http://www.youtube.com/embed/8VcoNbk3HVE (visual-vocal-verbal) and https://www.youtube.com/watch?v=5XDGUAUh54A (vocal-verbal)

Yesterday, I went to see my mother-in-law. It was actually her birthday the day before yesterday, but I couldn’t go because I had a business meeting. And I bought her a very nice bunch of flowers. Very nice. And when I got there, she said:”What is this about?”. And I said: “Well, it is your birthday, Maria. Happy birthday!” And she said: “It’s not my birthday, it was my birthday yesterday.” So anyway, I really hope she liked the flowers.

3) Happiness: http://www.youtube.com/embed/x1S3IzTmf6A (visual-vocal-verbal) and https://www.youtube.com/watch?v=7SlWJk5N-iQ (vocal-verbal)

So, I went to my Pilates class after a really long time of absence of a few weeks, which you start to really notice if you haven’t been. But the teacher is absolutely amazing. What she’s really into is torturing us, basically. And she, she wants you to work really really hard. And she says: “Oh, when I’m coming in a … You know, if I am in a bad mood, if I see you there and I can hear you groaning a little bit, and gasping and running out of breath, then I think “Brilliant, I’m really getting them to do some good work”.

4) Fear: http://www.youtube.com/embed/T_5uBEYC8Wc (visual-vocal-verbal) and https://www.youtube.com/watch?v=WAWeL1UQhac (vocal-verbal)

So, I’ve got quite bad back pain, and it’s been like that for about three weeks. It’s really on my right side. And I suppose what I want to know is what … what it … you know, because I’ve tried doing some stretching but they haven’t … haven’t really worked at all. And I just kind of wondered whether you could tell me if you could exclude some things that it could be. It’s just that I know that one of the indications is some kind of … I know this sounds stupid but … some indications of … And I know I’m probably fine but … some indications of.. of … of … certain kinds of cancers can be … to do with back pain. And that’s kind of when … I don’t know if you can kind of just eliminate it. That would be really helpful.

5) Surprise: http://www.youtube.com/embed/rHuCJ6rojzE (visual-vocal-verbal) and https://www.youtube.com/watch?v=C8cO-UYimcY (vocal-verbal)

So this is like a really beautiful restaurant. It’s just really really nice, and … I just, you know, kind of … Oh my god! Really? Yes, Okay!

6) Sadness: http://www.youtube.com/embed/B-k3ivqrVDw (visual-vocal-verbal) and https://www.youtube.com/watch?v=rwDKHSamKp8 (vocal-verbal)

So, yesterday, I went to see my mother in law. It was actually her birthday the day before that and I actually couldn’t go. I was, you know, away working. So, I went the following day. And I bought her some flowers, and gave her the bouquet. And she was asking me why I bought her some flowers. And I said: “Well, because it is your birthday, Maria.” And she said: “No, it isn’t, it was my birthday the day before.” So, yeah, well anyway, I really hope she liked the flowers.

References

Altarriba, Jeanette. 2003. Does cariño equal “liking”? A theoretical approach to conceptual nonequivalence between languages. International Journal of Bilingualism 7. 305–322.10.1177/13670069030070030501Search in Google Scholar

Baenziger, Tanja, Didier Grandjean & Klaus R. Scherer. 2009. Emotion recognition from expressions in face, voice, and body. The multimodal emotion recognition test (MERT). Emotion 9(5). 691–704.10.1037/a0017088Search in Google Scholar

Barrett, Lisa F. 2006. Solving the emotion paradox: Categorization and the experience of emotion. Personality and Social Psychology Review 10(1). 20–46.10.1207/s15327957pspr1001_2Search in Google Scholar

Briggs, Jean L. 1970. Never in anger: Portrait of an Eskimo family. Cambridge, MA: Harvard University Press.Search in Google Scholar

Burns, Kenton L & Ernst G. Beier. 1973. Significance of vocal and visual channels in the decoding of emotional meaning. The Journal of Communication 23. 118–130.10.1111/j.1460-2466.1973.tb00936.xSearch in Google Scholar

Collignon, Olivier, Simon Girard, Frederic Gosselin, Sylvain Roy, Dave Saint-Amour, Maryse Lassonde & Franco Lepore. 2008. Audio-visual integration of emotion expression. Brain Research 1242. 126–135.10.1016/j.brainres.2008.04.023Search in Google Scholar

De Gelder, Beatrice & Jean Vroomen. 2000. The perception of emotions by ear and by eye. Cognition and Emotion 14(3). 289–311.10.1080/026999300378824Search in Google Scholar

Dewaele, Jean-Marc. 2010. Emotions in Multiple Languages. Basingstoke: Palgrave-MacMillan.10.1057/9780230289505Search in Google Scholar

Dewaele, Jean-Marc. 2018. Why the dichotomy ‘L1 Versus LX User’ is better than ‘Native Versus Non-native Speaker’. Applied Linguistics 39(2). 236–240.10.1093/applin/amw055Search in Google Scholar

Dromey, Christopher, Jose Silveira & Paul Sandor. 2005. Recognition of affective prosody by speakers of English as a first or foreign language. Speech Communication 47. 351–359.10.1016/j.specom.2004.09.010Search in Google Scholar

Ekman, Paul. 1972. Universals and cultural differences in facial expression of emotion. In James (ed.), Nebraska symposium on motivation, vol. 19. 207–283. Lincoln: University of Nebraska Press.Search in Google Scholar

Ekman, Paul. 1992. An argument for basic emotions. Cognition and Emotion 6. 169–200.10.1080/02699939208411068Search in Google Scholar

Ekman, Paul. 1999. Basic emotions. In Tim Dalgleish & Michaall Power (eds.), Handbook of cognition and emotion, 45–60. Sussex, UK: John Wiley.10.1002/0470013494.ch3Search in Google Scholar

Ekman, Paul. 2003. Sixteen enjoyable emotions. Emotion Researcher 18. 6–7.Search in Google Scholar

Ekman, Paul & Daniel Cordaro. 2011. What is meant by calling emotions basic. Emotion Review 3(4). 354–370.10.1177/1754073911410740Search in Google Scholar

Ekman, Paul & Wallace V. Friesen. 1969. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1. 49–98.10.1515/9783110880021.57Search in Google Scholar

Elfenbein, Hillary A & Nalini Ambady. 2002. On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin 128(2). 203–235.10.1037/0033-2909.128.2.203Search in Google Scholar

Gendron, Maria, Debi Roberson, Jacoba M. van der Vyver & Lisa F. Barrett. 2014. Perceptions of emotion from facial expressions are not culturally universal: Evidence from a remote culture. Emotion 14(2). 251–262.10.1037/a0036052Search in Google Scholar

Gladkova, Anna. 2014. Ethnosyntax. In Farzad Sharifian (ed.), The routledge handbook of language and culture, 52–88. London, New York: Routledge.Search in Google Scholar

Graham, Ray C., Arien W. Hamblin & Stanley Feldstein. 2001. Recognition of emotion in English voices by speakers of Japanese, Spanish and English. International Review of Applied Linguistics in Language Teaching 39. 19–37.10.1515/iral.39.1.19Search in Google Scholar

Irvine, Judith T. 1982. Language and affect: Some cross-cultural issues. In Heidi Byrnes (ed.), Contemporary perceptions of language: Interdisciplinary dimensions, 31–47. Washington, D.C: GU Press.Search in Google Scholar

Keltner, Dacher & Michelle N. Shiota. 2003. New displays and new emotions: A commentary on Rozin and Cohen (2003). Emotion 3. 86–91.10.1037/1528-3542.3.1.86Search in Google Scholar

Kramsch, Claire. 2015. Language and culture in foreign language learning. In Farzad Sharifian (ed.), The routledge handbook of language and culture, 634–644. Oxford: Routledge.Search in Google Scholar

Kreifelts, Benjamin, Thomas Ethofer, Wolfgang Grodd, Michael Erb & Dirk Wildgruber. 2007. Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage 37. 1445–1456.10.1016/j.neuroimage.2007.06.020Search in Google Scholar

Lakoff, George. 2016. Language and emotion. Emotion Review 8(3). 269–273.10.1177/1754073915595097Search in Google Scholar

Lemhöfer, Kristin & Mirjam Broersma. 2012. Introducing LexTALE: A quick and valid lexical test for advanced learners of English. Behavior Research Methods 44(2). 325–343.10.3758/s13428-011-0146-0Search in Google Scholar

Lorette, Pernelle & Jean-Marc Dewaele. 2015. Emotion recognition ability in English among L1 and LX users of English. International Journal of Language and Culture 2(1). 62–86.10.1075/ijolc.2.1.03lorSearch in Google Scholar

Matsumoto, David. (2009). Facial expressions are not universal. https://digest.bps.org.uk/2009/08/17/facial-emotional-expressions-are-not-universal/ (accessed 09 November 2015)Search in Google Scholar

Mehrabian, Albert & Morton Wiener. 1967. Decoding of inconsistent communications. Journal of Personality and Social Psychology 6. 108–114.10.1037/h0024532Search in Google Scholar

Mesquita, Batja & Nico H. Frijda. 1992. Cultural variations in emotions: A review. Psychological Bulletin 112(2). 179–204.10.1037/0033-2909.112.2.179Search in Google Scholar

Mulligan, Kevin & Klaus R. Scherer. 2012. Toward a working definition of emotion. Emotion Review 4(4). 345–357.10.1177/1754073912445818Search in Google Scholar

Oatley, Keith & Jennifer M. Jenkins. 1996. Understanding emotions. Oxford, UK: Blackwell Publishing.Search in Google Scholar

Paulmann, Silke & Marc D. Pell. 2011. Is there an advantage for recognizing multi-modal emotional stimuli?. Motivation and Emotion 35. 192–201.10.1007/s11031-011-9206-0Search in Google Scholar

Pavlenko, Aneta. 2008. Emotion and emotion-laden words in the bilingual lexicon. Bilingualism: Language and Cognition 11(2). 147–164.10.1017/S1366728908003283Search in Google Scholar

Pell, Marc D., Laura Monetta, Silke Paulmann & Sonja A. Kotz. 2009. Recognizing emotions in a foreign language. Journal of Nonverbal Behavior 33. 107–120.10.1007/s10919-008-0065-7Search in Google Scholar

Richins, Marsha L. 1997. Measuring Emotions in the Consumption Experience. Journal of Consumer Research 24(2). 127–146.10.1086/209499Search in Google Scholar

Rintell, Ellen. 1984. But how did you FEEL about that? The learner’s perception of emotion in speech. Applied Linguistics 5(3). 255–264.10.1093/applin/5.3.255Search in Google Scholar

Riviello, Maria Teresa, Anna Esposito, Mohamed Chetouani & David Cohen. 2011. Inferring emotional information from vocal and visual cues: A cross-cultural comparison. 2nd IEEE International Conference on Cognitive Infocommunications (CogInfoCom). 1–4.Search in Google Scholar

Robinson, Crystal J & Jeanette Altarrriba. 2014. Culture and language processing. In Farzad Sharifian (ed.), The routledge handbook of language and culture, 355–381. London, New York: Routledge.Search in Google Scholar

Russell, James A. 1991. Culture and the categorization of emotion. Psychological Bulletin 110. 426–450.10.1037/0033-2909.110.3.426Search in Google Scholar

Russell, James A. 1995. Facial expressions of emotion: What lies beyond minimal universality? Psychological Bulletin 118. 379–391.10.1037/0033-2909.118.3.379Search in Google Scholar

Russell, James A. 2003. Core affect and the psychological construction of emotion. Psychological Review 110. 145–172.10.1037/0033-295X.110.1.145Search in Google Scholar

Sauter, Disa A., Frank Eisner, Paul Ekman & Sophie K. Scott. 2010. Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proceedings of the National Academy of Sciences of the United States of America 107(6). 2408–2412.10.1073/pnas.0908239106Search in Google Scholar

Scherer, Klaus R. 1997. The role of culture in emotion-antecedent appraisal. Journal of Personality and Social Psychology 73(5). 902–922.10.1037/0022-3514.73.5.902Search in Google Scholar

Scherer, Klaus R., Reiner Banse & Harad G. Wallbott. 2001. Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-Cultural Psychology 32(1). 76–92.10.1177/0022022101032001009Search in Google Scholar

Tanaka, Akihiro, Ai Koizumi, Hisato Imai, Saori Hiramatsu, Eriko Hiramoto & Beatrice de Gelder. 2010. I feel your voice. Cultural differences in the multisensory perception of emotion. Psychological Sciences 21. 1259–1262.10.1177/0956797610380698Search in Google Scholar

Thompson, William F & Laura-Lee Balkwill. 2006. Decoding speech prosody in five languages. Semiotica 158. 407–424.10.1515/SEM.2006.017Search in Google Scholar

Tombs, Alastair G., Rebekah Russell-Bennett & Neal M. Ashkanasy. 2014. Recognising emotional expressions of complaining customers: A cross-cultural study. European Journal of Marketing 48(7/8). 1354–1374.10.1108/EJM-02-2011-0090Search in Google Scholar

Vaillant-Molina, Mariana, Lorraine E. Bahrick & Ross Flom. 2013. Young infants match facial and vocal emotional expressions of other infants. Infancy 18. E97–E111.10.1111/infa.12017Search in Google Scholar

Wang, Qi & Michael Ross. 2005. What we remember and what we tell: the effects of culture and self-priming on memory representations and narratives. Memory 13(6). 594–606.10.1080/09658210444000223Search in Google Scholar

Zhu, Yinyin. 2013. Expression and recognition of emotion in native and foreign speech: The case of Mandarin and Dutch. PhD dissertation, Leiden University.Search in Google Scholar

Published Online: 2018-10-24

Published in Print: 2020-03-26

You are currently not able to access this content.

Articles in the same Issue

https://doi.org/10.1515/applirev-2017-0015

Keywords for this article

emotion recognition; English L1 versus LX; modality; proficiency; cultural background