Multimodal Analysis of Speech Attractiveness Expression
-
Sandra Madureira
Abstract
Charisma can be interpreted in terms of semantic-pragmatic features related to attractiveness, trustworthiness, and persuasion. In a multimodal framework, charismatic speech can be tackled through vocal and visual feature analysis. Our aim, in this chapter, is to develop an experiment to investigate the prosodic aspects of voice quality, vocal dynamics, and facial expressions in a corpus of audiovisual samples. The research subject is an influential communicator. The audiovisual samples were analyzed perceptually, acoustically, and visually. The experiment comprises the following aspects: the application of a perceptual semantic questionnaire to a group of 64 judges; acoustic measures extracted automatically through the Prosody Descriptor Extractor Script; a perceptual analysis of voice quality and prosodic settings using the Voice Profile Analysis; the analysis of facial Action Unities performed by means of an automated system which provides information on the movements of the face muscles and their intensities, valence, activation, and heart rate; multivariate statistical analysis, applying the PCA method to consider correlations among variables. In the evaluation of the speaker’s charismatic behavior, attractiveness was found to play a less important role than the ability to convince and be trusted. Both vocal and visual prosodies were relevant to characterize speech charisma.
Abstract
Charisma can be interpreted in terms of semantic-pragmatic features related to attractiveness, trustworthiness, and persuasion. In a multimodal framework, charismatic speech can be tackled through vocal and visual feature analysis. Our aim, in this chapter, is to develop an experiment to investigate the prosodic aspects of voice quality, vocal dynamics, and facial expressions in a corpus of audiovisual samples. The research subject is an influential communicator. The audiovisual samples were analyzed perceptually, acoustically, and visually. The experiment comprises the following aspects: the application of a perceptual semantic questionnaire to a group of 64 judges; acoustic measures extracted automatically through the Prosody Descriptor Extractor Script; a perceptual analysis of voice quality and prosodic settings using the Voice Profile Analysis; the analysis of facial Action Unities performed by means of an automated system which provides information on the movements of the face muscles and their intensities, valence, activation, and heart rate; multivariate statistical analysis, applying the PCA method to consider correlations among variables. In the evaluation of the speaker’s charismatic behavior, attractiveness was found to play a less important role than the ability to convince and be trusted. Both vocal and visual prosodies were relevant to characterize speech charisma.
Chapters in this book
- Frontmatter I
- Preface V
- Contents IX
- Prosody and L2 Learning Interface: The Case of Spanish L2 and Brazilian Portuguese L1 Intonation 1
- The Role of Prosody in the Processing of Ambiguities in Brazilian Portuguese 31
- Defining and Identifying Discourse Markers in Spontaneous Speech 65
- A Contribution to a Better Understanding of Silent Pause 103
- Perceptual and Physiological Correlates of Voice Quality Settings 127
- Multimodal Analysis of Speech Attractiveness Expression 151
- Posture and Gestures Can Affect the Prosodic Speaker Impact in a Remote Presentation 181
- An Acoustic Analysis of Creaky Voice Patterns in Singing 223
- Evaluating OpenAI’s Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person 247
- Index
Chapters in this book
- Frontmatter I
- Preface V
- Contents IX
- Prosody and L2 Learning Interface: The Case of Spanish L2 and Brazilian Portuguese L1 Intonation 1
- The Role of Prosody in the Processing of Ambiguities in Brazilian Portuguese 31
- Defining and Identifying Discourse Markers in Spontaneous Speech 65
- A Contribution to a Better Understanding of Silent Pause 103
- Perceptual and Physiological Correlates of Voice Quality Settings 127
- Multimodal Analysis of Speech Attractiveness Expression 151
- Posture and Gestures Can Affect the Prosodic Speaker Impact in a Remote Presentation 181
- An Acoustic Analysis of Creaky Voice Patterns in Singing 223
- Evaluating OpenAI’s Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person 247
- Index
