Chapter 7. Gaze and face-to-face interaction
-
Gérard Bailly
Abstract
This chapter describes experimental and modeling work aiming at describing gaze patterns that are mutually exchanged by interlocutors during situated and task-directed face-to-face two-ways interactions. We will show that these gaze patterns (incl. blinking rate) are significantly influenced by the cognitive states of the interlocutors (speaking, listening, thinking, etc.), their respective roles in the conversation (e.g. instruction giver, respondent) as well as their social relationship (e.g. colleague, supervisor).
This chapter provides insights into the (micro-)coordination of gaze with other components of attention management as well as methodologies for capturing and modeling behavioral regularities observed in experimental data. A particular emphasis is put on statistical models, which are able to learn behaviors in a data-driven way.
We will introduce several statistical models of multimodal behaviors that can be trained on such multimodal signals and generate behaviors given perceptual cues. We will notably compare performances and properties of models which explicitly model the temporal structure of studied signals, and which relate them to internal cognitive states. In particular we study Semi-Hidden Markov Models and Dynamic Bayesian Networks and compare them to classifiers without sequential models (Support Vector Machines and Decision Trees).
We will further show that the gaze of conversational agents (virtual talking heads, speaking robots) may have a strong impact on communication efficiency. One of the conclusions we draw from these experiments is that multimodal behavioral models able to generate co-verbal gaze patterns of interactive avatars should be designed with great care in order not to increase the cognitive load of human partners. Experiments involving an impoverished or irrelevant control of the gaze of artificial agents (virtual talking heads and humanoid robots) have demonstrated its negative impact on communication (Garau, Slater, Bee, & Sasse, 2001).
Abstract
This chapter describes experimental and modeling work aiming at describing gaze patterns that are mutually exchanged by interlocutors during situated and task-directed face-to-face two-ways interactions. We will show that these gaze patterns (incl. blinking rate) are significantly influenced by the cognitive states of the interlocutors (speaking, listening, thinking, etc.), their respective roles in the conversation (e.g. instruction giver, respondent) as well as their social relationship (e.g. colleague, supervisor).
This chapter provides insights into the (micro-)coordination of gaze with other components of attention management as well as methodologies for capturing and modeling behavioral regularities observed in experimental data. A particular emphasis is put on statistical models, which are able to learn behaviors in a data-driven way.
We will introduce several statistical models of multimodal behaviors that can be trained on such multimodal signals and generate behaviors given perceptual cues. We will notably compare performances and properties of models which explicitly model the temporal structure of studied signals, and which relate them to internal cognitive states. In particular we study Semi-Hidden Markov Models and Dynamic Bayesian Networks and compare them to classifiers without sequential models (Support Vector Machines and Decision Trees).
We will further show that the gaze of conversational agents (virtual talking heads, speaking robots) may have a strong impact on communication efficiency. One of the conclusions we draw from these experiments is that multimodal behavioral models able to generate co-verbal gaze patterns of interactive avatars should be designed with great care in order not to increase the cognitive load of human partners. Experiments involving an impoverished or irrelevant control of the gaze of artificial agents (virtual talking heads and humanoid robots) have demonstrated its negative impact on communication (Garau, Slater, Bee, & Sasse, 2001).
Chapters in this book
- Prelim pages i
- Table of contents v
- Chapter 1. Introduction 1
-
Part 1. Theoretical considerations
- Chapter 2. Eye gaze as a cue for recognizing intention and coordinating joint action 21
- Chapter 3. Effects of a speaker’s gaze on language comprehension and acquisition 47
- Chapter 4. Weaving oneself into others 67
- Chapter 5. On the role of gaze for successful and efficient communication 91
-
Part 2. Methodological considerations
- Chapter 6. Quantifying the interplay of gaze and gesture in deixis using an experimental-simulative approach 109
- Chapter 7. Gaze and face-to-face interaction 139
- Chapter 8. Automatic analysis of in-the-wild mobile eye-tracking experiments using object, face and person detection 169
-
Part 3. Case studies
- Chapter 9. Gaze, addressee selection and turn-taking in three-party interaction 197
- Chapter 10. Gaze as a predictor for lexical and gestural alignment 233
- Chapter 11. Mobile dual eye-tracking in face-to-face interaction 265
- Chapter 12. Displaying recipiency in an interpreter-mediated dialogue 301
- Index 323
Chapters in this book
- Prelim pages i
- Table of contents v
- Chapter 1. Introduction 1
-
Part 1. Theoretical considerations
- Chapter 2. Eye gaze as a cue for recognizing intention and coordinating joint action 21
- Chapter 3. Effects of a speaker’s gaze on language comprehension and acquisition 47
- Chapter 4. Weaving oneself into others 67
- Chapter 5. On the role of gaze for successful and efficient communication 91
-
Part 2. Methodological considerations
- Chapter 6. Quantifying the interplay of gaze and gesture in deixis using an experimental-simulative approach 109
- Chapter 7. Gaze and face-to-face interaction 139
- Chapter 8. Automatic analysis of in-the-wild mobile eye-tracking experiments using object, face and person detection 169
-
Part 3. Case studies
- Chapter 9. Gaze, addressee selection and turn-taking in three-party interaction 197
- Chapter 10. Gaze as a predictor for lexical and gestural alignment 233
- Chapter 11. Mobile dual eye-tracking in face-to-face interaction 265
- Chapter 12. Displaying recipiency in an interpreter-mediated dialogue 301
- Index 323
