Coordinating action and language
-
Thomas A. Farmer
Abstract
The overarching aim of this chapter is to highlight the strong interfacing between linguistic and motor systems, focusing especially on the degree to which manipulations of a visual context interact with linguistic manipulations in order to modulate the properties of manual movements elicited in response to a linguistic signal. First, we provide a review of recent work in the domain of embodied cognition that highlights the engagement of motor systems during language comprehension, including gestures, eye movements, and postural sway. In the context of this body of work, we then focus on the notion that continuous and non-ballistic computer-mouse movement trajectories can serve as a viable motoric index of various aspects of language processing. The results of studies that have recorded mouse-movement trajectories in visual-world displays will be reviewed, with the goal of highlighting the methodological benefits of data contained in the recorded trajectories, along with their theoretical consequences. New data will be presented from a study in which both eye- and hand-movements were simultaneously recorded, speaking to the relationship between the two movement-related dependent measures. Finally, we will highlight the degree to which mouse-movements around a visual display can be informative for domains other than syntactic processing, focusing on new data from research designed to investigate the processing of verbal aspect.
Abstract
The overarching aim of this chapter is to highlight the strong interfacing between linguistic and motor systems, focusing especially on the degree to which manipulations of a visual context interact with linguistic manipulations in order to modulate the properties of manual movements elicited in response to a linguistic signal. First, we provide a review of recent work in the domain of embodied cognition that highlights the engagement of motor systems during language comprehension, including gestures, eye movements, and postural sway. In the context of this body of work, we then focus on the notion that continuous and non-ballistic computer-mouse movement trajectories can serve as a viable motoric index of various aspects of language processing. The results of studies that have recorded mouse-movement trajectories in visual-world displays will be reviewed, with the goal of highlighting the methodological benefits of data contained in the recorded trajectories, along with their theoretical consequences. New data will be presented from a study in which both eye- and hand-movements were simultaneously recorded, speaking to the relationship between the two movement-related dependent measures. Finally, we will highlight the degree to which mouse-movements around a visual display can be informative for domains other than syntactic processing, focusing on new data from research designed to investigate the processing of verbal aspect.
Chapters in this book
- Prelim pages i
- Table of contents v
- Preface vii
- Towards a situated view of language 1
- Perception of the visual environment 31
- Attention and eye movement metrics in visual world eye tracking 67
- The role of syntax in sentence and referential processing 83
- Reaching sentence and reference meaning 127
- Discourse level processing 151
- Figurative language processing 185
- The role of affordances in visually situated language comprehension 205
- Characterising visual context effects 227
- Visual world studies of conversational perspective taking 261
- Visual environment and interlocutors in situated dialogue 291
- Coordinating action and language 323
- Index 357
Chapters in this book
- Prelim pages i
- Table of contents v
- Preface vii
- Towards a situated view of language 1
- Perception of the visual environment 31
- Attention and eye movement metrics in visual world eye tracking 67
- The role of syntax in sentence and referential processing 83
- Reaching sentence and reference meaning 127
- Discourse level processing 151
- Figurative language processing 185
- The role of affordances in visually situated language comprehension 205
- Characterising visual context effects 227
- Visual world studies of conversational perspective taking 261
- Visual environment and interlocutors in situated dialogue 291
- Coordinating action and language 323
- Index 357
