Volume 2005, Issue 156

That human social interaction involves the intertwined cooperation of different modalities is uncontroversial. Researchers in several allied fields have, however, only recently begun to document the precise ways in which talk, gesture, gaze, and aspects of the material surround are brought together to form coherent courses of action. The papers in this volume are attempts to develop this line of inquiry. Although the authors draw on a range of analytic, theoretical, and methodological traditions (conversation analysis, ethnography, distributed cognition, and workplace studies), all are concerned to explore and illuminate the inherently multimodal character of social interaction. Recent studies, including those collected in this volume, suggest that different modalities work together not only to elaborate the semantic content of talk but also to constitute coherent courses of action. In this introduction we present evidence for this position. We begin by reviewing some select literature focusing primarily on communicative functions and interactive organizations of specific modalities before turning to consider the integration of distinct modalities in interaction.

This article explores how participants in Japanese conversation use language and the body as resources for jointly constructing turns at talk during ‘description activities’. I examine how language and participants’ bodily conduct mutually contextualize one another to build temporally-unfolding frameworks of co-participation, and explore ways in which participants utilize such frameworks as resources to accomplish joint turn construction. By examining intricate processes of joint turn construction achieved through the use of language and embodied actions, I argue that, while turns-at-talk are often treated as if they were a bounded slot for speaking given to one participant at a time, they may be more adequately conceptualized as a temporally unfolding, interactively sustained domain of multimodal conduct through which both the speaker and recipients build in concert with one another relevant actions that contribute to the further progression of the activity in progress.

In conversation, speakers recurrently refer to non-present persons whom they believe their recipients should be able to identify. Speakers frequently, and preferentially, employ names to accomplish such an outcome but may use recognitional descriptors such as ‘the man from town’, where a name is not available. Drawing on videotaped interaction from a small Caribbean community, this paper examines the pursuit of recognition through the use of such descriptors. It is suggested that, in such sequences, gesture, talk, and gaze combine in multiple ways so as to form coherent courses of multimodal action.

This research examines how representational gestures (Kita 2000) made by scientists during collaborative discussions in a biochemistry lab, are used as cognitive artifacts for formulating scientific theory. By analyzing digital video of lab meetings and interviews with scientists, we find that representational gestures serve to reference and animate portions of existing material structure such as models, diagrams, and graphs. Representational gestures appear to play a significant role in how the scientific group both conceptualize and communicate theories about molecules. We believe that representational gestures operate as instantiations of essential spatiodynamic features that are not efficiently conveyed in other modalities, like language and graphical representations, and as such, are vital resources for shaping theoretical understandings in collaborative, face-to-face scientific activity. Gestures may also serve to package theoretical conceptions into semiotic entities that can be used symbolically in the ‘community of practice’ (Lave 1991). We use a theoretical framework provided by distributed cognition and embodied cognition to examine communal representational gestures as cognitive artifacts produced and modified by the biochemistry lab community during instances where theories are being negotiated.

This article examines the use of imagination as a communicative resource in interaction. Using actual examples of imagination in action collected during anthropological fieldwork at an architecture firm in California, I develop the concept of ‘collaborative imagining’ as a social, jointly-produced activity in which the objects of thought are created and manipulated in the shared space of face-to-face interaction. Such an activity relies upon a number of semiotic resources, and the paper examines three in particular: talk, gestures, and material objects, taking into consideration the talk and gestures of all participants in an interaction as well as the material surround available to the interactants. This view suggests two new ways to think about imagination. First, imagining is not a purely individual act but is also a product of and resource for group interaction; second, the objects of imaginative thought are not necessarily mental images, but can potentially be constructed in the material world with real, material components.

Using data from videotaped recordings of photography workshops and critiques, this paper investigates the discursive practices used by photographers and apprentice photographers to transform events in their domain of professional scrutiny into the knowledge representative of their profession. Specifically, it investigates the linguistic and embodied practices photographers deploy and the mediated activities in which they engage in order to determine appropriate color balance in a photograph.

Cognitive Neuroscientists use functional magnetic resonance imaging (fMRI) technology to generate digital images of the human brain. An fMRI image, as a final product of the scientific work, does not document movements and sounds that were present when such an image was recorded. Yet, a focus on actual moments of scientific practice reveals that such forgotten elements of practice can play important roles in understanding and knowledge acquisition. The multimodal interaction among scientists and digital screens shows how movements of the experimental subject and the scanner noise are performed to make images meaningful. Moreover, it suggests that the phenomena whose detection is crucial for a scientific reading of the brain images, such as motion artifacts, become visible as a result of coordination of various semiotic modalities (i.e., images, talk, body movements, gesture, etc.).

Drawing from ethnographic, empirical, and historical / cultural perspectives, we examine the extent to which visual aspects of music contribute to the communication that takes place between performers and their listeners. First, we introduce a framework for understanding how media and genres shape aural and visual experiences of music. Second, we present case studies of two performances, and describe the relation between visual and aural aspects of performance. Third, we report empirical evidence that visual aspects of performance reliably influence perceptions of musical structure (pitch related features) and affective interpretations of music. Finally, we trace new and old media trajectories of aural and visual dimensions of music, and highlight how our conceptions, perceptions and appreciation of music are intertwined with technological innovation and media deployment strategies.