Beyond speech balloons and thought bubbles: The integration of text and image
-
Neil Cohn
Neil Cohn (b. 1980) is a postdoctoral fellow at University of California at San Diego 〈neilcohn@emaki.net 〉. His research interests include narrative structure, comics, and visual language. His publications include “A different kind of cultural frame: An analysis of panels in American comics and Japanese manga” (2011); “Comics, linguistics, and visual language: The past and future of a field” (2012); “(Pea)nuts and bolts of visual narratives: Structure and meaning in sequential image comprehension” (with M. Paczynski, R. Jackendoff, P. Holcomb & G. Kuperberg, 2012); andThe visual language of comics: Structure, psychology, culture (2013).
Abstract
Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.
About the author
Neil Cohn (b. 1980) is a postdoctoral fellow at University of California at San Diego 〈neilcohn@emaki.net〉. His research interests include narrative structure, comics, and visual language. His publications include “A different kind of cultural frame: An analysis of panels in American comics and Japanese manga” (2011); “Comics, linguistics, and visual language: The past and future of a field” (2012); “(Pea)nuts and bolts of visual narratives: Structure and meaning in sequential image comprehension” (with M. Paczynski, R. Jackendoff, P. Holcomb & G. Kuperberg, 2012); and The visual language of comics: Structure, psychology, culture (2013).
©[2013] by Walter de Gruyter Berlin Boston
Articles in the same Issue
- Masthead
- Between the grid and composition: Layout in PowerPoint's design and use
- Beyond speech balloons and thought bubbles: The integration of text and image
- Abstraction as a limit to semiosis
- The multimodal representation of emotion in film: Integrating cognitive and semiotic approaches
- Hearing a shakkei: The semiotics of the audible in a Japanese stroll garden
- Towards a social semiotics of rhythm in popular music
- A carnival pilgrimage: Cultural semiotics in China
- Photography and intermediality: Analytical perspectives on notions referred to by the term “photography”
- An exploration of possible unconscious ethnic biases in higher education: The role of implicit attitudes on selection for university posts
- New insights into the medium hand: Discovering recurrent structures in gestures
- The multimodal construal of the experiential domain of recipes in Japanese and Chinese
- Cyberterrorist messages: A semiotic perspective
- A necessary condition for proof of abiotic semiosis
- Review of From First to Third Via Cybersemiotics
Articles in the same Issue
- Masthead
- Between the grid and composition: Layout in PowerPoint's design and use
- Beyond speech balloons and thought bubbles: The integration of text and image
- Abstraction as a limit to semiosis
- The multimodal representation of emotion in film: Integrating cognitive and semiotic approaches
- Hearing a shakkei: The semiotics of the audible in a Japanese stroll garden
- Towards a social semiotics of rhythm in popular music
- A carnival pilgrimage: Cultural semiotics in China
- Photography and intermediality: Analytical perspectives on notions referred to by the term “photography”
- An exploration of possible unconscious ethnic biases in higher education: The role of implicit attitudes on selection for university posts
- New insights into the medium hand: Discovering recurrent structures in gestures
- The multimodal construal of the experiential domain of recipes in Japanese and Chinese
- Cyberterrorist messages: A semiotic perspective
- A necessary condition for proof of abiotic semiosis
- Review of From First to Third Via Cybersemiotics
