Abstract
I analyze multimodal viewpoint construction in comics to engage with how modalities function within the medium as a specific discourse context with distinct conventions and material qualities. I show how comics employ established storytelling practices with character, narrator, and narrative viewpoint levels, while building up and interweaving these through strategic uses of the modalities of the medium. I mobilize the cognitive theories of embodiment, domains, mental simulation, and mental space blending as an analytical framework. I examine the asynchronicity of viewpoint elements between modalities and their synthesis into composite character viewpoints in several examples. I show how modalities can be prioritized and their different qualities and functions strategically manipulated for viewpoint construal. These brief examples show the complexity inherent in multimodal communication and interpretation and the usefulness of encouraging the medium-specific and interdisciplinary analyses of cultural works from a cognitive linguistic perspective.
Acknowledgements
My gratitude to Barbara Dancygier and Lieven Vandelanotte for organizing the special session at ICLC 2016 for which this material was initially developed. My sincere thanks to Elaine M. Will, Top Shelf Productions, and Marvel Comics for their permission to include the figures in this article, and to the editors and reviewers of this journal for their valuable feedback. The Social Science and Humanities Research Council of Canada funded this research. This paper would not have been possible without the support of my loved ones, all of whom contribute immensely to improving my limited viewpoint.
References
Baetens, Jan. 2007. Revealing traces: A new theory of graphic enunciation. In Robin Varnum & Christina T. Gibbons (eds.), The language of comics: Word and image, 145–155. Jackson: University Press of Mississippi.Search in Google Scholar
Barsalou, Lawrence W. 1999. Perceptual symbol systems. Behavioral and Brain Sciences 22(4). 577–609.10.1017/S0140525X99002149Search in Google Scholar
Bateman, John A. 2008. Multimodality and genre: A foundation for the systematic analysis of multimodal documents. New York: Palgrave Macmillan.10.1057/9780230582323Search in Google Scholar
Bergen, Benjamin. 2003. To awaken a sleeping giant: Cognition and culture in September 11 political cartoons. In Michel Achard & Suzanne Kemmer (eds.), Language, culture, and mind, 1–12. Stanford: CSLI Publications.Search in Google Scholar
Bergen, Benjamin. 2012. Louder than words: The new science of how the mind makes meaning. New York: Basic Books.Search in Google Scholar
Borkent, Mike. 2010. Illusions of simplicity: A cognitive approach to visual poetry. English Text Construction 3(2). 145–164.10.1075/bct.40.02borSearch in Google Scholar
Borkent, Mike. 2014. Visual improvisation: Cognition, materiality, and postlinguistic visual poetry. Visible Language 48(3). 4–27.Search in Google Scholar
Chow, Kenny K. N. & D. Fox Harrell. 2013. Elastic anchors for conceptual blends: A framework for analyzing animated computer interfaces. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 427–444. Stanford: CSLI Publications.Search in Google Scholar
Chute, Hillary. 2008. Comics as literature? Reading graphic narrative. Publications of the Modern Languages Association of America (PMLA) 123(2). 452–465.10.1632/pmla.2008.123.2.452Search in Google Scholar
Cienki, Alan. 2007. Frames, idealized cognitive models, and domains. In Dirk Geeraerts & Hubert Cuyckens (eds.), The Oxford handbook of cognitive linguistics, 170–187. New York: Oxford University Press.Search in Google Scholar
Clausner, Timothy C. & William Croft. 1999. Domains and image schemas. Cognitive Linguistics 10(1). 1–31.10.1515/cogl.1999.001Search in Google Scholar
Cohn, Neil. 2012. Comics, linguistics, and visual language: The past and future of a field. In Frank Bramlett (ed.), Linguistics and the study of comics, 92–118. New York: Palgrave Macmillan.10.1057/9781137004109_5Search in Google Scholar
Cohn, Neil. 2013. The visual language of comics: Introduction to the structure and cognition of sequential images. New York: Bloomsbury.Search in Google Scholar
Cohn, Neil. 2014. The architecture of visual narrative comprehension: The interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology 5. 1–9.10.3389/fpsyg.2014.00680Search in Google Scholar
Cohn, Neil & Stephen Maher. 2015. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Research 1601. 73–84.10.1016/j.brainres.2015.01.018Search in Google Scholar
Cook, Amy. 2010. Shakespearean neuroplay: Reinvigorating the study of dramatic texts and performance through cognitive science. New York: Palgrave Macmillan.10.1057/9780230113053Search in Google Scholar
Dancygier, Barbara. 2005. Blending and narrative viewpoint: Jonathan Raban’s travels through mental spaces. Language and Literature 14(2). 99–127.10.1177/0963947005051281Search in Google Scholar
Dancygier, Barbara. 2012. The language of stories: A cognitive approach. New York: Cambridge University Press.10.1017/CBO9780511794414Search in Google Scholar
Dancygier, Barbara & Eve Sweetser (eds.). 2012. Viewpoint in language: A multimodal perspective. New York: Cambridge University Press.10.1017/CBO9781139084727Search in Google Scholar
Eisner, Will. 2008. Expressive anatomy for comics and narrative. New York: W.W. Norton.Search in Google Scholar
El Refaie, Elisabeth. 2009. Multiliteracies: How readers interpret political cartoons. Visual Communication 8(2). 181–205.10.1177/1470357209102113Search in Google Scholar
El Refaie, Elisabeth. 2013. Cross-modal resonances in creative multimodal metaphors: Breaking out of conceptual prisons. Review of Cognitive Linguistics 11(2). 236–249.10.1075/bct.78.02elrSearch in Google Scholar
Fauconnier, Gilles & Mark Turner. 2002. The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.Search in Google Scholar
Fillmore, Charles. 1982. Frame semantics. In The Linguistics Society of Korea (eds.), Linguistics in the morning calm, 111–137. Seoul: Hanshin.Search in Google Scholar
Fillmore, Charles. 2003. Double-decker definitions: The role of frames in meaning explanations. Sign Language Studies 3(3). 263–295.10.1353/sls.2003.0008Search in Google Scholar
Forceville, Charles. 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37(1). 69–88.10.1016/j.pragma.2003.10.002Search in Google Scholar
Forceville, Charles. 2008. Metaphor in pictures and multimodal representations. In Raymond W. Gibbs, Jr. (ed.), The Cambridge handbook of metaphor and thought, 462–482. New York: Cambridge University Press.10.1017/CBO9780511816802.028Search in Google Scholar
Forceville, Charles. 2013. Creative visual duality in comics balloons. In Tony Veale, Kurt Feyaerts & Charles Forceville (eds.), Creativity and the agile mind, 253–273. Berlin: Walter de Gruyter.10.1515/9783110295290.253Search in Google Scholar
Forceville, Charles & Eduardo Urios-Aparisi. (eds.). 2009. Multimodal metaphor. Berlin: Mouton de Gruyter.10.1515/9783110215366Search in Google Scholar
Foulsham, Tom, Dean Wybrow & Neil Cohn. 2016. Reading without words: Eye movements in the comprehension of comic strips. Applied Cognitive Psychology 30. 566–579.10.1002/acp.3229Search in Google Scholar
Fukada, Chie. 2016. The dynamic interplay between words and pictures in picture storybooks: How visual and verbal information interact and affect the readers’ viewpoint and understanding. In Barbara Dancygier, Wei-Iun Lu & Arie Verhagen (eds.), Viewpoint and the fabric of meaning: Form and use of viewpoint tools across languages and modalities, 217–236. Berlin: Walter de Gruyter.10.1515/9783110365467-010Search in Google Scholar
Gibbons, Alison. 2012. Multimodal literature and experimentation. In Joe Bray, Alison Gibbons & Brian McHale (eds.), The Routledge handbook of experimental literature, 420–434. New York: Routledge.Search in Google Scholar
Gibbs, Raymond W., Jr. 2006. Embodiment and cognitive science. New York, NY: Cambridge University Press.10.1017/CBO9780511805844Search in Google Scholar
Gibbs, Raymond W., Jr. & Teenie Matlock. 2008. Metaphor, imagination, and simulation: Psycholinguistic evidence. In Raymond W. Gibbs, Jr. (ed.), The Cambridge handbook of metaphor and thought, 161–176. New York: Cambridge University Press.10.1017/CBO9780511816802.011Search in Google Scholar
Grady, Joseph. 2005. Primary metaphors as inputs to conceptual integration. Journal of Pragmatics 37(10). 1595–1614.10.1016/j.pragma.2004.03.012Search in Google Scholar
Groensteen, Thierry. 2007. The system of comics. (Translated by Bart Beaty & Nick Nguyen). Jackson: University Press of Mississippi.Search in Google Scholar
Hart, Christopher. 2016. The visual basis of linguistic meaning and its implications for critical discourse studies: Integrating cognitive linguistic and multimodal methods. Discourse & Society 27(3). 335–350.10.1177/0957926516630896Search in Google Scholar
Hatfield, Charles. 2005. Alternative comics: An emerging literature. Jackson: University Press of Mississippi.Search in Google Scholar
Hayles, N. Katherine. 2004. Print is flat, code is deep: The importance of media-specific analysis. Poetics Today 25(1). 67–90.10.1215/03335372-25-1-67Search in Google Scholar
Hickok, Gregory. 2014. The myth of mirror neurons: The real neuroscience of communication and cognition. New York: W.W. Norton.Search in Google Scholar
Hiraga, Masako. 2005. Metaphor and iconicity: A cognitive approach to analysing texts. New York: Palgrave Macmillan.10.1057/9780230510708Search in Google Scholar
Holsanova, Jana, Nils Holmberg & Kenneth Holmqvist. 2009. Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology 23. 1215–1226.10.1002/acp.1525Search in Google Scholar
Horstkotte, Silke & Nancy Pedri. 2011. Focalization in graphic narrative. Narrative 19(3). 330–357.10.1353/nar.2011.0021Search in Google Scholar
Hutchins, Edwin. 2005. Material anchors for conceptual blends. Journal of Pragmatics 37(10). 1555–1577.10.1016/j.pragma.2004.06.008Search in Google Scholar
Kendall, L.N., Quentin Raffaelli, Alan Kingstone & Rebecca M. Todd. 2016. Iconic faces are not real faces: Enhanced emotion detection and altered neural processing as faces become more iconic. Cognitive Research: Principles and Implications 1(1). 19. doi:10.1186/s41235-016-0021-8 (accessed 10 January 2017).Search in Google Scholar
Kress, Gunther & Theo van Leeuwen. 1996. Reading images: The grammar of visual design. London & New York: Routledge.Search in Google Scholar
Kukkonen, Karin. 2013. Space, time and causality in graphic narratives: An embodied approach. In Daniel Stein & Jan-Noël Thon (eds.), From comic strips to graphic novels: Contributions on the theory and history of graphic narrative, 49–66. Berlin: De Gruyter.10.1515/9783110282023.49Search in Google Scholar
Lakoff, George. 1996. Sorry, I’m not myself today: The metaphor system for conceptualizing the self. In Gilles Fauconnier & Eve Sweetser (eds.), Spaces, worlds, and grammar, 91–123. Chicago: University of Chicago Press.Search in Google Scholar
Lakoff, George & Mark Johnson. 1980. Conceptual metaphor in everyday language. The Journal of Philosophy 77(8). 453–486.10.4324/9781315161921-21Search in Google Scholar
Langacker, Ronald W. 2001. Discourse in cognitive grammar. Cognitive Linguisics 12(2). 143–188.10.1515/cogl.12.2.143Search in Google Scholar
Lee, Stan & Jack Kirby. 1965. Captain America: Among Us, Wreckers Dwell. Tales of Suspense 64. New York: Marvel Comics.Search in Google Scholar
Lemire, Jeff. 2012. The underwater welder. Atlanta: Top Shelf Productions.Search in Google Scholar
Mar, Raymond A. & Keith Oatley. 2008. The function of fiction is the abstraction and simulation of social experience. Perspectives on Psychological Science 3(3). 173–192.10.1111/j.1745-6924.2008.00073.xSearch in Google Scholar
McCloud, Scott. 1994. Understanding comics: The invisible art. New York: Harper Perennial.Search in Google Scholar
Meskin, Aaron. 2009. Comics as literature?. British Journal of Aesthetics 49(3). 219–239.10.1093/aesthj/ayp025Search in Google Scholar
Miodrag, Hannah. 2013. Comics and language: Reimagining critical discourse on the form. Jackson: University Press of Mississippi.10.14325/mississippi/9781617038044.001.0001Search in Google Scholar
Mittelberg, Irene. 2013. Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.). Language and the creative mind, 325–346. Stanford: CSLI Publications.Search in Google Scholar
Murray, Chris. 2000. Popaganada: Superhero comics and propaganda in World War Two. In Anne Magnussen & Hans-Christian Christiansen (eds.), Comics & Culture: Analytical and Theoretical Approaches to Comics, 141–156. Copenhagen: University of Copenhagen.Search in Google Scholar
Narayan, Shweta. 2012. Maybe what he means is he actually got the spot: Physical and cognitive viewpoint in a gesture study. In Barbara Dancygier & Eve Sweetser (eds.), Viewpoint in language: A multimodal perspective. New York: Cambridge University Press.10.1017/CBO9781139084727.009Search in Google Scholar
Newall, Michael. 2011. What is a picture? Depiction, realism, abstraction. New York: Palgrave Macmillan.10.1057/9780230297531Search in Google Scholar
Potsch, Elisabeth & Robert F. Williams. 2012. Image schemas and conceptual metaphor in action comics. In Frank Bramlett (ed.), Linguistics and the Study of Comics, 13–36. New York: Palgrave Macmillan.10.1057/9781137004109_2Search in Google Scholar
Richie, L. David. 2008. X is a journey: Embodied simulation in metaphor interpretation. Metaphor and Symbol 23. 174–199.10.1080/10926480802223085Search in Google Scholar
Steen, Francis & Mark Turner. 2013. Multimodal construction grammar. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the Creative Mind, 255–274. Stanford: CSLI Publications.Search in Google Scholar
Stockwell, Peter. 2009. Texture: A cognitive aesthetics of reading. Edinburgh: Edinburgh University Press.10.1515/9780748631209Search in Google Scholar
Sweetser, Eve. 2013. Creativity across modalities in viewpoint construction. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the Creative Mind, 239–254. Stanford: CSLI Publications.Search in Google Scholar
Sweetser, Eve & Gilles Fauconnier. 1996. Cognitive links and domains: Basic aspects of mental space theory. In Eve Sweetser & Gilles Fauconnier (eds.), Spaces, worlds and grammars, 1–28. Chicago: University of Chicago Press.Search in Google Scholar
Talmy, Leonard. 1996. Fictive motion in language and ‘ception.’ In Paul Bloom, Mary A. Peterson, Lynn Nadel & Merrill F. Garrett (eds.), Language and space, 211–276. Cambridge, MA: MIT Press.Search in Google Scholar
Will, Elaine M. 2013. Look straight ahead. Saskatoon: Cuckoo’s Nest Press.Search in Google Scholar
Zwaan, Rolf A. 2009. Mental simulation in language comprehension and social cognition. European Journal of Social Psychology 39(7). 1142–1150.10.1002/ejsp.661Search in Google Scholar
© 2017 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Research Articles
- Viewpoint phenomena in multimodal communication
- Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy
- How politicians express different viewpoints in gesture and speech simultaneously
- Eye gaze and viewpoint in multimodal interaction management
- Alignment and empathy as viewpoint phenomena: The case of amplifiers and comical hypotheticals
- Composite utterances in a signed language: Topic constructions and perspective-taking in ASL
- Mediated characters: Multimodal viewpoint construction in comics
- Internet memes as multimodal constructions
Articles in the same Issue
- Frontmatter
- Research Articles
- Viewpoint phenomena in multimodal communication
- Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy
- How politicians express different viewpoints in gesture and speech simultaneously
- Eye gaze and viewpoint in multimodal interaction management
- Alignment and empathy as viewpoint phenomena: The case of amplifiers and comical hypotheticals
- Composite utterances in a signed language: Topic constructions and perspective-taking in ASL
- Mediated characters: Multimodal viewpoint construction in comics
- Internet memes as multimodal constructions