Home Mediated characters: Multimodal viewpoint construction in comics
Article
Licensed
Unlicensed Requires Authentication

Mediated characters: Multimodal viewpoint construction in comics

  • Mike Borkent EMAIL logo
Published/Copyright: August 3, 2017

Abstract

I analyze multimodal viewpoint construction in comics to engage with how modalities function within the medium as a specific discourse context with distinct conventions and material qualities. I show how comics employ established storytelling practices with character, narrator, and narrative viewpoint levels, while building up and interweaving these through strategic uses of the modalities of the medium. I mobilize the cognitive theories of embodiment, domains, mental simulation, and mental space blending as an analytical framework. I examine the asynchronicity of viewpoint elements between modalities and their synthesis into composite character viewpoints in several examples. I show how modalities can be prioritized and their different qualities and functions strategically manipulated for viewpoint construal. These brief examples show the complexity inherent in multimodal communication and interpretation and the usefulness of encouraging the medium-specific and interdisciplinary analyses of cultural works from a cognitive linguistic perspective.

Acknowledgements

My gratitude to Barbara Dancygier and Lieven Vandelanotte for organizing the special session at ICLC 2016 for which this material was initially developed. My sincere thanks to Elaine M. Will, Top Shelf Productions, and Marvel Comics for their permission to include the figures in this article, and to the editors and reviewers of this journal for their valuable feedback. The Social Science and Humanities Research Council of Canada funded this research. This paper would not have been possible without the support of my loved ones, all of whom contribute immensely to improving my limited viewpoint.

References

Baetens, Jan. 2007. Revealing traces: A new theory of graphic enunciation. In Robin Varnum & Christina T. Gibbons (eds.), The language of comics: Word and image, 145–155. Jackson: University Press of Mississippi.Search in Google Scholar

Barsalou, Lawrence W. 1999. Perceptual symbol systems. Behavioral and Brain Sciences 22(4). 577–609.10.1017/S0140525X99002149Search in Google Scholar

Bateman, John A. 2008. Multimodality and genre: A foundation for the systematic analysis of multimodal documents. New York: Palgrave Macmillan.10.1057/9780230582323Search in Google Scholar

Bergen, Benjamin. 2003. To awaken a sleeping giant: Cognition and culture in September 11 political cartoons. In Michel Achard & Suzanne Kemmer (eds.), Language, culture, and mind, 1–12. Stanford: CSLI Publications.Search in Google Scholar

Bergen, Benjamin. 2012. Louder than words: The new science of how the mind makes meaning. New York: Basic Books.Search in Google Scholar

Borkent, Mike. 2010. Illusions of simplicity: A cognitive approach to visual poetry. English Text Construction 3(2). 145–164.10.1075/bct.40.02borSearch in Google Scholar

Borkent, Mike. 2014. Visual improvisation: Cognition, materiality, and postlinguistic visual poetry. Visible Language 48(3). 4–27.Search in Google Scholar

Chow, Kenny K. N. & D. Fox Harrell. 2013. Elastic anchors for conceptual blends: A framework for analyzing animated computer interfaces. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 427–444. Stanford: CSLI Publications.Search in Google Scholar

Chute, Hillary. 2008. Comics as literature? Reading graphic narrative. Publications of the Modern Languages Association of America (PMLA) 123(2). 452–465.10.1632/pmla.2008.123.2.452Search in Google Scholar

Cienki, Alan. 2007. Frames, idealized cognitive models, and domains. In Dirk Geeraerts & Hubert Cuyckens (eds.), The Oxford handbook of cognitive linguistics, 170–187. New York: Oxford University Press.Search in Google Scholar

Clausner, Timothy C. & William Croft. 1999. Domains and image schemas. Cognitive Linguistics 10(1). 1–31.10.1515/cogl.1999.001Search in Google Scholar

Cohn, Neil. 2012. Comics, linguistics, and visual language: The past and future of a field. In Frank Bramlett (ed.), Linguistics and the study of comics, 92–118. New York: Palgrave Macmillan.10.1057/9781137004109_5Search in Google Scholar

Cohn, Neil. 2013. The visual language of comics: Introduction to the structure and cognition of sequential images. New York: Bloomsbury.Search in Google Scholar

Cohn, Neil. 2014. The architecture of visual narrative comprehension: The interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology 5. 1–9.10.3389/fpsyg.2014.00680Search in Google Scholar

Cohn, Neil & Stephen Maher. 2015. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Research 1601. 73–84.10.1016/j.brainres.2015.01.018Search in Google Scholar

Cook, Amy. 2010. Shakespearean neuroplay: Reinvigorating the study of dramatic texts and performance through cognitive science. New York: Palgrave Macmillan.10.1057/9780230113053Search in Google Scholar

Dancygier, Barbara. 2005. Blending and narrative viewpoint: Jonathan Raban’s travels through mental spaces. Language and Literature 14(2). 99–127.10.1177/0963947005051281Search in Google Scholar

Dancygier, Barbara. 2012. The language of stories: A cognitive approach. New York: Cambridge University Press.10.1017/CBO9780511794414Search in Google Scholar

Dancygier, Barbara & Eve Sweetser (eds.). 2012. Viewpoint in language: A multimodal perspective. New York: Cambridge University Press.10.1017/CBO9781139084727Search in Google Scholar

Eisner, Will. 2008. Expressive anatomy for comics and narrative. New York: W.W. Norton.Search in Google Scholar

El Refaie, Elisabeth. 2009. Multiliteracies: How readers interpret political cartoons. Visual Communication 8(2). 181–205.10.1177/1470357209102113Search in Google Scholar

El Refaie, Elisabeth. 2013. Cross-modal resonances in creative multimodal metaphors: Breaking out of conceptual prisons. Review of Cognitive Linguistics 11(2). 236–249.10.1075/bct.78.02elrSearch in Google Scholar

Fauconnier, Gilles & Mark Turner. 2002. The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.Search in Google Scholar

Fillmore, Charles. 1982. Frame semantics. In The Linguistics Society of Korea (eds.), Linguistics in the morning calm, 111–137. Seoul: Hanshin.Search in Google Scholar

Fillmore, Charles. 2003. Double-decker definitions: The role of frames in meaning explanations. Sign Language Studies 3(3). 263–295.10.1353/sls.2003.0008Search in Google Scholar

Forceville, Charles. 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37(1). 69–88.10.1016/j.pragma.2003.10.002Search in Google Scholar

Forceville, Charles. 2008. Metaphor in pictures and multimodal representations. In Raymond W. Gibbs, Jr. (ed.), The Cambridge handbook of metaphor and thought, 462–482. New York: Cambridge University Press.10.1017/CBO9780511816802.028Search in Google Scholar

Forceville, Charles. 2013. Creative visual duality in comics balloons. In Tony Veale, Kurt Feyaerts & Charles Forceville (eds.), Creativity and the agile mind, 253–273. Berlin: Walter de Gruyter.10.1515/9783110295290.253Search in Google Scholar

Forceville, Charles & Eduardo Urios-Aparisi. (eds.). 2009. Multimodal metaphor. Berlin: Mouton de Gruyter.10.1515/9783110215366Search in Google Scholar

Foulsham, Tom, Dean Wybrow & Neil Cohn. 2016. Reading without words: Eye movements in the comprehension of comic strips. Applied Cognitive Psychology 30. 566–579.10.1002/acp.3229Search in Google Scholar

Fukada, Chie. 2016. The dynamic interplay between words and pictures in picture storybooks: How visual and verbal information interact and affect the readers’ viewpoint and understanding. In Barbara Dancygier, Wei-Iun Lu & Arie Verhagen (eds.), Viewpoint and the fabric of meaning: Form and use of viewpoint tools across languages and modalities, 217–236. Berlin: Walter de Gruyter.10.1515/9783110365467-010Search in Google Scholar

Gibbons, Alison. 2012. Multimodal literature and experimentation. In Joe Bray, Alison Gibbons & Brian McHale (eds.), The Routledge handbook of experimental literature, 420–434. New York: Routledge.Search in Google Scholar

Gibbs, Raymond W., Jr. 2006. Embodiment and cognitive science. New York, NY: Cambridge University Press.10.1017/CBO9780511805844Search in Google Scholar

Gibbs, Raymond W., Jr. & Teenie Matlock. 2008. Metaphor, imagination, and simulation: Psycholinguistic evidence. In Raymond W. Gibbs, Jr. (ed.), The Cambridge handbook of metaphor and thought, 161–176. New York: Cambridge University Press.10.1017/CBO9780511816802.011Search in Google Scholar

Grady, Joseph. 2005. Primary metaphors as inputs to conceptual integration. Journal of Pragmatics 37(10). 1595–1614.10.1016/j.pragma.2004.03.012Search in Google Scholar

Groensteen, Thierry. 2007. The system of comics. (Translated by Bart Beaty & Nick Nguyen). Jackson: University Press of Mississippi.Search in Google Scholar

Hart, Christopher. 2016. The visual basis of linguistic meaning and its implications for critical discourse studies: Integrating cognitive linguistic and multimodal methods. Discourse & Society 27(3). 335–350.10.1177/0957926516630896Search in Google Scholar

Hatfield, Charles. 2005. Alternative comics: An emerging literature. Jackson: University Press of Mississippi.Search in Google Scholar

Hayles, N. Katherine. 2004. Print is flat, code is deep: The importance of media-specific analysis. Poetics Today 25(1). 67–90.10.1215/03335372-25-1-67Search in Google Scholar

Hickok, Gregory. 2014. The myth of mirror neurons: The real neuroscience of communication and cognition. New York: W.W. Norton.Search in Google Scholar

Hiraga, Masako. 2005. Metaphor and iconicity: A cognitive approach to analysing texts. New York: Palgrave Macmillan.10.1057/9780230510708Search in Google Scholar

Holsanova, Jana, Nils Holmberg & Kenneth Holmqvist. 2009. Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology 23. 1215–1226.10.1002/acp.1525Search in Google Scholar

Horstkotte, Silke & Nancy Pedri. 2011. Focalization in graphic narrative. Narrative 19(3). 330–357.10.1353/nar.2011.0021Search in Google Scholar

Hutchins, Edwin. 2005. Material anchors for conceptual blends. Journal of Pragmatics 37(10). 1555–1577.10.1016/j.pragma.2004.06.008Search in Google Scholar

Kendall, L.N., Quentin Raffaelli, Alan Kingstone & Rebecca M. Todd. 2016. Iconic faces are not real faces: Enhanced emotion detection and altered neural processing as faces become more iconic. Cognitive Research: Principles and Implications 1(1). 19. doi:10.1186/s41235-016-0021-8 (accessed 10 January 2017).Search in Google Scholar

Kress, Gunther & Theo van Leeuwen. 1996. Reading images: The grammar of visual design. London & New York: Routledge.Search in Google Scholar

Kukkonen, Karin. 2013. Space, time and causality in graphic narratives: An embodied approach. In Daniel Stein & Jan-Noël Thon (eds.), From comic strips to graphic novels: Contributions on the theory and history of graphic narrative, 49–66. Berlin: De Gruyter.10.1515/9783110282023.49Search in Google Scholar

Lakoff, George. 1996. Sorry, I’m not myself today: The metaphor system for conceptualizing the self. In Gilles Fauconnier & Eve Sweetser (eds.), Spaces, worlds, and grammar, 91–123. Chicago: University of Chicago Press.Search in Google Scholar

Lakoff, George & Mark Johnson. 1980. Conceptual metaphor in everyday language. The Journal of Philosophy 77(8). 453–486.10.4324/9781315161921-21Search in Google Scholar

Langacker, Ronald W. 2001. Discourse in cognitive grammar. Cognitive Linguisics 12(2). 143–188.10.1515/cogl.12.2.143Search in Google Scholar

Lee, Stan & Jack Kirby. 1965. Captain America: Among Us, Wreckers Dwell. Tales of Suspense 64. New York: Marvel Comics.Search in Google Scholar

Lemire, Jeff. 2012. The underwater welder. Atlanta: Top Shelf Productions.Search in Google Scholar

Mar, Raymond A. & Keith Oatley. 2008. The function of fiction is the abstraction and simulation of social experience. Perspectives on Psychological Science 3(3). 173–192.10.1111/j.1745-6924.2008.00073.xSearch in Google Scholar

McCloud, Scott. 1994. Understanding comics: The invisible art. New York: Harper Perennial.Search in Google Scholar

Meskin, Aaron. 2009. Comics as literature?. British Journal of Aesthetics 49(3). 219–239.10.1093/aesthj/ayp025Search in Google Scholar

Miodrag, Hannah. 2013. Comics and language: Reimagining critical discourse on the form. Jackson: University Press of Mississippi.10.14325/mississippi/9781617038044.001.0001Search in Google Scholar

Mittelberg, Irene. 2013. Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.). Language and the creative mind, 325–346. Stanford: CSLI Publications.Search in Google Scholar

Murray, Chris. 2000. Popaganada: Superhero comics and propaganda in World War Two. In Anne Magnussen & Hans-Christian Christiansen (eds.), Comics & Culture: Analytical and Theoretical Approaches to Comics, 141–156. Copenhagen: University of Copenhagen.Search in Google Scholar

Narayan, Shweta. 2012. Maybe what he means is he actually got the spot: Physical and cognitive viewpoint in a gesture study. In Barbara Dancygier & Eve Sweetser (eds.), Viewpoint in language: A multimodal perspective. New York: Cambridge University Press.10.1017/CBO9781139084727.009Search in Google Scholar

Newall, Michael. 2011. What is a picture? Depiction, realism, abstraction. New York: Palgrave Macmillan.10.1057/9780230297531Search in Google Scholar

Potsch, Elisabeth & Robert F. Williams. 2012. Image schemas and conceptual metaphor in action comics. In Frank Bramlett (ed.), Linguistics and the Study of Comics, 13–36. New York: Palgrave Macmillan.10.1057/9781137004109_2Search in Google Scholar

Richie, L. David. 2008. X is a journey: Embodied simulation in metaphor interpretation. Metaphor and Symbol 23. 174–199.10.1080/10926480802223085Search in Google Scholar

Steen, Francis & Mark Turner. 2013. Multimodal construction grammar. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the Creative Mind, 255–274. Stanford: CSLI Publications.Search in Google Scholar

Stockwell, Peter. 2009. Texture: A cognitive aesthetics of reading. Edinburgh: Edinburgh University Press.10.1515/9780748631209Search in Google Scholar

Sweetser, Eve. 2013. Creativity across modalities in viewpoint construction. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the Creative Mind, 239–254. Stanford: CSLI Publications.Search in Google Scholar

Sweetser, Eve & Gilles Fauconnier. 1996. Cognitive links and domains: Basic aspects of mental space theory. In Eve Sweetser & Gilles Fauconnier (eds.), Spaces, worlds and grammars, 1–28. Chicago: University of Chicago Press.Search in Google Scholar

Talmy, Leonard. 1996. Fictive motion in language and ‘ception.’ In Paul Bloom, Mary A. Peterson, Lynn Nadel & Merrill F. Garrett (eds.), Language and space, 211–276. Cambridge, MA: MIT Press.Search in Google Scholar

Will, Elaine M. 2013. Look straight ahead. Saskatoon: Cuckoo’s Nest Press.Search in Google Scholar

Zwaan, Rolf A. 2009. Mental simulation in language comprehension and social cognition. European Journal of Social Psychology 39(7). 1142–1150.10.1002/ejsp.661Search in Google Scholar

Received: 2016-8-25
Revised: 2017-1-28
Accepted: 2017-2-25
Published Online: 2017-8-3
Published in Print: 2017-8-28

© 2017 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 23.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/cog-2016-0098/html
Scroll to top button