Abstract
This study examines the visual representation of diegetic time in visual narrative stories, and explores why the invisible diegetic time flow could be expressed through visible elements, and how patterns and textures could be used to specify states of happenings. The study demonstrates that metaphorical relations between time and space are visually realized in visual narratives, leading to the representation of diegetic time and its related concepts on different levels of two-dimensional space. The segmentation of page space and the visual components within panel space are employed to build a diegetic timeline as well as specify temporal aspects of actions and states in visual narrative stories. Neuroscientific evidence which suggests that a common metric is used to compute magnitudes of time and space in the human brain might help explain why time as an abstract concept could undergo visual materialization in visual narratives.
Funding source: Youth Foundation for Humanities and Social Sciences Research, Ministry of Education of China
Award Identifier / Grant number: No. 19YJC740104
-
Research funding: This work was funded by the Ministry of Education of China with Youth Foundation for Humanities and Social Sciences Research (No. 19YJC740104).
References
Bakhtin, Mikhail. 2002. Forms of time and of the chronotope in the novel: Notes toward a historical poetics. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 15–24. Columbus: The Ohio State University Press.Search in Google Scholar
Binetti, Nicola, Nobuhiro Hagura, Charles Fadipe, Alessandro Tomassini, Vincent Walsh & Sven Bestmann. 2015. Binding space and time through action. Proceedings of the Royal Society B: Biological Sciences 282. 20150381.10.1098/rspb.2015.0381Search in Google Scholar
Bordwell, David & Kristin Thompson. 2008. Film art: An introduction, 8th edn. New York: McGraw-Hill.Search in Google Scholar
Boroditsky, Lera. 2000. Metaphoric structuring: Understanding time through spatial metaphors. Cognition 75(1). 1–28. https://doi.org/10.1016/s0010-0277(99)00073-6.Search in Google Scholar
Boroditsky, Lera. 2001. Does language shape thought?: Mandarin and English speakers’ conceptions of time. Cognitive Psychology 43(1). 1–22. https://doi.org/10.1006/cogp.2001.0748.Search in Google Scholar
Boroditsky, Lera. 2018. Language and the construction of time through space. Trends in Neurosciences 41(10). 651–653. https://doi.org/10.1016/j.tins.2018.08.004.Search in Google Scholar
Buelens, Geert, Harald Hendrix & Monica Jansen (eds.). 2012. The history of futurism: The precursors, protagonists, and legacies. Lanham, MD: Lexington.Search in Google Scholar
Bueti, Domenica & Vincent Walsh. 2009. The parietal cortex and the representation of time, space, number, and other magnitudes. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 364(1525). 1831–1840. https://doi.org/10.1098/rstb.2009.0028.Search in Google Scholar
Casasanto, Daniel & Lera Boroditsky. 2008. Time in the mind: Using space to think about time. Cognition 106(2). 579–593. https://doi.org/10.1016/j.cognition.2007.03.004.Search in Google Scholar
Casasanto, Daniel & Roberto Bottini. 2014. Mirror reading can reverse the flow of time. Journal of Experimental Psychology: General 143(2). 473–479. https://doi.org/10.1037/a0033297.Search in Google Scholar
Casasanto, Daniel & Kyle Jasmin. 2012. The hands of time: Temporal gestures in English speakers. Cognitive Linguistics 23(4). 643–674. https://doi.org/10.1515/cog-2012-0020.Search in Google Scholar
Clark, Herbert. 1973. Space, time, semantics and the child. In Timothy E. Moore (ed.), Cognitive development and the acquisition of language, 27–63. New York: Academic Press.10.1016/B978-0-12-505850-6.50008-6Search in Google Scholar
Cohen, Jonathan, William Perlstein, Braver Todd, Leigh Nystrom, Noll Douglas, John Jonides & Edward Smith. 1997. Temporal dynamics of brain activation during a working memory task. Nature 386. 604–608. https://doi.org/10.1038/386604a0.Search in Google Scholar
Cohn, Neil. 2013. Beyond word balloons and thought bubbles: The integration of text and image. Semiotica 197(1/4). 35–63.10.1515/sem-2013-0079Search in Google Scholar
Cohn, Neil. 2020. Who understands comics? Questioning the universality of visual language comprehension. London: Bloomsbury.10.5040/9781350156074Search in Google Scholar
Dancygier, Barbara & Eve Sweetser. 2014. Figurative language. New York: Cambridge University Press.Search in Google Scholar
Dannenberg, Hilary. 2008. Coincidence and counterfactuality: Plotting time and space in narrative fiction. Lincoln, NE: University of Nebraska Press.10.2307/j.ctt1dgn486Search in Google Scholar
Davey, Graham (ed.). 2011. Applied psychology. Chichester: Blackwell.Search in Google Scholar
Day, Sean. 1996. Synaesthesia and synaesthetic metaphors. Psyche 2(32). 1–16.Search in Google Scholar
Dirven, René & Ralf Pörings (eds.). 2003. Metaphor and metonymy in comparison and contrast. Berlin & New York: Walter de Gruyter.10.1515/9783110219197Search in Google Scholar
Dormal, Valérie, Giulia Dormal, Frédéric Joassin & Mauro Pesenti. 2012. A common right fronto-parietal network for numerosity and duration processing: An fMRI study. Human Brain Mapping 33. 1490–1501. https://doi.org/10.1002/hbm.21300.Search in Google Scholar
Duncan, Randy & Matthew Smith. 2009. The power of comics: History, form, and culture. New York & London: Continuum.Search in Google Scholar
El Refaie, Elisabeth. 2012. Autobiographical comics: Life writing in pictures. Jackson: University Press of Mississippi.10.14325/mississippi/9781617036132.001.0001Search in Google Scholar
Fauconnier, Gilles & Mark Turner. 2008. Rethinking metaphor. In Raymond W. GibbsJr (ed.), The Cambridge handbook of metaphor and thought, 53–66. Cambridge: Cambridge University Press.10.1017/CBO9780511816802.005Search in Google Scholar
Feng, Dezheng & KayO’Halloran. 2012. Representing emotive meaning in visual images: A social semiotic approach. Journal of Pragmatics 44(14). 2067–2084. https://doi.org/10.1016/j.pragma.2012.10.003.Search in Google Scholar
Forceville, Charles. 2002. The identification of target and source in pictorial metaphors. Journal of Pragmatics 34(1). 1–14. https://doi.org/10.1016/s0378-2166(01)00007-8.Search in Google Scholar
Forceville, Charles. 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37(1). 69–88. https://doi.org/10.1016/j.pragma.2003.10.002.Search in Google Scholar
Forceville, Charles, Elisabeth El Refaie & Gert Meesters. 2014. Stylistics and comics. In Michael Burke (ed.), The Routledge handbook of stylistics, 485–499. London: Routledge.10.4324/9781315795331.ch30Search in Google Scholar
Friedman, Michael. 1983. Foundations of space-time theories: Relativistic physics and philosophy of science. Princeton: Princeton University Press.Search in Google Scholar
Friston, Karl, John Ashburner, Stefan Kiebel, Thomas Nichols & William Penny (eds.). 2007. Statistical parametric mapping: The analysis of functional brain images. London: Academic Press.Search in Google Scholar
Galton, Antony. 2011. Time flies but space does not: Limits to the spatialisation of time. Journal of Pragmatics 43. 695–703.10.1016/j.pragma.2010.07.002Search in Google Scholar
Genette, Gérard. 2002. Order, duration, and frequency. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 25–34. Columbus: The Ohio State University Press.Search in Google Scholar
Groensteen, Thierry. 2007. System of comics, Bart Beaty & Nick Nguyen (trans.) Jackson, MS: University Press of Mississippi.Search in Google Scholar
Jewitt, Carey (ed.). 2009. The Routledge handbook of multimodal analysis, 2nd edn. London: Routledge.Search in Google Scholar
Jewitt, Carey, Jeff Bezemer & O’Halloran Kay. 2016. Introducing multimodality. London & New York: Routledge.10.4324/9781315638027Search in Google Scholar
Kennedy, John. 1982. Metaphor in pictures. Perception 11(5). 589–605. https://doi.org/10.1068/p110589.Search in Google Scholar
Kishimoto, Masashi. 2013 [2000]. Naruto-1. Tokyo: Shueisha.Search in Google Scholar
Kress, Gunther & Theo Van Leeuwen. 2001. Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.Search in Google Scholar
Kress, Gunther & Theo Van Leeuwen. 2006. Reading images: The grammar of visual design, 2nd edn. London: Routledge.10.4324/9780203619728Search in Google Scholar
Lakoff, George & Mark Johnson. 1980. Metaphors we live by. Chicago & London: University of Chicago Press.Search in Google Scholar
Lakoff, George & Mark Johnson. 1999. Philosophy in the flesh. New York: Basic.Search in Google Scholar
Lorentz, Hendrik, Einstein Albert, Minkowski Hermann, Hermann Weyl & Sommerfeld Arnold. 1952. The principle of relativity: A collection of original memoirs on the special and general theory of relativity. New York: Dover.Search in Google Scholar
Magnani, Barbara, Massimiliano Oliveri, Giovanni Mancuso, Emanuela Galante & Francesca Frassinetti. 2011. Time and spatial attention: Effects of prism adaptation on temporal deficits in brain damaged patients. Neuropsychologia 49. 1016–1023. https://doi.org/10.1016/j.neuropsychologia.2010.12.014.Search in Google Scholar
McCloud, Scott. 1994. Understanding comics: The invisible art. New York: Harper Perennial.Search in Google Scholar
Milner, David & Mel Goodale. 2006. The visual brain in action, 2nd edn. Oxford: Oxford University Press.10.1093/acprof:oso/9780198524724.001.0001Search in Google Scholar
MODE. 2012. Glossary of multimodal terms. https://multimodalityglossary.wordpress.com/ (accessed 1 January 2022).Search in Google Scholar
Moore, Kevin. 2014. The spatial language of time: Metaphor, metonymy, and frames of reference. Amsterdam/Philadelphia: John Benjamins.10.1075/hcp.42Search in Google Scholar
Núñez, Rafael & Eve Sweetser. 2006. With the future behind them: Convergent evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time. Cognitive Science 30(3). 401–450.10.1207/s15516709cog0000_62Search in Google Scholar
Núñez, Rafael, Benjamin Motz & Ursina Teuscher. 2006. Time after time: The psychological reality of the ego-and time-reference-point distinction in metaphorical construals of time. Metaphor and Symbol 21(3). 133–146.10.1207/s15327868ms2103_1Search in Google Scholar
O’Halloran, Kay (ed.). 2004. Multimodal discourse analysis: Systemic functional perspectives. London & New York: Continuum.Search in Google Scholar
Rao, Stephen, Andrew Mayer & Deborah Harrington. 2001. The evolution of brain activation during temporal processing. Nature Neuroscience 4(3). 317–323. https://doi.org/10.1038/85191.Search in Google Scholar
Rapp, Brenda. 2001. The handbook of cognitive neuropsychology: What deficits reveal about the human mind. New York & London: Psychology Press.Search in Google Scholar
Ricoeur, Paul. 1988. Time and narrative, vol. 3, Kathleen Blamey & David Pessauer (trans.), Chicago & London: The University of Chicago Press.Search in Google Scholar
Ricoeur, Paul. 2002. Narrative time. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 35–46. Columbus: The Ohio State University Press.10.1086/448093Search in Google Scholar
Riemer, Martin, Nadine Diersch, Florian Bublatzky & Thomas Wolbers. 2016. Space, time, and numbers in the right posterior parietal cortex: Differences between response code associations and congruency effects. NeuroImage 129. 72–79. https://doi.org/10.1016/j.neuroimage.2016.01.030.Search in Google Scholar
Roalf, David & Ruben Gur. 2017. Functional brain imaging in neuropsychology over the past 25 years. Neuropsychology 31(8). 954–971. https://doi.org/10.1037/neu0000426.Search in Google Scholar
Roland, Per. 1993. Brain activation. New York: Wiley-Liss.Search in Google Scholar
Saraceni, Mario. 2003. The language of comics. London & New York: Routledge.Search in Google Scholar
Shen, Yeshayahu & Ravid Aisenman. 2008. “Heard melodies are sweet, but those unheard are sweeter”: Synaesthetic metaphors and cognition. Language and Literature 17(2). 107–121. https://doi.org/10.1177/0963947007088222.Search in Google Scholar
Stevens, Stanley. 1957. On the psychophysical law. Psychological Review 64(3). 153–181. https://doi.org/10.1037/h0046162.Search in Google Scholar
Stevens, Stanley. 1971. Issues in psychophysical measurement. Psychological Review 78(5). 426–450. https://doi.org/10.1037/h0031324.Search in Google Scholar
Tay, Dennis. 2017. Exploring the metaphor–body–psychotherapy relationship. Metaphor and Symbol 32(3). 178–191. https://doi.org/10.1080/10926488.2017.1338021.Search in Google Scholar
Tendahl, Markus & Raymond GibbsJr. 2008. Complementary perspectives on metaphor: Cognitive linguistics and relevance theory. Journal of Pragmatics 40(11). 1823–1864. https://doi.org/10.1016/j.pragma.2008.02.001.Search in Google Scholar
Togashi, Yoshihiro. 2012 [1998]. Hunter X Hunter-1. Tokyo: Shueisha.Search in Google Scholar
Van Fraassen, Bas. 1970. An introduction to the philosophy of time and space. New York: Random House.Search in Google Scholar
Walsh, Vincent. 2003. A theory of magnitude: Common cortical metrics of time, space, and quantity. Trends in Cognitive Sciences 7(11). 483–488. https://doi.org/10.1016/j.tics.2003.09.002.Search in Google Scholar
Walsh, Vincent & Alvaro Pascual-Leone. 2003. Transcranial magnetic stimulation: A neurochronometrics of mind. Cambridge, MA: MIT Press.10.7551/mitpress/6896.001.0001Search in Google Scholar
Whitrow, Gerald. 1973. The nature of time. New York: Holt Rinehart & Winston.Search in Google Scholar
Yang, Xiran & Jonathan Webster. 2015. To be continued: Meaning-making in serialized manga as functional-multimodal narrative. Semiotica 207(1/4). 583–606. https://doi.org/10.1515/sem-2015-0066.Search in Google Scholar
Yu, Ning. 2012. The metaphorical orientation of time in Chinese. Journal of Pragmatics 44. 1335–1354. https://doi.org/10.1016/j.pragma.2012.06.002.Search in Google Scholar
© 2022 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Symbolicity, language, and mediality
- Logonomic signs as three-phase constraints of multimodal social semiosis
- Masked Covid life: a socio-semiotic investigation
- Language and radical anthropocentrism: the view from the supercategory
- Mechanisms of homonym transformations: on Catholic variants of Stalinist discourse in Poland
- Moral character, moral choice and the existential semiotics of space awareness
- A rhetoric of inauthenticity: critical object images in Woolf’s Victorian scenes
- The visual expression of temporal concepts in visual narratives
- Intersemiotic projection and academic comics: towards a social semiotic framework of multimodal paratactic and hypotactic projection
- Semiotics of humor in Nigerian politics
- Redundancies of traffic signs: an exploratory study
Articles in the same Issue
- Frontmatter
- Symbolicity, language, and mediality
- Logonomic signs as three-phase constraints of multimodal social semiosis
- Masked Covid life: a socio-semiotic investigation
- Language and radical anthropocentrism: the view from the supercategory
- Mechanisms of homonym transformations: on Catholic variants of Stalinist discourse in Poland
- Moral character, moral choice and the existential semiotics of space awareness
- A rhetoric of inauthenticity: critical object images in Woolf’s Victorian scenes
- The visual expression of temporal concepts in visual narratives
- Intersemiotic projection and academic comics: towards a social semiotic framework of multimodal paratactic and hypotactic projection
- Semiotics of humor in Nigerian politics
- Redundancies of traffic signs: an exploratory study