Home Linguistics & Semiotics The visual expression of temporal concepts in visual narratives
Article
Licensed
Unlicensed Requires Authentication

The visual expression of temporal concepts in visual narratives

  • Xiran Yang EMAIL logo
Published/Copyright: May 16, 2022

Abstract

This study examines the visual representation of diegetic time in visual narrative stories, and explores why the invisible diegetic time flow could be expressed through visible elements, and how patterns and textures could be used to specify states of happenings. The study demonstrates that metaphorical relations between time and space are visually realized in visual narratives, leading to the representation of diegetic time and its related concepts on different levels of two-dimensional space. The segmentation of page space and the visual components within panel space are employed to build a diegetic timeline as well as specify temporal aspects of actions and states in visual narrative stories. Neuroscientific evidence which suggests that a common metric is used to compute magnitudes of time and space in the human brain might help explain why time as an abstract concept could undergo visual materialization in visual narratives.


Corresponding author: Xiran Yang, School of Foreign Languages, Shanghai Jiao Tong University, Shanghai, P.R. China, E-mail:

Funding source: Youth Foundation for Humanities and Social Sciences Research, Ministry of Education of China

Award Identifier / Grant number: No. 19YJC740104

  1. Research funding: This work was funded by the Ministry of Education of China with Youth Foundation for Humanities and Social Sciences Research (No. 19YJC740104).

References

Bakhtin, Mikhail. 2002. Forms of time and of the chronotope in the novel: Notes toward a historical poetics. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 15–24. Columbus: The Ohio State University Press.Search in Google Scholar

Binetti, Nicola, Nobuhiro Hagura, Charles Fadipe, Alessandro Tomassini, Vincent Walsh & Sven Bestmann. 2015. Binding space and time through action. Proceedings of the Royal Society B: Biological Sciences 282. 20150381.10.1098/rspb.2015.0381Search in Google Scholar

Bordwell, David & Kristin Thompson. 2008. Film art: An introduction, 8th edn. New York: McGraw-Hill.Search in Google Scholar

Boroditsky, Lera. 2000. Metaphoric structuring: Understanding time through spatial metaphors. Cognition 75(1). 1–28. https://doi.org/10.1016/s0010-0277(99)00073-6.Search in Google Scholar

Boroditsky, Lera. 2001. Does language shape thought?: Mandarin and English speakers’ conceptions of time. Cognitive Psychology 43(1). 1–22. https://doi.org/10.1006/cogp.2001.0748.Search in Google Scholar

Boroditsky, Lera. 2018. Language and the construction of time through space. Trends in Neurosciences 41(10). 651–653. https://doi.org/10.1016/j.tins.2018.08.004.Search in Google Scholar

Buelens, Geert, Harald Hendrix & Monica Jansen (eds.). 2012. The history of futurism: The precursors, protagonists, and legacies. Lanham, MD: Lexington.Search in Google Scholar

Bueti, Domenica & Vincent Walsh. 2009. The parietal cortex and the representation of time, space, number, and other magnitudes. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 364(1525). 1831–1840. https://doi.org/10.1098/rstb.2009.0028.Search in Google Scholar

Casasanto, Daniel & Lera Boroditsky. 2008. Time in the mind: Using space to think about time. Cognition 106(2). 579–593. https://doi.org/10.1016/j.cognition.2007.03.004.Search in Google Scholar

Casasanto, Daniel & Roberto Bottini. 2014. Mirror reading can reverse the flow of time. Journal of Experimental Psychology: General 143(2). 473–479. https://doi.org/10.1037/a0033297.Search in Google Scholar

Casasanto, Daniel & Kyle Jasmin. 2012. The hands of time: Temporal gestures in English speakers. Cognitive Linguistics 23(4). 643–674. https://doi.org/10.1515/cog-2012-0020.Search in Google Scholar

Clark, Herbert. 1973. Space, time, semantics and the child. In Timothy E. Moore (ed.), Cognitive development and the acquisition of language, 27–63. New York: Academic Press.10.1016/B978-0-12-505850-6.50008-6Search in Google Scholar

Cohen, Jonathan, William Perlstein, Braver Todd, Leigh Nystrom, Noll Douglas, John Jonides & Edward Smith. 1997. Temporal dynamics of brain activation during a working memory task. Nature 386. 604–608. https://doi.org/10.1038/386604a0.Search in Google Scholar

Cohn, Neil. 2013. Beyond word balloons and thought bubbles: The integration of text and image. Semiotica 197(1/4). 35–63.10.1515/sem-2013-0079Search in Google Scholar

Cohn, Neil. 2020. Who understands comics? Questioning the universality of visual language comprehension. London: Bloomsbury.10.5040/9781350156074Search in Google Scholar

Dancygier, Barbara & Eve Sweetser. 2014. Figurative language. New York: Cambridge University Press.Search in Google Scholar

Dannenberg, Hilary. 2008. Coincidence and counterfactuality: Plotting time and space in narrative fiction. Lincoln, NE: University of Nebraska Press.10.2307/j.ctt1dgn486Search in Google Scholar

Davey, Graham (ed.). 2011. Applied psychology. Chichester: Blackwell.Search in Google Scholar

Day, Sean. 1996. Synaesthesia and synaesthetic metaphors. Psyche 2(32). 1–16.Search in Google Scholar

Dirven, René & Ralf Pörings (eds.). 2003. Metaphor and metonymy in comparison and contrast. Berlin & New York: Walter de Gruyter.10.1515/9783110219197Search in Google Scholar

Dormal, Valérie, Giulia Dormal, Frédéric Joassin & Mauro Pesenti. 2012. A common right fronto-parietal network for numerosity and duration processing: An fMRI study. Human Brain Mapping 33. 1490–1501. https://doi.org/10.1002/hbm.21300.Search in Google Scholar

Duncan, Randy & Matthew Smith. 2009. The power of comics: History, form, and culture. New York & London: Continuum.Search in Google Scholar

El Refaie, Elisabeth. 2012. Autobiographical comics: Life writing in pictures. Jackson: University Press of Mississippi.10.14325/mississippi/9781617036132.001.0001Search in Google Scholar

Fauconnier, Gilles & Mark Turner. 2008. Rethinking metaphor. In Raymond W. GibbsJr (ed.), The Cambridge handbook of metaphor and thought, 53–66. Cambridge: Cambridge University Press.10.1017/CBO9780511816802.005Search in Google Scholar

Feng, Dezheng & KayO’Halloran. 2012. Representing emotive meaning in visual images: A social semiotic approach. Journal of Pragmatics 44(14). 2067–2084. https://doi.org/10.1016/j.pragma.2012.10.003.Search in Google Scholar

Forceville, Charles. 2002. The identification of target and source in pictorial metaphors. Journal of Pragmatics 34(1). 1–14. https://doi.org/10.1016/s0378-2166(01)00007-8.Search in Google Scholar

Forceville, Charles. 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37(1). 69–88. https://doi.org/10.1016/j.pragma.2003.10.002.Search in Google Scholar

Forceville, Charles, Elisabeth El Refaie & Gert Meesters. 2014. Stylistics and comics. In Michael Burke (ed.), The Routledge handbook of stylistics, 485–499. London: Routledge.10.4324/9781315795331.ch30Search in Google Scholar

Friedman, Michael. 1983. Foundations of space-time theories: Relativistic physics and philosophy of science. Princeton: Princeton University Press.Search in Google Scholar

Friston, Karl, John Ashburner, Stefan Kiebel, Thomas Nichols & William Penny (eds.). 2007. Statistical parametric mapping: The analysis of functional brain images. London: Academic Press.Search in Google Scholar

Galton, Antony. 2011. Time flies but space does not: Limits to the spatialisation of time. Journal of Pragmatics 43. 695–703.10.1016/j.pragma.2010.07.002Search in Google Scholar

Genette, Gérard. 2002. Order, duration, and frequency. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 25–34. Columbus: The Ohio State University Press.Search in Google Scholar

Groensteen, Thierry. 2007. System of comics, Bart Beaty & Nick Nguyen (trans.) Jackson, MS: University Press of Mississippi.Search in Google Scholar

Jewitt, Carey (ed.). 2009. The Routledge handbook of multimodal analysis, 2nd edn. London: Routledge.Search in Google Scholar

Jewitt, Carey, Jeff Bezemer & O’Halloran Kay. 2016. Introducing multimodality. London & New York: Routledge.10.4324/9781315638027Search in Google Scholar

Kennedy, John. 1982. Metaphor in pictures. Perception 11(5). 589–605. https://doi.org/10.1068/p110589.Search in Google Scholar

Kishimoto, Masashi. 2013 [2000]. Naruto-1. Tokyo: Shueisha.Search in Google Scholar

Kress, Gunther & Theo Van Leeuwen. 2001. Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.Search in Google Scholar

Kress, Gunther & Theo Van Leeuwen. 2006. Reading images: The grammar of visual design, 2nd edn. London: Routledge.10.4324/9780203619728Search in Google Scholar

Lakoff, George & Mark Johnson. 1980. Metaphors we live by. Chicago & London: University of Chicago Press.Search in Google Scholar

Lakoff, George & Mark Johnson. 1999. Philosophy in the flesh. New York: Basic.Search in Google Scholar

Lorentz, Hendrik, Einstein Albert, Minkowski Hermann, Hermann Weyl & Sommerfeld Arnold. 1952. The principle of relativity: A collection of original memoirs on the special and general theory of relativity. New York: Dover.Search in Google Scholar

Magnani, Barbara, Massimiliano Oliveri, Giovanni Mancuso, Emanuela Galante & Francesca Frassinetti. 2011. Time and spatial attention: Effects of prism adaptation on temporal deficits in brain damaged patients. Neuropsychologia 49. 1016–1023. https://doi.org/10.1016/j.neuropsychologia.2010.12.014.Search in Google Scholar

McCloud, Scott. 1994. Understanding comics: The invisible art. New York: Harper Perennial.Search in Google Scholar

Milner, David & Mel Goodale. 2006. The visual brain in action, 2nd edn. Oxford: Oxford University Press.10.1093/acprof:oso/9780198524724.001.0001Search in Google Scholar

MODE. 2012. Glossary of multimodal terms. https://multimodalityglossary.wordpress.com/ (accessed 1 January 2022).Search in Google Scholar

Moore, Kevin. 2014. The spatial language of time: Metaphor, metonymy, and frames of reference. Amsterdam/Philadelphia: John Benjamins.10.1075/hcp.42Search in Google Scholar

Núñez, Rafael & Eve Sweetser. 2006. With the future behind them: Convergent evidence from Aymara language and gesture in the crosslinguistic comparison of spatial construals of time. Cognitive Science 30(3). 401–450.10.1207/s15516709cog0000_62Search in Google Scholar

Núñez, Rafael, Benjamin Motz & Ursina Teuscher. 2006. Time after time: The psychological reality of the ego-and time-reference-point distinction in metaphorical construals of time. Metaphor and Symbol 21(3). 133–146.10.1207/s15327868ms2103_1Search in Google Scholar

O’Halloran, Kay (ed.). 2004. Multimodal discourse analysis: Systemic functional perspectives. London & New York: Continuum.Search in Google Scholar

Rao, Stephen, Andrew Mayer & Deborah Harrington. 2001. The evolution of brain activation during temporal processing. Nature Neuroscience 4(3). 317–323. https://doi.org/10.1038/85191.Search in Google Scholar

Rapp, Brenda. 2001. The handbook of cognitive neuropsychology: What deficits reveal about the human mind. New York & London: Psychology Press.Search in Google Scholar

Ricoeur, Paul. 1988. Time and narrative, vol. 3, Kathleen Blamey & David Pessauer (trans.), Chicago & London: The University of Chicago Press.Search in Google Scholar

Ricoeur, Paul. 2002. Narrative time. In Brian Richardson (ed.), Narrative dynamics: Essays on time, plot, closure, and frames, 35–46. Columbus: The Ohio State University Press.10.1086/448093Search in Google Scholar

Riemer, Martin, Nadine Diersch, Florian Bublatzky & Thomas Wolbers. 2016. Space, time, and numbers in the right posterior parietal cortex: Differences between response code associations and congruency effects. NeuroImage 129. 72–79. https://doi.org/10.1016/j.neuroimage.2016.01.030.Search in Google Scholar

Roalf, David & Ruben Gur. 2017. Functional brain imaging in neuropsychology over the past 25 years. Neuropsychology 31(8). 954–971. https://doi.org/10.1037/neu0000426.Search in Google Scholar

Roland, Per. 1993. Brain activation. New York: Wiley-Liss.Search in Google Scholar

Saraceni, Mario. 2003. The language of comics. London & New York: Routledge.Search in Google Scholar

Shen, Yeshayahu & Ravid Aisenman. 2008. “Heard melodies are sweet, but those unheard are sweeter”: Synaesthetic metaphors and cognition. Language and Literature 17(2). 107–121. https://doi.org/10.1177/0963947007088222.Search in Google Scholar

Stevens, Stanley. 1957. On the psychophysical law. Psychological Review 64(3). 153–181. https://doi.org/10.1037/h0046162.Search in Google Scholar

Stevens, Stanley. 1971. Issues in psychophysical measurement. Psychological Review 78(5). 426–450. https://doi.org/10.1037/h0031324.Search in Google Scholar

Tay, Dennis. 2017. Exploring the metaphor–body–psychotherapy relationship. Metaphor and Symbol 32(3). 178–191. https://doi.org/10.1080/10926488.2017.1338021.Search in Google Scholar

Tendahl, Markus & Raymond GibbsJr. 2008. Complementary perspectives on metaphor: Cognitive linguistics and relevance theory. Journal of Pragmatics 40(11). 1823–1864. https://doi.org/10.1016/j.pragma.2008.02.001.Search in Google Scholar

Togashi, Yoshihiro. 2012 [1998]. Hunter X Hunter-1. Tokyo: Shueisha.Search in Google Scholar

Van Fraassen, Bas. 1970. An introduction to the philosophy of time and space. New York: Random House.Search in Google Scholar

Walsh, Vincent. 2003. A theory of magnitude: Common cortical metrics of time, space, and quantity. Trends in Cognitive Sciences 7(11). 483–488. https://doi.org/10.1016/j.tics.2003.09.002.Search in Google Scholar

Walsh, Vincent & Alvaro Pascual-Leone. 2003. Transcranial magnetic stimulation: A neurochronometrics of mind. Cambridge, MA: MIT Press.10.7551/mitpress/6896.001.0001Search in Google Scholar

Whitrow, Gerald. 1973. The nature of time. New York: Holt Rinehart & Winston.Search in Google Scholar

Yang, Xiran & Jonathan Webster. 2015. To be continued: Meaning-making in serialized manga as functional-multimodal narrative. Semiotica 207(1/4). 583–606. https://doi.org/10.1515/sem-2015-0066.Search in Google Scholar

Yu, Ning. 2012. The metaphorical orientation of time in Chinese. Journal of Pragmatics 44. 1335–1354. https://doi.org/10.1016/j.pragma.2012.06.002.Search in Google Scholar

Received: 2019-10-05
Accepted: 2022-04-19
Published Online: 2022-05-16
Published in Print: 2022-07-26

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 8.1.2026 from https://www.degruyterbrill.com/document/doi/10.1515/sem-2019-0107/html
Scroll to top button