Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy

Irene Mittelberg

doi:10.1515/cog-2016-0124

Article

Experiencing and construing spatial artifacts from within: Simulated artifact immersion as a multimodal viewpoint strategy

Irene Mittelberg

Published/Copyright: August 8, 2017

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Cognitive Linguistics Volume 28 Issue 3

Abstract

Viewpoint has been shown to be a powerful construal mechanism in multimodal spoken and signed discourse, as well as in various other modalities and genres. This paper investigates embodied viewpoint strategies that have been observed when speakers combine speech, gestures, postures, gaze, and simulated action to describe their interaction with spatial artifacts such as gallery buildings, virtual architectural models, and paintings. Simulated artifact immersion is introduced as a multimodal viewpoint strategy whereby speakers submerge into their mental representation of an artifact by perceiving and experiencing it from an internal vantage point. It is argued that this viewpoint strategy tends to be employed when there is no narrative structure for the speakers to fall back on. The paper’s aim is twofold: (a) to show that when speakers talk about their own experiences with spatial artifacts, distinguishing between immersed and non-immersed experiential viewpoint strategies may be more fitting than distinguishing between character and observer viewpoint; and (b) to discuss how considering the interaction of iconic, indexical, and metonymic principles in gesture may elucidate viewpoint phenomena in general.

Keywords: viewpoint; gesture; indexicality; metonymy; immersion

Funding statement: Excellence Initiative of the German Federal and State Governments.

Acknowledgements

I would like to thank Barbara Dancygier and Lieven Vandelanotte for their valuable input and for including my work in this special issue on viewpoint phenomena in multimodal communication. I also thank John Newman and two anonymous reviewers for their insightful comments and suggestions. Special thanks also go to Hannah Groninger and Thomas Schmitz for cooperating on the architectural design studies, to Bela Brenger and Matthias Priesters for creating the motion-capture plots, and to Mary Copple for helpful feedback on the manuscript.

References

Arnheim, R. 1969. Visual thinking. Berkeley, CA: University of California Press.Search in Google Scholar

Bergen, Benjamin K. 2005. Mental simulation in literal and figurative language understanding. In Seana Coulson & Barbara Lewandowska-Tomaszczyk (eds.), The literal and the nonliteral in language and thought, 255–278. Frankfurt a. M: Peter Lang.Search in Google Scholar

Bergen, Benjamin K. 2012. Louder than words: The new science of how the mind makes meaning. New York: Basic Books.Search in Google Scholar

Bouvet, Danielle. 1997. Le corps et la métaphore dans les langues gestuelles: A la recherche des modes de production des signes. Paris: L’Harmattan.Search in Google Scholar

Bredekamp, Horst. 2010. Theorie des Bildakts. Frankfurt a.m.: Suhrkamp.Search in Google Scholar

Brône, Geert, Bert Oben, Annelies Jehoul, Jelena Vranjes & Kurt Feyaerts. 2017. Eye gaze and viewpoint in multimodal interaction management. DOI:10.1515/COGL-2016-0119.Search in Google Scholar

Calbris, Geneviève. 1990. The semiotics of French gestures. Cambridge: Cambridge University Press.Search in Google Scholar

Cienki, Alan. 2013. Cognitive linguistics: Spoken language and gesture as expressions of conceptualization. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.1), 182–201. Berlin & Boston: De Gruyter Mouton.10.1515/9783110261318.182Search in Google Scholar

Cienki, Alan. 2015. Spoken language usage events. Language and Cognition 7(4). 499–514.10.1017/langcog.2015.20Search in Google Scholar

Cienki, Alan & Irene Mittelberg. 2013. Creativity in the forms and functions of gestures with speech. In Tony Veale, Kurt Feyaerts & Charles Forceville (eds.), Creativity and the agile mind: A multi-disciplinary study of a multi-faceted phenomenon, 231–252. Berlin & New York: Mouton de Gruyter.10.1515/9783110295290.231Search in Google Scholar

Clark, Herbert H. 2016. Depicting as a method of communication. Psychological Review 123(3). 324–347.10.1037/rev0000026Search in Google Scholar

Coulson, Seana. 2001. Semantic leaps. Cambridge: Cambridge University Press.10.1017/CBO9780511551352Search in Google Scholar

Croft, William & Alan Cruse. 2004. Cognitive linguistics. Cambridge: Cambridge University Press.10.1017/CBO9780511803864Search in Google Scholar

Dancygier, Barbara. 2012. The language of stories: A cognitive approach. Cambridge: Cambridge University Press.10.1017/CBO9780511794414Search in Google Scholar

Dancygier, Barbara & Eve Sweetser (eds.) 2012. Viewpoint in language: A multimodal perspective. Cambridge: Cambridge University Press.10.1017/CBO9781139084727Search in Google Scholar

Dancygier, Barbara & Eve Sweetser (eds.) 2014. Figurative language. Cambridge: Cambridge University Press.Search in Google Scholar

Dancygier, Barbara, Lu Wei-Lun & Arie Verhagen (eds.) 2016. Viewpoint and the fabric of meaning: Form and use of viewpoint tools across languages and modalities. Berlin: Mouton de Gruyter.10.1515/9783110365467Search in Google Scholar

Debras, Camille. 2013. Multimodal stancetaking in a videotaped corpus of discussions about the environment in British English. Paris: Sorbonne Nouvelle – Paris 3 University dissertation.Search in Google Scholar

Deliema, David & Eve Sweetser. 2016. Rethinking gestural viewpoint as multidimensional rather than a dichotomy. Conference talk at the International Society for Gesture Studies Conference (ISGS7), Paris, France, 18–22 July.Search in Google Scholar

Dudis, Paul. 2004. Body partitioning and real-space blends. Cognitive Linguistics 15(2). 223–238.10.1515/cogl.2004.009Search in Google Scholar

Duncan, Susan D., Justine Cassell & Elena T. Levy (eds.) 2007. Gesture and the dynamic dimension of language. Amsterdam & Philadelphia: John Benjamins.10.1075/gs.1Search in Google Scholar

Ehmer, Oliver. 2011. Imagination und Animation. Die Herstellung mentaler Räume durch animierte Rede. Berlin & New York: Mouton de Gruyter.10.1515/9783110237801Search in Google Scholar

Emmorey, Karen, Barbara Tversky & Holly A. Taylor. 2001. Using space to describe space: Perspective in speech, sign, and gesture. Spatial Cognition and Computation 2. 157–180.10.1023/A:1013118114571Search in Google Scholar

Fauconnier, Gilles & Mark Turner. 2002. The way we think. New York: Basic Books.Search in Google Scholar

Fillmore, Charles J. 1982. Frame semantics. In The Linguistic Society of Korea (eds.), Linguistics in the morning calm, 111–137. Seoul: Hanshin.Search in Google Scholar

Gibbs, Raymond W., Jr. 2006. Embodiment and cognitive science. Cambridge: Cambridge University Press.10.1017/CBO9780511805844Search in Google Scholar

Gibbs, Raymond W., Jr. & Teenie Matlock. 2008. Metaphor, imagination, and simulation: Psycholinguistic evidence. In Raymond W. Gibbs Jr. (ed.), The Cambridge handbook of metaphor and thought, 161–176. Cambridge: Cambridge University Press.10.1017/CBO9780511816802.011Search in Google Scholar

Hostetter, Autumn B. & Martha Alibali. 2008. Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review 15(3). 495–514.10.3758/PBR.15.3.495Search in Google Scholar

Hutchins, Edwin. 2005. Material anchors for conceptual blends. Journal of Pragmatics 37. 1555–1577.10.1016/j.pragma.2004.06.008Search in Google Scholar

Jakobson, Roman. 1971. Shifters, verbal categories, and the Russian verb. In Roman Jakobson (ed.), Selected wrtings, Vol. II: Words and language, 130–147. The Hague & Paris: Mouton.10.1515/9783110873269.130Search in Google Scholar

Jakobson, Roman & Krystyna Pomorska. 1983. Dialogues. Cambridge, MA: MIT Press.10.1515/9783112321218Search in Google Scholar

Janzen, Terry. 2012. Two ways of conceptualizing space: Motivating the use of static and rotated vantage point space in ASL. In Barbara Dancygier & Eve E. Sweetser (eds.), Viewpoint in language: A multimodal perspective, 156–176. Cambridge: Cambridge University Press.10.1017/CBO9781139084727.012Search in Google Scholar

Johnson, Mark. 1987. The body in the mind: The bodily basis of meaning, imagination and reasoning. Chicago: Chicago University Press.10.7208/chicago/9780226177847.001.0001Search in Google Scholar

Johnson, Mark. 2005. The philosophical significance of image schemas. In Beate Hampe (ed.), From perception to meaning: Image schemas in cognitive linguistics, 15–33. Berlin: Mouton de Gruyter.10.1515/9783110197532.1.15Search in Google Scholar

Johnson, Mark. 2007. The meaning of the body: Aesthetics of human understanding. Chicago: University of Chicago Press.10.7208/chicago/9780226026992.001.0001Search in Google Scholar

Krois, John M. 2011. Körperbilder und Bildschemata. Aufsätze zur Verkörperungstheorie ikonischer Formen (Actus et imago 2), hrsg. v. Horst Bredekamp & Marion Lauschke. Berlin: Akademie Verlag.Search in Google Scholar

Langacker, Ronald W. 1987. Foundations of cognitive grammar, Vol.1, Theoretical perspectives. Stanford: Stanford University Press.Search in Google Scholar

LeBaron, Curtis & Jürgen Streeck. 2000. Gestures, knowledge and the world. In David McNeill (ed.), Language and gesture, 118–138. Cambridge: Cambridge University Press.10.1017/CBO9780511620850.008Search in Google Scholar

Liddell, Scott. 2003. Grammar, gesture and meaning in American Sign Language. Cambridge: Cambridge University Press.10.1017/CBO9780511615054Search in Google Scholar

MacWhinney, Brian. 2005. The emergence of grammar from perspective. In Diane Pecher & Rolf Zwaan (eds.), Grounding cognition, 198–223. Cambridge: Cambridge University Press.10.1017/CBO9780511499968.009Search in Google Scholar

Marghetis, Tyler & Benjamin K. Bergen. 2014. Embodied meaning inside and out: The coupling of gestures and mental simulation. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.2), 2000–2007. Berlin & Boston: De Gruyter Mouton.10.1515/9783110302028.2000Search in Google Scholar

Matlock, Teenie. 2004. Fictive motion as cognitive simulation. Memory & Cognition 32. 1389–1400.10.3758/BF03206329Search in Google Scholar

Matlock, Teenie. 2006. Depicting fictive motion in drawings. In June Luchjenbroers (ed.), Cognitive linguistic investigations: Across languages, fields, and philosophical boundaries, 67–85. Amsterdam & Philadelphia: John Benjamins.10.1075/hcp.15.07matSearch in Google Scholar

Matlock, Teenie, David Sparks, Justin L. Matthews, Jeremy Hunter & Stephanie Huette. 2012. Smashing new results on aspectual framing: How people talk about car accidents. Studies in Language 36(3). 699–720.10.1075/bct.67.09matSearch in Google Scholar

McNeill, David. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Search in Google Scholar

McNeill, David (ed.) 2000. Language and gesture. Cambridge: Cambridge University Press.Search in Google Scholar

Mittelberg, Irene. 2006. Metaphor and metonymy in language and gesture: Discourse evidence for multimodal models of grammar. Ann Arbor, MI: UMI Cornell University dissertation.Search in Google Scholar

Mittelberg, Irene. 2013a. The exbodied mind: Cognitive-semiotic principles as motivating forces in gesture. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.1), 750–779. Berlin & Boston: de Gruyter Mouton.10.1515/9783110261318.755Search in Google Scholar

Mittelberg, Irene. 2013b. Balancing acts: Image schemas and force dynamics as experiential essence in pictures by Paul Klee and their gestural enactments. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 325–346. Stanford: CSLI Publications.Search in Google Scholar

Mittelberg, Irene. 2014. Gestures and iconicity. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.2), 1712–1732. Berlin & Boston: De Gruyter Mouton.Search in Google Scholar

Mittelberg, Irene. 2017. Embodied frames, metonymy and pragmatic inferencing in gesture. Gesture 16(2).10.1075/gest.16.2.03mitSearch in Google Scholar

Mittelberg, Irene, Thomas H. Schmitz & Hannah Groninger. 2017. Operative manufacts: Gestures as embodied sketches in early stages of the design process. In Sabine Ammon & Remei Capdevila Werning (eds.), The active image: Architecture and engineering in the age of modeling, 99–131. Berlin: Springer.10.1007/978-3-319-56466-1_5Search in Google Scholar

Mittelberg, Irene & Linda R. Waugh. 2009. Metonymy first, metaphor second: A cognitive-semiotic approach to multimodal figures of thought in co-speech gesture. In Charles Forceville and Eduardo Urios-Aparisi (eds.), Multimodal metaphor, 329–356. Berlin & New York: Mouton de Gruyter.10.1515/9783110215366.5.329Search in Google Scholar

Mittelberg, Irene & Linda R. Waugh. 2014. Gestures and metonymy. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.2), 1747–1766. Berlin & Boston: De Gruyter Mouton.10.1515/9783110302028.1747Search in Google Scholar

Müller, Cornelia. 1998. Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich. Berlin: Berlin Verlag Spitz.Search in Google Scholar

Müller, Cornelia. 2014. Gestural modes of representation as techniques of depiction. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.2), 1687–1702. Berlin & Boston: De Gruyter Mouton.Search in Google Scholar

Müller, Cornelia & Silva H. Ladewig. 2013. Metaphors for sensorimotor experiences: Gestures as embodied and dynamic conceptualizations of balance in dance lessons. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 295–324. Stanford: CSLI Publications.Search in Google Scholar

Murphy, Keith M. 2005. Collaborative imagining: The interactive use of gestures, talk, and graphic representation in architectural practice. Semiotica 156. 113–145.10.1515/semi.2005.2005.156.113Search in Google Scholar

Parrill, Fey. 2009. Dual viewpoint gestures. Gesture 9(3). 271–289.10.1075/gest.9.3.01parSearch in Google Scholar

Parrill, Fey. 2010. Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 25(5). 650–668.10.1080/01690960903424248Search in Google Scholar

Parrill, Fey. 2012. Interactions between discourse status and viewpoint in co-speech gesture. In Barbara Dancygier & Eve E. Sweetser (eds.), Viewpoint in language: A multimodal perspective, 97–112. Cambridge: Cambridge University Press.10.1017/CBO9781139084727.008Search in Google Scholar

Peirce, Charles S. 1960. In Charles Hartshorne & Weiss. Paul (eds.), Collected papers of Charles Sanders Peirce (1931–1958). Vol. I.: Principles of philosophy, Vol. II: Elements of logic. Cambridge: The Belknap Press of Harvard University Press.Search in Google Scholar

Perlman, Markus & Raymond W. Gibbs Jr. 2013. Sensorimotor simulation in speaking, gesturing, and understanding. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), Body – Language – Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.1), 512–533. Berlin & Boston: De Gruyter Mouton.10.1515/9783110261318.512Search in Google Scholar

Saussure, Ferdinand de. 1986. In par Charles Bally, Albert Sechehaye & T. De Mauro. (eds.), Cours de linguistique générale de Ferndinand de Saussure. Paris: Payot.Search in Google Scholar

Schoonjans, Steven. 2014. Modalpartikel als multimodale Konstruktionen. Eine korpusbasierte Kookkurrenzanalyse von Modalpartikeln und Gestik im Deutschen. Leuven: KU Leuven unpublished dissertation.Search in Google Scholar

Stec, Kashmiri. 2012. Meaningful shifts: A review of viewpoint markers in co-speech gesture and sign language. Gesture 12(3). 327–360.10.1075/gest.12.3.03steSearch in Google Scholar

Streeck, Jürgen. 2009. Gesturecraft: The manu-facture of meaning. Amsterdam & Philadelphia: John Benjamins.10.1075/gs.2Search in Google Scholar

Sweetser, Eve. 2007. Looking at space to study mental spaces: Co-speech gesture as a crucial data source in cognitive linguistics. In Monica Gonzalez-Marquez, Irene Mittelberg, Seana Coulson & Michael Spivey (eds.), Methods in cognitive linguistics, 201–224. Amsterdam & Philadelphia: John Benjamins.10.1075/hcp.18.15sweSearch in Google Scholar

Sweetser, Eve. 2012. Introduction: Viewpoint and perspective in language and gesture, from the ground down. In Barbara Dancygier & Eve E. Sweetser (eds.), Viewpoint in language: A multimodal perspective, 1–22. Cambridge: Cambridge University Press.10.1017/CBO9781139084727Search in Google Scholar

Sweetser, Eve. 2013. Creativity across modalities in viewpoint construction. In Mike Borkent, Barbara Dancygier & Jennifer Hinnell (eds.), Language and the creative mind, 239–254. Stanford, CA: CSLI Publications.Search in Google Scholar

Sweetser, Eve & Kashmiri Stec. 2016. Maintaining multiple viewpoints with gaze. In Barbara Dancygier, Lu Wei-Lun & Arie Verhagen (eds.), Viewpoint and the fabric of meaning: Form and use of viewpoint tools across languages and modalities, 237–257. Berlin: Mouton de Gruyter.10.1515/9783110365467-011Search in Google Scholar

Talmy, Leonard. 2000. Towards a cognitive semantics. Cambridge, MA: MIT Press.10.7551/mitpress/6847.001.0001Search in Google Scholar

Taub, Sarah. 2001. Language from the body. Iconicity and metaphor in American Sign Language. Cambridge: Cambridge University Press.10.1017/CBO9780511509629Search in Google Scholar

Turner, Mark (ed.) 2006. The artful mind. Oxford: Oxford University Press.10.1093/acprof:oso/9780195306361.001.0001Search in Google Scholar

Vandelanotte, Lieven. 2009. Speech and thought representation in English: A cognitive-functional approach. Berlin: Mouton de Gruyter.10.1515/9783110215373Search in Google Scholar

Verhagen, Arie. 2007. Construal and perspectivation. In Dirk Geeraerts & Hubert Cuyckens (eds.), The Oxford handbook of cognitive linguistics, 48–81. Oxford: Oxford University Press.Search in Google Scholar

Received: 2016-10-25

Revised: 2017-3-30

Accepted: 2017-4-15

Published Online: 2017-8-8

Published in Print: 2017-8-28

You are currently not able to access this content.

Articles in the same Issue

https://doi.org/10.1515/cog-2016-0124

Keywords for this article

viewpoint; gesture; indexicality; metonymy; immersion