Home Growing theory for practice: empirical multimodality beyond the case study
Article
Licensed
Unlicensed Requires Authentication

Growing theory for practice: empirical multimodality beyond the case study

  • John A. Bateman ORCID logo EMAIL logo
Published/Copyright: September 30, 2021

Abstract

Many studies investigating the use and effectiveness of multimodal communication are now confronting the need to engage with larger bodies of data in order to achieve more empirically robust accounts, moving beyond the earlier prevalence of small-scale ‘case studies’. In this article, I briefly characterise how recent developments in the theory of multimodality can be drawn upon to encourage and support this change in both scale and breadth. In particular, the contribution will show how refinements in the degree of formality of definitions of the core multimodal constructs of ‘semiotic mode’ and ‘materiality’ can help bridge the gap between exploratory investigations of complex multimodal practices and larger-scale corpus studies.


Corresponding author: John A. Bateman, Linguistics, Fachbereich 10 Sprach- und Literaturwissenschaften, Universitat Bremen, Bremen, Germany, E-mail:

References

Ayaß, R. (2015). Doing data: the status of transcripts in conversation analysis. Discourse Stud. 17: 505–528, https://doi.org/10.1177/1461445615590717.Search in Google Scholar

Bateman, J.A. (2020). The foundational role of discourse semantics beyond language. In: Zappavigna, M. and Dreyfus, S. (Eds.), Discourses of hope and reconciliation. On J. R. Martin’s contribution to systemic functional linguistics. Routledge Studies in Multimodality. Bloomsbury, London, New York, pp. 39–55.10.5040/9781350116092.ch-003Search in Google Scholar

Bateman, J.A. (2021a). Dimensions of materiality: towards an external language of description for empirical multimodality research. In: Jana, P., Wildfeuer, J., and Bateman, J.A. (Eds.), Empirical multimodality research: methods, evaluations, implications. de Gruyter, Berlin, pp. 35–63.10.1515/9783110725001-002Search in Google Scholar

Bateman, J.A. (2021b). What are digital media? Discourse, Context & Media 41, https://doi.org/10.1016/j.dcm.2021.100502.Search in Google Scholar

Bateman, J.A., Veloso, F.O.D., and Lau, Y.L. (2021). On the track of visual style: a diachronic study of page composition in comics and its functional motivation. Vis. Commun. 20: 209–247, doi:https://doi.org/10.1177/1470357219839101.Search in Google Scholar

Bateman, J.A., Wildfeuer, J., and Hiippala, T. (2017). Multimodality – foundations, research and analysis. A problem-oriented introduction. de Gruyter Mouton, Berlin.10.1515/9783110479898Search in Google Scholar

Bezemer, J. (2014). Multimodal transcription: a case study. In: Norris, S. and Maier, C.D. (Eds.), Interactions, images and texts: a reader in multimodality. de Gruyter Mouton, Berlin, pp. 155–170.10.1515/9781614511175.155Search in Google Scholar

Bezemer, J. and Mavers, D. (2011). Multimodal transcription as academic practice: a social semiotic perspective. Int. J. Soc. Res. Methodol. 14: 191–206, https://doi.org/10.1080/13645579.2011.563616.Search in Google Scholar

Cowan, K. (2020). Tracing the ephemeral: mapping young children’s running games. Des. Learn. 12: 81–93, https://doi.org/10.16993/dfl.90.Search in Google Scholar

Cowan, K. and Kress, G. (2017). Documenting and transferring meaning in the multimodal world: reconsidering “transcription”. In: Frank, S. and Gee, E. (Eds.), Remixing multiliteracies: theory and practice from New London to new times. Teachers College Press, New York, London, pp. 50–61.Search in Google Scholar

Crawford Camiciottoli, B. and Bonsignori, V. (2015). The pisa audiovisual corpus project: a multimodal approach to ESP research and teaching. E.S.P. Today: J. Engl. Specif. Purp. Tert. Level 3: 139–159.Search in Google Scholar

Flewitt, R. (2006). Using video to investigate preschool classroom interaction: education research assumptions and methodological practices. Vis. Commun. 5: 25–51, https://doi.org/10.1177/1470357206060917.Search in Google Scholar

Heath, C., Hindmarsh, J., and Paul, L. (2010). Video in qualitative research: analysing social interaction in everyday life. Sage, Los Angeles, London.10.4135/9781526435385Search in Google Scholar

Kress, G., Carey, J., Ogborn, J., and Tsatsarelis, C. (2000). Multimodal teaching and learning. Continuum, London.Search in Google Scholar

Kress, G. and van Leeuwen, T. (2006 [1996]). Reading images: the grammar of visual design. Routledge, London, New York.10.4324/9780203619728Search in Google Scholar

Lapadat, J.C. (2000). Problematizing transcription: purpose, paradigm and quality. Int. J. Soc. Res. Methodol. 3: 203–219, https://doi.org/10.1080/13645570050083698.Search in Google Scholar

Levinson, S.C. (2006). On the human ‘interaction engine’. In: Enfield, N.J. and Levinson, S.C. (Eds.), Roots of human socialiy: culture, cognition and interaction. Berg, Oxford, New York, pp. 39–69.10.4324/9781003135517-3Search in Google Scholar

MacNaught, L. (2019). Multimodal metalanguage. In: de Silva Joyce, H. and Feez, S. (Eds.), Multimodality across classrooms. Learning about and through different modalities. Routledge, London, New York, pp. 144–159.10.4324/9780203701072-10Search in Google Scholar

Malinowski, B. (1923). The problem of meaning in primitive languages. Harcourt, Brace, and Co., Inc, New York, pp. 451–510, Supplement I to C.K.I.A. Ogden and Richards The Meaning of Meaning.Search in Google Scholar

Mills, K.A., and Unsworth, L. (2017). Multimodal literacy. In: Oxford research encyclopedia of education. Oxford University Press, New York. http://education.oxfordre.com/view/10.1093/acrefore/9780190264093.001.0001/acrefore-9780190264093-e-232.10.1093/acrefore/9780190264093.013.232Search in Google Scholar

Mondada, L. (2007). Commentary: transcript variations and the indexicality of transcribing practices. Discourse Stud. 9: 809–821, https://doi.org/10.1177/1461445607082581.Search in Google Scholar

Mondada, L. (2008). Using video for a sequential and multimodal analysis of social interaction: videotaping institutional telephone calls [88 paragraphs]. Forum Qual. Sozialforsch./Forum Qual. Soc. Res. 9: Art.-39, https://doi.org/10.17169/fqs-9.3.1161.Search in Google Scholar

Murphey, K.M. (2005). Collaborative imagining: the interactive use of gestures, talk, and graphic representation in architectural practice. Semiotica 156: 113–145.10.1515/semi.2005.2005.156.113Search in Google Scholar

Norris, S. (2014). Modal density and modal configurations: multimodal actions. In: Carey, J. (Ed.), The Routledge handbook of multimodal analysis, 2nd ed. Routledge, London, pp. 86–99.Search in Google Scholar

Ochs, E. (1979). Social foundations of language. In: Freedle, R.O. (Ed.), Discourse processes: advances in research and theory. Volume 2: new directions in discourse processing. Norwood, New Jersey, Ablex, pp. 207–221.Search in Google Scholar

Pike, K.L. (1967). Language in relation to a unified theory of the structure of human behaviour, 2nd ed. The Hague: Mouton.10.1515/9783111657158Search in Google Scholar

Potter, J. and Cowan, K. (2020). Playground as meaning-making space: multimodal making and re-making of meaning in the (virtual) playground. Global Stud. Child. 10: 248–263, https://doi.org/10.1177/2043610620941527.Search in Google Scholar

Prior, P. (2013). Multimodality and ESP research. In: Paltridge, B. and Starfield, S. (Eds.), The handbook of English for specific purposes. Wiley-Blackwell, Chichester, U.K., pp. 519–534.10.1002/9781118339855.ch27Search in Google Scholar

Steen, F.F., Anders, H., Joo, J., Olza, I., Cánovas, C.P., Anna, P., Ray, S., Peter, U., Valenzuela, J., and Woźny, J., et al.. (2018). Toward an infrastructure for data-driven multimodal communication research. Ling. Vanguard 4: 20170041, doi:https://doi.org/10.1515/lingvan-2017-0041.Search in Google Scholar

Received: 2021-03-09
Accepted: 2021-09-17
Published Online: 2021-09-30

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 26.11.2025 from https://www.degruyterbrill.com/document/doi/10.1515/mc-2021-0006/html
Scroll to top button