Abstract
Many studies investigating the use and effectiveness of multimodal communication are now confronting the need to engage with larger bodies of data in order to achieve more empirically robust accounts, moving beyond the earlier prevalence of small-scale ‘case studies’. In this article, I briefly characterise how recent developments in the theory of multimodality can be drawn upon to encourage and support this change in both scale and breadth. In particular, the contribution will show how refinements in the degree of formality of definitions of the core multimodal constructs of ‘semiotic mode’ and ‘materiality’ can help bridge the gap between exploratory investigations of complex multimodal practices and larger-scale corpus studies.
References
Ayaß, R. (2015). Doing data: the status of transcripts in conversation analysis. Discourse Stud. 17: 505–528, https://doi.org/10.1177/1461445615590717.Search in Google Scholar
Bateman, J.A. (2020). The foundational role of discourse semantics beyond language. In: Zappavigna, M. and Dreyfus, S. (Eds.), Discourses of hope and reconciliation. On J. R. Martin’s contribution to systemic functional linguistics. Routledge Studies in Multimodality. Bloomsbury, London, New York, pp. 39–55.10.5040/9781350116092.ch-003Search in Google Scholar
Bateman, J.A. (2021a). Dimensions of materiality: towards an external language of description for empirical multimodality research. In: Jana, P., Wildfeuer, J., and Bateman, J.A. (Eds.), Empirical multimodality research: methods, evaluations, implications. de Gruyter, Berlin, pp. 35–63.10.1515/9783110725001-002Search in Google Scholar
Bateman, J.A. (2021b). What are digital media? Discourse, Context & Media 41, https://doi.org/10.1016/j.dcm.2021.100502.Search in Google Scholar
Bateman, J.A., Veloso, F.O.D., and Lau, Y.L. (2021). On the track of visual style: a diachronic study of page composition in comics and its functional motivation. Vis. Commun. 20: 209–247, doi:https://doi.org/10.1177/1470357219839101.Search in Google Scholar
Bateman, J.A., Wildfeuer, J., and Hiippala, T. (2017). Multimodality – foundations, research and analysis. A problem-oriented introduction. de Gruyter Mouton, Berlin.10.1515/9783110479898Search in Google Scholar
Bezemer, J. (2014). Multimodal transcription: a case study. In: Norris, S. and Maier, C.D. (Eds.), Interactions, images and texts: a reader in multimodality. de Gruyter Mouton, Berlin, pp. 155–170.10.1515/9781614511175.155Search in Google Scholar
Bezemer, J. and Mavers, D. (2011). Multimodal transcription as academic practice: a social semiotic perspective. Int. J. Soc. Res. Methodol. 14: 191–206, https://doi.org/10.1080/13645579.2011.563616.Search in Google Scholar
Cowan, K. (2020). Tracing the ephemeral: mapping young children’s running games. Des. Learn. 12: 81–93, https://doi.org/10.16993/dfl.90.Search in Google Scholar
Cowan, K. and Kress, G. (2017). Documenting and transferring meaning in the multimodal world: reconsidering “transcription”. In: Frank, S. and Gee, E. (Eds.), Remixing multiliteracies: theory and practice from New London to new times. Teachers College Press, New York, London, pp. 50–61.Search in Google Scholar
Crawford Camiciottoli, B. and Bonsignori, V. (2015). The pisa audiovisual corpus project: a multimodal approach to ESP research and teaching. E.S.P. Today: J. Engl. Specif. Purp. Tert. Level 3: 139–159.Search in Google Scholar
Flewitt, R. (2006). Using video to investigate preschool classroom interaction: education research assumptions and methodological practices. Vis. Commun. 5: 25–51, https://doi.org/10.1177/1470357206060917.Search in Google Scholar
Heath, C., Hindmarsh, J., and Paul, L. (2010). Video in qualitative research: analysing social interaction in everyday life. Sage, Los Angeles, London.10.4135/9781526435385Search in Google Scholar
Kress, G., Carey, J., Ogborn, J., and Tsatsarelis, C. (2000). Multimodal teaching and learning. Continuum, London.Search in Google Scholar
Kress, G. and van Leeuwen, T. (2006 [1996]). Reading images: the grammar of visual design. Routledge, London, New York.10.4324/9780203619728Search in Google Scholar
Lapadat, J.C. (2000). Problematizing transcription: purpose, paradigm and quality. Int. J. Soc. Res. Methodol. 3: 203–219, https://doi.org/10.1080/13645570050083698.Search in Google Scholar
Levinson, S.C. (2006). On the human ‘interaction engine’. In: Enfield, N.J. and Levinson, S.C. (Eds.), Roots of human socialiy: culture, cognition and interaction. Berg, Oxford, New York, pp. 39–69.10.4324/9781003135517-3Search in Google Scholar
MacNaught, L. (2019). Multimodal metalanguage. In: de Silva Joyce, H. and Feez, S. (Eds.), Multimodality across classrooms. Learning about and through different modalities. Routledge, London, New York, pp. 144–159.10.4324/9780203701072-10Search in Google Scholar
Malinowski, B. (1923). The problem of meaning in primitive languages. Harcourt, Brace, and Co., Inc, New York, pp. 451–510, Supplement I to C.K.I.A. Ogden and Richards The Meaning of Meaning.Search in Google Scholar
Mills, K.A., and Unsworth, L. (2017). Multimodal literacy. In: Oxford research encyclopedia of education. Oxford University Press, New York. http://education.oxfordre.com/view/10.1093/acrefore/9780190264093.001.0001/acrefore-9780190264093-e-232.10.1093/acrefore/9780190264093.013.232Search in Google Scholar
Mondada, L. (2007). Commentary: transcript variations and the indexicality of transcribing practices. Discourse Stud. 9: 809–821, https://doi.org/10.1177/1461445607082581.Search in Google Scholar
Mondada, L. (2008). Using video for a sequential and multimodal analysis of social interaction: videotaping institutional telephone calls [88 paragraphs]. Forum Qual. Sozialforsch./Forum Qual. Soc. Res. 9: Art.-39, https://doi.org/10.17169/fqs-9.3.1161.Search in Google Scholar
Murphey, K.M. (2005). Collaborative imagining: the interactive use of gestures, talk, and graphic representation in architectural practice. Semiotica 156: 113–145.10.1515/semi.2005.2005.156.113Search in Google Scholar
Norris, S. (2014). Modal density and modal configurations: multimodal actions. In: Carey, J. (Ed.), The Routledge handbook of multimodal analysis, 2nd ed. Routledge, London, pp. 86–99.Search in Google Scholar
Ochs, E. (1979). Social foundations of language. In: Freedle, R.O. (Ed.), Discourse processes: advances in research and theory. Volume 2: new directions in discourse processing. Norwood, New Jersey, Ablex, pp. 207–221.Search in Google Scholar
Pike, K.L. (1967). Language in relation to a unified theory of the structure of human behaviour, 2nd ed. The Hague: Mouton.10.1515/9783111657158Search in Google Scholar
Potter, J. and Cowan, K. (2020). Playground as meaning-making space: multimodal making and re-making of meaning in the (virtual) playground. Global Stud. Child. 10: 248–263, https://doi.org/10.1177/2043610620941527.Search in Google Scholar
Prior, P. (2013). Multimodality and ESP research. In: Paltridge, B. and Starfield, S. (Eds.), The handbook of English for specific purposes. Wiley-Blackwell, Chichester, U.K., pp. 519–534.10.1002/9781118339855.ch27Search in Google Scholar
Steen, F.F., Anders, H., Joo, J., Olza, I., Cánovas, C.P., Anna, P., Ray, S., Peter, U., Valenzuela, J., and Woźny, J., et al.. (2018). Toward an infrastructure for data-driven multimodal communication research. Ling. Vanguard 4: 20170041, doi:https://doi.org/10.1515/lingvan-2017-0041.Search in Google Scholar
© 2021 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Research Articles
- Towards developing multimodal literacies in the ESP classroom: methodological insights and practical applications
- Teaching communication strategies for the workplace: a multimodal framework
- Enhancing multimodal communicative competence in ESP: the case of job interviews
- Developing strategies for conceptual accessibility through multimodal literacy in the English for tourism classroom
- Engaging students in multimodal literacy practices in a university ESP context: towards understanding identity and ideology in government debates
- Growing theory for practice: empirical multimodality beyond the case study
Articles in the same Issue
- Frontmatter
- Research Articles
- Towards developing multimodal literacies in the ESP classroom: methodological insights and practical applications
- Teaching communication strategies for the workplace: a multimodal framework
- Enhancing multimodal communicative competence in ESP: the case of job interviews
- Developing strategies for conceptual accessibility through multimodal literacy in the English for tourism classroom
- Engaging students in multimodal literacy practices in a university ESP context: towards understanding identity and ideology in government debates
- Growing theory for practice: empirical multimodality beyond the case study