Multimodal digital semiotics: the interaction of language with other resources
-
Kay L. O'Halloran
Kay O'Halloran is Director of the Multimodal Analysis Lab, Deputy Director of the Interactive & Digital Media Institute, and Associate Professor in the Department of English Language and Literature at the National University of Singapore. Her areas of research include multimodal analysis, social semiotics, mathematics discourse, and the development of interactive digital media technologies and mathematical and scientific visualization techniques for multimodal and sociocultural analytics. She is Principal Investigator for large research projects in these areas in the Multimodal Analysis Lab at the National University of Singapore and has published widely in these fields. Address for correspondence: Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore, 9 Prince George's Park, Singapore 118408, Singapore 〈kay.ohalloran@multimodal-analysis.com 〉.
,
Marissa K. L. E
Marissa K. L. E is Research Associate at the Multimodal Analysis Lab in the Interactive & Digital Media Institute at the National University of Singapore. Her interests lie in systemic functional linguistics, multimodal discourse analysis, and their applications in analyzing the constitution, perpetuation, and evolution of ideologies and culture as manifested in traditional and emergent forms of media communication. She has worked on interdisciplinary projects involving the application of mathematical modeling techniques to the analysis of multimodal data and the development of interactive and collaborative software for multimodal analysis for research and educational purposes. Her publications reflect this synthesis of digital and traditional humanities in various areas such as climate science news, institutional business news, corporate advertisements, and education. Address for correspondence: 〈ekwanlinmarissa@gmail.com 〉.Alexey Podlasov worked as Post-Doctoral Research Fellow at the Multimodal Analysis Lab in the Interactive and Digital Media Institute (IDMI) at the National University of Singapore from 2008 to 2012. His background is in computer science and applied mathematics with research interests in data modeling, visualization and analytics, computer vision, image and video processing, as well as with software prototyping and development. He is currently working with photorealistic rendering techniques for mobile 3D graphics. Address for correspondence: 〈alexey.podlasov@gmail.com 〉.Sabine Tan is Research Associate at the Multimodal Analysis Lab, Interactive & Digital Media Institute, at the National University of Singapore. Her research interests include multimodal discourse analysis, social semiotics, and visual communication. She has published on topics that include the application of multidisciplinary perspectives within social semiotic theory to the analysis of institutional discourses involving traditional and new media, such as business news mediated on the Internet, corporate television advertisements, corporate Web pages, and other emergent multimodal discourse genres. She has worked on interdisciplinary projects involving the development of interactive and collaborative software for the multimodal analysis of images and video texts for research and educational purposes. Address for correspondence: 〈sabinetan@gmail.com 〉.
Abstract
A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allen's interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Halliday's systemic theory.
About the authors
Kay O'Halloran is Director of the Multimodal Analysis Lab, Deputy Director of the Interactive & Digital Media Institute, and Associate Professor in the Department of English Language and Literature at the National University of Singapore. Her areas of research include multimodal analysis, social semiotics, mathematics discourse, and the development of interactive digital media technologies and mathematical and scientific visualization techniques for multimodal and sociocultural analytics. She is Principal Investigator for large research projects in these areas in the Multimodal Analysis Lab at the National University of Singapore and has published widely in these fields. Address for correspondence: Multimodal Analysis Lab, Interactive & Digital Media Institute (IDMI), National University of Singapore, 9 Prince George's Park, Singapore 118408, Singapore 〈kay.ohalloran@multimodal-analysis.com〉.
Marissa K. L. E is Research Associate at the Multimodal Analysis Lab in the Interactive & Digital Media Institute at the National University of Singapore. Her interests lie in systemic functional linguistics, multimodal discourse analysis, and their applications in analyzing the constitution, perpetuation, and evolution of ideologies and culture as manifested in traditional and emergent forms of media communication. She has worked on interdisciplinary projects involving the application of mathematical modeling techniques to the analysis of multimodal data and the development of interactive and collaborative software for multimodal analysis for research and educational purposes. Her publications reflect this synthesis of digital and traditional humanities in various areas such as climate science news, institutional business news, corporate advertisements, and education. Address for correspondence: 〈ekwanlinmarissa@gmail.com〉.
Alexey Podlasov worked as Post-Doctoral Research Fellow at the Multimodal Analysis Lab in the Interactive and Digital Media Institute (IDMI) at the National University of Singapore from 2008 to 2012. His background is in computer science and applied mathematics with research interests in data modeling, visualization and analytics, computer vision, image and video processing, as well as with software prototyping and development. He is currently working with photorealistic rendering techniques for mobile 3D graphics. Address for correspondence: 〈alexey.podlasov@gmail.com〉.
Sabine Tan is Research Associate at the Multimodal Analysis Lab, Interactive & Digital Media Institute, at the National University of Singapore. Her research interests include multimodal discourse analysis, social semiotics, and visual communication. She has published on topics that include the application of multidisciplinary perspectives within social semiotic theory to the analysis of institutional discourses involving traditional and new media, such as business news mediated on the Internet, corporate television advertisements, corporate Web pages, and other emergent multimodal discourse genres. She has worked on interdisciplinary projects involving the development of interactive and collaborative software for the multimodal analysis of images and video texts for research and educational purposes. Address for correspondence: 〈sabinetan@gmail.com〉.
©[2013] by Walter de Gruyter Berlin Boston
Articles in the same Issue
- Masthead
- Editorial: The Halliday potential
- Introduction
- Applying systemic functional linguistics in healthcare contexts
- Users in uses of language: embodied identity in Youth Justice Conferencing
- Using systemic functional linguistics to explore digital technologies in educational contexts
- What do texts do? The context-construing work of news
- Evaluating experience in funny ways: how friends bond through conversational hum
- Interlingual re-instantiation – a new systemic functional perspective on translation
- Clause complex manifestation in depression
- Systemic functional linguistics, corpus linguistics, and the ideology of science
- Hallidayan systemic-functional semiotics and the analysis of the moving audiovisual image
- Multimodal digital semiotics: the interaction of language with other resources
- Visualizing patterns of appraisal in texts and corpora
Articles in the same Issue
- Masthead
- Editorial: The Halliday potential
- Introduction
- Applying systemic functional linguistics in healthcare contexts
- Users in uses of language: embodied identity in Youth Justice Conferencing
- Using systemic functional linguistics to explore digital technologies in educational contexts
- What do texts do? The context-construing work of news
- Evaluating experience in funny ways: how friends bond through conversational hum
- Interlingual re-instantiation – a new systemic functional perspective on translation
- Clause complex manifestation in depression
- Systemic functional linguistics, corpus linguistics, and the ideology of science
- Hallidayan systemic-functional semiotics and the analysis of the moving audiovisual image
- Multimodal digital semiotics: the interaction of language with other resources
- Visualizing patterns of appraisal in texts and corpora
