Band 34, Heft 3 - Special Issue: Multimodality, meaning-making, and the issue of “text”

Aiming to trace emerging conventions in video interaction, the paper adopts Kress's (2010) notions of prompt and response to examine how video responses relate to one of YouTube “Most Responded videos.” In the interactants' creative use of the video response option, which affords multimodal text production through copy and paste, any implicit or explicit element of the responded video can prompt a response. Exchanges can be (a) fully cohesive and attuned; (b) cohesive and variously coherent; (c) cohesive but incoherent; (d) marginally related; (e) non-cohesive and inferentially related; or (f) can present no clues of relatedness. Driven by the participants' diversified interests, videos are often linked as responses disregarding the meaning of the responded video, while the selection and recontextualization of previously made texts in new exchanges re-shapes and even scatters patterns of cohesion and coherence. Instead of hampering communication, these incoherent chains are accepted and acknowledged by interactants. Like other forms of contemporary communication produced through copy-and-paste techniques, video exchanges frequently prioritize an interested re-interpretation, transformation, assemblage, and recontextualization of signs/texts, often irrespectively of the authors' intended meaning. The analysis suggests the need for a reformulation of the criteria defining text and successful communication in contemporary forms of text production.

This article examines the social shaping of texts in digital environments and proposes the notion of migrating literacies to theorize how urban youth materialize meaning using grammars of digitally enabled design. A working definition of migrating literacies identifies digitally enabled text making as having social, technological, and semiotic affordances for managing affiliations across discourse communities. Drawing from ethnographic data, the analysis will account for the appearance of cultural remix through the shaping of multimodal ensembles and how they move across digital spaces. Key findings contribute to the need for addressing the challenge of multimodality to understanding shifting notions of text in an increasingly diverse and digital society.

The aim of this paper is to investigate the postmodern multimodal language of some contemporary American TV series and its gradual hybridization with the language (and practices) of social media. The paper begins with the analysis of popular television texts that have experimented with visual/musical/narrative form in order to outline the main characteristics of their meaning-making processes. Then, it will explore the concept of “expanded transmediality” according to which audiences participate in the co-construction of narratives (Stein and Busse 2012) to highlight the top-down/bottom-up ongoing exchange in franchise storytelling and it will focus on the textual features of some fan-generated artifacts that enter into these global narrations and where fans exploit postmodern diegetic and editing forms. Finally, the paper will return to the analysis of television to investigate how the on-screen emergence of social media networks has influenced postmodern TV series in terms of their semantic and semiotic realization and/or fruition processes, giving rise to a ‘webridization’ of postmodern television textuality.

This article reports a longitudinal study of new media and digital technologies producers (Rowsell 2013) looking at their multimodal logic and practices to challenge notions of text and multimodality. Focusing on filmmakers, I build on previous research (Sheridan and Rowsell 2010) to extend traditional notions of print-based texts to more contemporary ways of making meaning with moving-image texts. Working within a multimodal framework (Kress 1997, 2010), I present the logic and practices of two producers. One filmmaker produces documentaries about wide-ranging topics from cricket to Jim Carrey to sex scandals and religion. The other producer creates 3-D animated “texts” for film and television. Both are assiduous about their process and product, both highly competent at editing filmic texts, both intimately acquainted with the art and logic of multimodality. Their production stories and expertise inform the article to challenge perceptions of what modes can do and what they can evoke. Whether it is done through expressions, movements, images, sounds, filmmakers exploit the affordances of modes to emotionalize moving-image texts.

This paper provides a theoretical framework for the study of the video chat, a spontaneous web-based synchronous text that allows forms of communication in which social semiotic resources come into play and produce a new terrain of investigation for researchers in the field of linguistics, multimodality, communication and media studies, visual ethnography, and digital literacy. In particular, the paper singles out some aspects for analysis, such as the alternation of speech and writing, new proxemic and kinetic patterns, gaze management, and impossibility of eye contact and discusses some examples from digital field work on multiparty video-based interactions. Speech and writing are technologically integrated, allowing participants to mode-switch , i.e., to alternate between spoken and written discourse. New arrangements of verbal and nonverbal resources attempt to simulate face-to-face conversations. However, the illusion of a face-to-face conversation dissolves as soon as video chat-specific resources are unpacked. Despite growing research into nonverbal behavior, video chat data challenge visual analysts and researchers for a number of reasons. A transcription model, developed for the purpose of analyzing these specific texts, will be sketched to give a brief account of significant data that need to be incorporated into multimodal transcription and annotation. Reflections and conclusions are drawn according to the contribution that intersemiotic studies can potentially provide for web text analysis, given the constant expansion of web-based texts and the challenge it brings as regards new notions of textuality.

The ubiquitous software PowerPoint has significant influence on evaluations of professional and academic success, and has attracted considerable attention from both social commentators and researchers in various fields. Existing research on PowerPoint considers the software, slideshows created with it, and PowerPoint-supported presentations in isolation from each other and is therefore unable to promote better understanding of the interaction between the software's design and its use. This article proposes a model for exploring this interaction. Specifically, it introduces a multimodal social semiotic approach to studying PowerPoint as a semiotic practice comprising three dimensions – the software's design, the multimodal composition of slideshows, and their presentation – and two semiotic artefacts, the software and the slideshow. It discusses the challenges each dimension presents for discourse analysis and social semiotic research, focusing especially on the need to step away from the notion of text and to develop a holistic, non-logocentric, and adaptive multimodal approach to researching semiotic technologies. Using PowerPoint as a case study, this article takes a step toward developing a social semiotic multimodal theory of the relation between semiotic technologies, or technologies for making meaning, and semiotic practices.