Epilogue

Like the authors in this special issue and many others, I found Gunther Kress and his ideas insightful and inspiring. He generated many significant concepts, ‘tools’ for thinking with which, underpinned by a set of social semiotic principles and political commitments at the centre of his work, laid the foundation for social semiotic multimodality. During his lifetime and since his sudden sad death, multimodal scholars (including myself) have worked to test, interrogate, extend, re-define, and/or challenge these concepts and collectively advance multimodal research. Kress’s dynamic approach to theorising fosters this type of intellectual work. He was never formulaic. His work was concerned with authentically responding to the artefacts and interactions that caught his attention, often everyday artefacts, and interactions (indeed he would actively seek out seemingly ‘banal’ or ‘mundane’ artefacts e.g., a child’s drawing) as sites of ideology. His research imagination was alert to the social insights he might gain by being grounded in the everyday world, rather than one of conceptual fixedness, precision or ‘being right’. He pushed the frontiers of theoretical exploration. Van Leeuwen, one of Kress’s significant collaborators, elaborates on his approach to semiotics, data, text analysis and theory-formation in this special issue. Kress shared ideas and concepts, for others to use and further refine. He worked to shape these concepts through his analysis, notably working on the concept of mode (Kress 2016). This special issue is further evidence that Gunther’s theoretical work continues to provoke productive critical engagement through its interrogation of a cluster of multimodal concepts (agency, affordance, design, motivation, materiality, meaning-making, rhetorical aptness, semiotic mode, and sign), all of which are brought newly into focus through this collection of papers.

My appreciation of Kress’s influence on multimodality is grounded in our collaborations, and my work as a founding co-editor of the journal, Visual Communication, and Multimodality & Society (established in 2020), which aims to consolidate and advance multimodal theory, methodologies, and empirical understanding of interaction and communication. These two journals continue to grow alongside multimodality. Indeed, there are not many papers in either journal that do not reference at least one of Kress’s many publications or his and van Leeuwen’s seminal book, Reading Images (1996), now in its third edition, which continues to have a major impact on multimodal scholars working with the visual. Many of the papers published in these two journals and others push at multimodal theoretical and methodological boundaries through the work of critiquing, mapping, consolidating, and advancing multimodal theory, concepts, and methods. The work of opening multimodality up to critique and questioning its role within the academic disciplinary landscape is crucial to its future (Bateman 2022), and this special issue reflecting on Kress’s contribution is a significant contribution to this unfolding dialogue.

Change is a central concern of multimodality as are the new semiotic regimes in contemporary society. It attends to how existing semiotic resources are used in new ways and the discovery of novel semiotic resources, both of which can provide exciting and innovative directions for researching digital technologies. When a new technology enters the Technoscape (Appadurai 1990), societies reach a consensus over time and develop a set of norms for their use that shape the ways we communicate. With each new technology, the process begins again (Licoppe 2004). Previous multimodal work has pointed to the ways that people’s use of a wide range of technology has reshaped meaning making resources and practices over time. Kress got there early, as always, with his book Literacy in the New Media Age (2003). My work builds on and extends this work by bringing a multimodal concern with processes of meaning making and the dynamic ways in which semiotic resources and the social shape one another to explore emergent technologies in this shifting communicational landscape.

The papers in this special issue have interrogated the futures of individual multimodal concepts; in this epilogue I point to the future possibilities of stretching a multimodal perspective by combining it with other methods and approaches (Jewitt et al. 2016). Interdisciplinarity often involves exploring multimodality in new empirical contexts, which challenges it as this often requires stretching its concepts, and further refining them to address new questions and do new work. For example, recent multimodal studies have used mixed methods to look at bodily movement in space (McMurtrie 2022), taken a historical perspective on multimodality (Paterson 2021), explored synergies with digital humanities on distant viewing (Hiippala 2021), explored the potentials of multimodal anthropology to engage with memory, senses, feminism and urbanism, migration, and the colonial gaze (Moretti 2021), taken an interdisciplinary approach to touch (Jewitt et al. 2022b), and brought multimodality to questions of race (Sekimoto and Brown 2023). This resonates with Stöckl’s call in this special issue, that multimodality would benefit from bringing in ideas on pragmatic and cognitive mechanisms and explore phenomena through multimodal corpora.

There are many new areas where the influence of multimodality is beginning to be felt and papers are emerging. Three of particular interest to me are multimodal sensory studies (e.g., Allen 2023; Harris 2021), Human Computer Interaction (HCI) (e.g., Cope and Kalantzis 2023; Khot et al. 2021; Turchet 2022), and embodiment (e.g., Ericsson 2023; Gorham and Amgott 2024; Samuelsson 2022) as they sit at the multimodal intersection of socio-technical futures. These studies bring multimodality into new conversations with other perspectives and disciplines.

Interdisciplinarity is at the core of my work and the InTouch project, a five-year exploration of touch and its digital remediation (Jewitt and Price 2024). InTouch deploys multimodality to theorise how the use of emergent touch technologies are reshaping the digital communicational landscape or might do so in the future. The sensory and the social are both paramount in the development of digital touch devices and environments in ways that point to the ‘shifting, contingent, dynamic and alive’ character of the senses, specifically in this case, touch (Jones 2007: 8). The ever-closer relationship between the sociality of touch, technology, and sensory communication underpinned the decision to combine a multimodal approach, sensory ethnography (as well as collaborations with artists and computer scientists) to ask how the now ubiquitous use of touch-interfaces and new digital devices are reconfiguring who, what and how people touch.

InTouch approached touch at a crucial period of its development into a digital mode that can serve a full range of semiotic functions within a community. A fundamental premise of multimodal approaches is a concern with the cultural and social resources for making meaning rather than with the senses. However, while it is important not to conflate the senses and the means for making meaning, there are important relations to be explored between the two. For instance, while touch is not ‘new’ the sensory resources of touch are being semiotically shaped and brought into the frame of digital communication in some new ways. This analytical approach combined a multimodal and sensory perspective to bring the body, technology, and environment into dialogue through a simultaneous concern with the semiotic, material, and experiential dimensions of touch (Jewitt and Leder Mackley 2019; Jewitt and Price 2024). This connects, albeit from a different starting point, with Selander’s call in this issue for a stronger focus on the role of bodily experience and digital media.

We mapped the semiotic resources and affordances of touch, the qualities of touch and their experiences and associations, and the meaning potentials these represent, as a descriptive inventory of the resources and types of touch made available always situated in the sign maker’s broad social, cultural, and historical context. Bezemer’s paper in this special issue engages with the concept of affordances, one which has a long history and is a much-critiqued term (Oliver 2005), yet it persists. He reminds us that Kress understood that affordances always need to be considered in context: “features of material form and social convention … alongside the sign maker’s lifeworld, audience, situation, and conditions of sign making”. We approached digital touch as a mode of communication, albeit one that is currently in a state of flux and development, a sensorial experience entangled in the materiality and sociality of the body, the environment, and technologies. Specifically, we asked how modes and media shift (a distinction that Stöckl, this issue, usefully elaborates on) and reconfigure through their digital remediation, and what new affordances (potentials and constraints) for meaning making and communication this may herald.

Multimodality helps me to explore how technologies are inextricably embedded in our lives, our relationships, how we communicate and interact. For me, it is central to understanding how technological innovation and the social and cultural histories, practices, and contexts mutually shape and transform one another. For instance, the personal social relationships that technologies mediate also constantly shape technologies through the ways that people take up and use them. Multimodality helped InTouch to investigate these semiotic potentials at the same time as we question why touch is newly important now. What is it that has brought it to the fore in contemporary society? How does it relate to other significant social conditions that are shaping the communicational landscape?

Touch is socially grounded and expressive, connected to social and individual tactile trajectories, emotions, and histories, creating a complex space for touch technology to mediate. Technological innovation has a long history of trying to bring touch into the digital realm (Parisi 2018), and there is considerable interest in the technologies of touch and their futures (Huisman 2017), with touch increasingly central to the futures imagined for virtual and immersive reality. Advances in haptic technologies enable new ways to remotely communicate a feeling of touch, however they are unable to replicate the cutaneous and kinaesthetic properties of physical touch, and most typically rely on vibration, pressure, and temperature. Nevertheless, these degraded touch cues have been shown to successfully create meaningful tactile experiences, trigger emotions akin to physical touch and support remote communication (Huisman et al. 2016; Parisi 2018; van Erp and Toet 2015). Yet, touch, being a complex but critical sense in human interaction and communication, presents ongoing challenges for technological design and development, notably how to embed digital touch into social sensory contexts of established touch communication (Price et al. 2022).

While multimodality offers a way to explore the digital remediation of touch and its potential to stretch the possibilities of how we ‘feel’ the world around us – the how, what, whom and when of touch – multimodality has largely neglected touch, with a few exceptions (Cranny Francis 2013). Within multimodality, objects and sequences of interaction are understood as meaningful signs – the outcome of a person’s or people’s actions, imbued with the maker’s interests, mediated through the environment in which the sign was produced or encountered (Kress 2010). The role of motivated sign, brought into new focus by Hodge’s paper in this special issue, remains a useful concept for many multimodal scholars including myself. Meaning is understood as a socially situated choice from a (dynamic) set of available resources. The affordances of these resources are shaped through their historical, cultural, and social usage and their materiality – all of which relate to and are shaped by technologies.

The different affordances associated with modes of communication, such as touch or visual, affect the kinds of semiotic work that they can be used for, the ease with which a meaning can be achieved, and the ways in which different modes can be used to achieve broadly similar semiotic work (Jewitt 2017). We explored how people create meaningful touch experiences through their selection of semiotic and sensory/experiential resources available to them in a variety of communicational contexts including parenting (e.g., Jewitt et al. 2021a, 2021b), personal relationships (e.g., Jewitt et al. 2020), virtual reality environments (Jewitt et al. 2022b) and work-placed robotics (Barker and Jewitt 2021, 2022). Multimodality enabled us to describe (and categorize) the social and material resources and affordances of digital touch, as well as how these are used by people to make meaning. For instance, how remote couples use digital devices to send and receive tactile messages by configuring the resources of touch (e.g., temperature, pressure, texture) to generate shared meanings and to communicate connection (Price et al. 2022). We used multimodality to map the emergent dimensions of digital touch, the social conditions and the contexts that shape it, as well as to characterise people’s use of touch for communication with attention to the cultural and social norms and power relations that shape their use (Jewitt and Price 2024).

Multimodality asks how meaning is made and communicated, what meanings are made, and by whom, in contrast a sensory approach accounts for the experiential aspects of meaning. Sensory studies (e.g., Howes 2004) argue for placing sensory experiences at the forefront of cultural analysis. From this perspective the senses are gateways of knowledge, instruments of power, sources of pleasure and pain – and are subject to significantly different constructions in different cultures, societies, and periods of history. In short, touch is part of the wider sensorium which is shaped by social, cultural, and environmental factors and its functions vary considerably across cultures and historical periods (Classen 2012).

Indeed, what comprises touch as a sense, its relationship to the senses more generally has been a matter of continual challenge and contestation (Macpherson 2011). Touch is a part of how meaning is perceived, and sensory ethnography can support multimodality through a set of phenomenological approaches that are attuned to people’s sensory worlds that are often tacit, embodied, and unspoken (Barker and Jewitt 2023; Pink 2009). This approach enables us to find routes through which to share or imaginatively empathize with the actions of people, collaboratively exploring participants’ ways of knowing, being and doing, whilst drawing on their own embodied and emplaced understandings in ways that account for their multi-sensoriality and functions, as a way for the researcher to feel their way back into the research context (Jewitt et al. 2022a). Combining multimodality and sensory approaches to analyse digital touch communication grounds the analysis in the broad social framing provided by both approaches to emphasize the social-cultural embedded-ness of digital touch and to capture the nuances of the lived sensory accounts that shape the digital design and use of touch.

Throughout our studies, participants made sense of digital touch through the multimodal orchestration of resources rooted in their material expectations and experiences of touch, sensory-bodily experiences, multimodal representations, and social imagination of touch (Jewitt and Price 2024). It is significant that, in isolation, none of these resources were meaningful; rather meaning was consistently realised through the dynamic social and sensorial encounters between body, technology, and social context. In short, making sense of digital touch was an integrated multimodal and multisensory accomplishment rather than either a wholly technical or sensory-physical endeavour (the two primary factors attended to in the design and research of digital touch).

The extent to which our study participants across the project generated a sense of digital touch differed; some felt it as weak, others as a partial kind-of-touch, others a different kind of touch, and for some, in some moments, a ‘like in real-life’ touch. Through the designed dynamic collision between sensory physical, semiotic resources, and experiential resources for meaning-making the digital is reshaping touch. This points to digital touch being beyond as a solely physical experience to an extended experience shaped through social, cultural, historical, and technological factors. Given the emergent state of digital touch technologies it is unsurprising that people in our studies often found the material basis of digital touch experiences lacking. What is surprising, however, is that against a backdrop of tactile limitation and lack the majority reported experiencing a meaningful sense of touch in digital environments.

The participants experienced challenges in orchestrating semiotic and experiential resources to create meaningful touch experiences. They brought their different ‘interests’, touch trajectories and histories to the process of meaning making and in doing so they created different routes to touchy engagement with the future possibilities of this complex emergent digital landscape. It also points to the ways in which the physicality of touch and the body, albeit differently configured, persists, and contributes to the realisation of digital touch and creates tactile continuities between the virtual and the real.

Our multimodal mapping of digital touch highlighted the ways in which touch is reconfigured through its remediation to, on the one hand, strip-back touch in ways that lose its nuance, individuality, and affect; and, on the other, open up possibilities for painless or effortless touch, enable touching in new ways or with things or environments that we cannot usually touch ‘in real-life’, to provide new kinds of proximity and interactions to objects and experiences, as well as altering the consequences for breaking touch norms.

As digital touch increasingly enters the technoscape, many questions are raised including concerning tactile continuities and change, modal-ambiguity, and multimodal sensory integration. Other emergent technologies of the near future will raise yet more questions and methodological challenges, ones that a multimodal approach and concepts set out and newly articulated in this special issue might help to address. My work shows that a multimodal perspective can be combined with a sensorial one, and that this exploratory interdisciplinary methodological lens can be trained on continuities and changes to help understand how semiotic resources and wider discourses (e.g., of digital touch) are reshaped through technologies. This is vital as digital communicational devices and systems move ever nearer to the body, drawing in sensory and embodied forms of interaction. Interdisciplinary scholars demonstrate that multimodality can be brought into new productive dialogues with many other approaches. To expand its explanatory reach and power we need to build on Kress’s theoretical legacy and its potential to theorise new semiotic forms, and to continue the conceptual work exemplified in this special issue. I am confident that Gunther Kress’s work will continue to inspire and foster dialogue and this special issue makes a significant contribution to this long conversation.