Meaning-making and transformative engagement – notes on Gunther Kress’s social semiotic and multimodal approach to learning

1 Introduction: the question of learning

During the 1970s and 80s, studies of learning shifted from psychologically oriented approaches, such as Watson’s (1930) and Skinner's (1965) behaviourism and Piaget’s (2002) approach to the development of thinking, towards sociologically oriented approaches, as, for example, in the work of Vygotsky (2001) and Säljö (2021). At the same time there was an increasing amount of learning research in cognitive science and neuroscience, and in pedagogy and didactics. “Learning” became a broad area, covered by the term “learning sciences”.

Another approach towards the understanding of learning came from research informed by functional linguistics, where Gunther Kress and Theo van Leeuwen developed multi-modality within the frame of social semiotics, starting out with the book Reading Images (Kress and van Leeuwen 1990). In this article, I draw on my collaboration with Gunther Kress and his co-workers in London, which led towards a multimodal, design-oriented approach to learning that emphasised learning environments, learning resources and meaning-making activities (Björklund Boistrup and Selander 2022; Kress and Selander 2012; Kress et al. 2021b; Selander 2017; Selander and Kress 2021).

My discussions with Gunther Kress included members of our two research groups, and resulted in a wide range of studies and projects inspired by social semiotics and multimodality: museum exhibits and visitors’ meaning-making (Insulander 2010); collaboration with professionals working in schools and museums (Insulander and Öhman 2022); collaborative design work with professionals at The Swedish Museum of Natural History in order to develop a new, multimodal exhibit (Insulander and Svärdemo-Åberg 2022); collaboration between researchers and teachers to develop a new digital educator’s tool for pre-schools (Kjällander 2022); multimodal knowledge representations and re-designs – a case study of film-making in schools (Lindstrand 2022); semiotic and modal affordances in representations of dinosaurs (Lindstrand 2022); the role of teachers’ body language in mathematics education (Björklund Boistrup 2010); social interaction in the social science classroom (Kjällander 2011); and knowledge representation in a digitized school (Åkerfeldt 2014).

Against this background, I will reflect on Gunther Kress’s views on learning as meaning-making and as a transformative activity, and on his emphasis on agency and the question of whose voice is heard and whose voice counts as valid. Finally, I will address two themes: (1) How can we “see” learning?; and (2) How can we understand the role of context, and, related to this, of implicit learning and the hidden curriculum? It is important to note that Kress’s views on these central topics were never carved in stone. He constantly reflected on, and over time also partially changed the meanings and uses of key concepts such as “mode” and “semiotic resource”. Is “writing”, he would ask, the same mode when it is produced by hand, by typing on a typewriter or using the computer keyboard? Are there not differences in terms of planning, changing a text, linking it to other texts, and so on, even though the same words are written in each case? (Kress 2010: 84). He also increasingly emphasised the learning potential of changing modes, for example reading a printed text and then re-presenting it as a new text or as a video, and, in this connection, distinguished between transformation, a change within the same mode, and transduction, a change into a different mode. However, in all this, ‘mode’ remained a useful concept for analysing aspects of knowledge representation and sign-making, and hence of learning, while also showing the strength of a research perspective that can respond to changes of meaning-making in a changing world.

2 Learning as meaning-making and transformative engagement

In this section, I elaborate some central concepts in Gunther Kress’s approach to the question of learning. For Kress, learning was associated with creativity and ‘sign-making’:

Learning is not a term that belongs in semiotics; sign-making is. However, learning and sign-making are two sides of one sheet of paper […] Both are […] dynamic resources which change the resources through which the process takes place […] and change those who are involved in the process. (Kress 2003: 40)

And he emphasised that “knowledge production” is an active process:

That is, knowledge is always produced rather than acquired […] There is a need for careful considerations of designs for meaning and knowledge-making: the shaping of routes and environments of meaning making and production of knowledge […] (Kress 2010: 27).

Three aspects are fundamental in this process: information ‘afforded’ by the environment, the learner’s own interest (“prompt”), and learning as a sign-making, as a transformative or transductive activity. To begin with the concept of affordance, Kress was initially very much focussed on affordance in terms of re-presentation, even though he also referred to the interactive aspect of modes and interest:

The distinct affordances of modes offer the potential for better transcriptions of the world or those parts of it that we want to transcribe, along the lines that is offered by each sign (Kress 2010: 102).

But he also referred to the interactive aspects of modes and to the learner’s interest: something must act as a prompt, capturing the learner’s attention:

At any one time, any aspect of the complex dynamic communicational ensemble might be significant for the learner/trainee, so he has to be constantly and entirely attentive to potential significant cues as potential prompts. It is his interest […] that turns any of these – or none – into a prompt for him (Kress 2010: 33).

Finally, from a social semiotic and multimodal perspective, learning is first and foremost a transformative engagement. It is:

[…] the result of the transformative engagement with an aspect of the world which is the focus of attention by an individual, on the basis of principles brought by her or him in that engagement; leading to a transformation of the individual’s semiotic/conceptual resources (Kress 2010: 182).

Learning therefore involves the interpretation and re-configuration of signs in the environment, which Kress discusses in terms of transduction and transformation:

Transduction is seen as subordinate to – as one kind of – translation. It names the process of moving meaning-material from one mode to another – from speech to image; from writing to film. As each mode has its specific materiality – sound, movement, graphic ’stuff’, stone and has a different story of social uses, it also has different entities. […] Transformation is a less far-reaching process than that of transduction. It describes processes of meaning change through re-ordering of the elements in a text or other semiotic object, within the same culture and in the same mode; or across cultures in the same mode. In other words, the translation of a novel from German to English would fall under the heading of transformation. (Kress 2010: 125, 129)

These concepts apply, for instance, to the act of reading. Reading is always both an act of communication and an act of learning. Information is chosen, picked up and collected, and meaning is selected, revitalized, combined in partly new ways and re-designed into a new configuration. Or as Borges (2000: 6) phrased it: “Art happens every time we read a poem.” Olsson (2015: 135ff) uses the term imminent for this process: ‘Imminent’ (a word which does not exist in Swedish language) comes from the Latin imminere and has future-oriented meanings such as “impend” and “control” and also “call for attention”. Learning can therefore be understood as an active orientation towards phenomena that are then picked up and either transducted or transformed into a new constellation by way of sign-making.

The concept of “sign-making” can also give us clues as to whether, and what kind of, learning actually takes place. It allows us to observe “signs of learning”, to see whether, and if so how, new methods and tools have been used, and whether, and if so how, known methods and tools have been used in new ways. It might be worth mentioning that, as we discussed the idea of signs of learning (e.g., Selander 2008a: 43), Kress initially worried about it, although he later incorporated it in his writing, focussing on what is seen as, and counted as, learning (see also Kress and Selander 2012):

The focus of our analyses of learning and teaching is twofold. On the one hand we look at the existing distributions of agency in learning and teaching. We ask: who acts, where, when, how, with what means? Whose actions are accorded recognition and value by whom? […] On the other hand, we look at learning by documenting individual change in an environment of constant social change. That is, we see existing distributions of agency in constant, ongoing, transformations, usually slow, subtle, nearly imperceptible; at times with great rapidity, producing large-scale changes. Social and individual change is inextricably connected with and reflected in changes in the representational and communicational landscape. (Kress et al. 2021a: 72)

All this allows us to differentiate between: (a) already given knowledge representations, for instance the “affordances” of school textbooks and other learning resources; (b) the signs that actually trigger the learner’s interest at a certain moment (“prompts”); and (c) the new knowledge representations the individual learner constructs (see for example Kress 2003: 76ff, and 33–36, respectively). It follows that, if we want to design relevant learning environments, we need to understand learners’ individual preferences for the kind of resources they would like to use, or feel comfortable with, in their learning (Selander and Kress 2021).

3 Learning as sign-making

A number of other aspects of learning may be mentioned, all of them related to the question of learning and sign-making as “two sides of one sheet of paper”, as stated above. Firstly, in most semi-formal or non-formal learning situations, learning starts out from ‘interest’ and leads to an increased capacity to take part in specific domains of activity in a meaningful way, which includes knowledge, skills, judgements, collaboration and coordination of activities (Selander 2017, 2022). In Lindstrand and Selander (2022) we have referred to this in terms of “designs in learning and rhizomatic webs”. In formal learning sites, as in the school context, the teacher decides – on the basis of the curriculum – what shall be in focus, and what shall be counted as learning and knowledge.

Secondly, learning takes place in the process of communication. In communication we collaborate and negotiate, and coordinate emotions and actions, whether in peaceful or conflictual interactions. We use different modes and media – we make signs, and the signs that we make reveal our interests, capabilities, and fixations. And like learning, every communicative situation changes something, if only minimally. This means that the meaning of an utterance can rarely be determined once and for all, since in most cases there will be something new to add (Bakhtin 1988; Kress 2010). It also seems obvious that negotiation between two points of view not only changes the arguments as such, but also displaces their specific starting-points (Karcevskij 1982). Even the curriculum will be in flux during the process of teaching and learning.

Thirdly, to recognize learning, we need to introduce the dimension of time. In a specific situation, we cannot know whether new learning has occurred or whether the situation has brought older learning to the fore, unless we can refer to other, earlier situations. In order to observe and measure learning in terms of new skills, new knowledges and new capabilities, we need to observe individuals or groups on (at least) two different occasions – whether in formal, semi-formal or non-formal learning situations (Selander 2008a, 2022). What we see as learning will also always be related to the kind of resources (modes and media) that have been available at these occasions, as well as to the relevant cultures of recognition (Kress and Selander 2012).

Finally, in a school context, learning does not only entail knowing new facts or acquiring new capabilities for analysing phenomena in relation to specific goals or aims. It can also be indirect – the so-called “hidden curriculum”: learning what it means to be in a school and what kinds of interactions the school expects from learners. This involves another analytical layer, the layer of social and political aspects: “Learning in institutional settings is a political matter and as such highly subject to power and ideology” (Kress 2010: 178). It follows that we need to be a bit more specific about context, and what we mean by it, as I will discuss in more detail below, and also that we need to distinguish between “cultures of recognition” and “assessment standards and procedures”. The concept of “cultures of recognition” focuses, as mentioned, on what is seen as and recognized as learning and knowledge in a certain context, whilst “assessment standards and procedures” focus on specific instruments used to measure knowledge or learning (Kress 2010: 128; Kress and Selander 2012). It seems that in most school contexts it is knowledge, not learning, that is measured – that the focus is on how pupils can give standard answers to standard questions within the frame of school knowledge. If we were actually interested in learning in a broader sense, we would use portfolios and allow for a much greater variety of material and semiotic resources. It is also clear that Kress’s social-semiotic, multimodal approach to learning focuses on learning in the widest sense of the word, on the different resources needed for different kinds of knowledge, and on questions of designs of meaning and knowledge making in general (Kress 2010: 27), whether in non-formal, semi-formal or formal learning sites (Selander 2022), rather than on mastering the requirements of school disciplines, as in recent Australian work on learning based on genre theory (e.g., Derewianka 2015).

4 How can we understand context?

Context, as a broad term, is central to Kress’s understanding of learning. The connection between human beings and their environments has been a subject of lively discussion during the last decades (see for example Knappett 2005). Hutchins (1996: xiii) formulated it as follows:

The emphasis on finding and describing ‘knowledge structures’ that are ‘somewhere’ inside the individual encourages us to overlook the fact that human cognition is always situated in a complex sociocultural world and cannot be unaffected by it.

It is therefore difficult to make a sharp distinction between the individual and his or her context. However, for analytical reasons, a number of remarks can nevertheless be made. Firstly, contexts can be seen as dynamic and interrelated. Bernstein (1973: 30), for instance, formulated the principles of de-contextualizing and re-contextualizing in formal education as follows:

Any formal educational experience entails de-contextualizing and re-contextualizing. In ways we do not properly understand, informal everyday experience, everyday communication within the family and peer group, creates procedures and performances fundamental to formal education. However, formal education selects, re-focuses and abstracts from such experience, and in so doing de-contextualizes them.

Bernstein also analysed the differences between working class children’s “restricted and context-bound codes”, and middle class children’s “elaborated and context-independent codes,” with the latter closer to the codes that dominated the school context. In this way, Bernstein connected contexts and (social) codes. Kress (1993: 187) argued that a “[…] semiotic approach would attempt to dissolve the category of context itself, preferring to speak of series of interrelating semiotic systems”.

The concept “context” can mean different things at different social levels. Are we, for example, talking about classes, groups or individuals? And how do we understand stability versus change (Gustavsson and Selander 2011)? Or, are we more interested in social institutions and the degree to which they influence “thought styles” (Douglas 1996)? In short, we need to be aware of the delicate balance between considering contextual categories and keeping an open mind in our acts of interpretation, not least, as noted earlier, because both communication and learning involve acts of creation. Human action is never fully determined by context. There is always a risk of “over-interpretation” of the role of context, of deciding a priori what to look for in a text, and then “finding” exactly this, in a kind of circular argumentation. Our already established categories then lead us to see specific things, but do not allow us to see the unexpected, the obscured, the borderline cases, or the anomalous.

Gunther Kress, however, always emphasised the sign as a carrier of social and political (contextual) relations and underlined that the concept of semiotic mode must be understood in relation to context:

The first is: ‘What is the social and cultural domain that it covers or that it does not cover?’. The second is: ‘What can this mode – image, speech, gesture, writing – do in the cultural domain that it “covers” and what can it not do?’. The third is: ‘What semiotic features are in the mode and which are not, and why?’ (Kress 2010: 84).

But he also had an eye for the specific, “the making of signs now, in this environment, for this occasion” (Kress 2010: 13), and for creativity.

Competence is always limited by what is socially regarded as competence […] whereas design is not limited by a framing. You can make new things and my notion of the sign is that signs are always newly made, and they are used in designs that will always be, in some ways, different (Kress, quoted in Andersen et al. 2018: 84).

Nevertheless, in a formal learning context, much of what is called “learning” is focused and regulated by curricula and tests – as explicit expressions of rules and ideologies. This can affect both teachers’ and pupils’ understanding of what learning is about in this context. As a consequence, pupils may develop different strategies for coping with institutional rules and regulations: (a) they can accept what is going on simply because they like the subject; (b) they can accept it, not because they like the subject, but because, with an eye to their future, they want to achieve good marks; or (c) they can disagree, oppose, resist, or turn away from what school is about (Selander 2017). All this touches upon the dynamic relation between individual learners and the way institutions frame our ways of thinking about learning (Selander 2018).

5 Learning resources and a new media landscape – the role of technology

Today, the concepts of learning and context inevitably have to be related to the new media landscape and the new social conditions we are facing (Marimon-Martí et al. 2022; Susskind and Susskind 2022; Toh and Lim 2022). In this regard, Kress has played a key role in pointing out that the “traditional” textual landscape has become multi-modal, digital, and sometimes interactive. This involves both material resources (sound, paper, film etc.) and semiotic modes (speech, gesture, image):

Many modes of representation and communication which were formerly treated as marginal or irrelevant – image, gaze, layout, typography – are now recognized as significant, and central even, in any site of learning, changing both what and how learners learn (Kress et al. 2021a: 72).

In other words, semiotic resources – modes and media – have become central for understanding how knowledge is, and may be, represented. Each choice of material and semiotic resources has its rationale and offers different affordances and constraints. Not everything can be expressed with any given resource. Changes of the media landscape include not only a shift from the textual to multimodal resources, but also the development of interactive resources (like digital games, simulations, Virtual Reality [VR], and Artificial Intelligence [AI]), which means, for instance, that we have to extend a concept such as information linking (van Leeuwen 2005) to the idea of activity linking, and, more broadly, that we have to extend our idea of text to include both the materiality and the semiotic modes of the learning resources (Selander 2017, 2021), as outlined in Table 1.

However, the concept of “materiality”, in all its possible aspects, seems to be less developed in Kress’s social semiotic approach than the concept of ‘mode’. Modes can be used to analyse images, notations and icons, but also animations, simulation programs and games with different levels of complexity. Kress and van Leeuwen’s (2021) framework for analysing visual design can be applied to different media, paintings, drawings, photographs, sculptures, films and more. But such media can also carry meaning, as do the differences between technological media such as the screen, VR, bodily sensors, the Internet of things et cetera.

6 Concluding remarks

In my view, Gunther Kress has made a major contribution to a new understanding of learning, related to a social semiotic perspective of communication, meaning making, and material and semiotic resources. In this article there is no space to do justice to his detailed analyses of communication and multimodal texts, and of the relation between different multimodal elements and the way these influence meaning making and understanding. Kress has also touched on the importance of contextual understanding and the role of material resources (see, for further details, Archer and Newfield 2014; Bezemer and Kress 2016; Danielsson and Selander 2021; Diamantopoulou and Ørevik 2021; Kress et al. 2021a; Selander et al. 2021).

However, while social semiotics focuses on communication as individual and social meaning-making, a more pedagogic and didactic oriented view would focus on new capabilities (skills and knowledges) in relation to both individual interests and to formal, semi-formal or non-formal contextual framings. Multimodal, social semiotic analysis of communication is a powerful tool for the understanding of learning, but if we are interested in how “Learning Design Sequences” are contextualized and carried out, we also involve other methodologies, where we first take a closer look at the conditions and frames, norms and regulations, traditions and potential resources, and so on. We then analyse the communication and the meaning-making activities and the outcome of these activities. Finally, we look more closely at the practices of evaluation and meta reflections (Selander 2022: 14).

The concept of context also includes how signs are embedded in social relations, organizing principles, genre expectations and signifying practices. It is not arbitrary what a combination of signs usually “stands for”. Signs are parts of structuring social practices. A representation will therefore not primarily be understood as a “true” or “false” mirror of a phenomenon, but rather as a frozen moment that produces an affordance by way of its (situated) representation (Selander et al. 2021: 33).

Several aspects of Kress’s work would, in my view, deserve further research, and would indeed be fundamental for a future, and deeper, understanding of learning from a multimodal point of view. I single out four aspects:

1. the need to develop contextual and institutional understandings of learning, in relation to specific historical settings, and to different framings in terms of ideology, religion, political, social and cultural conditions, or for example professional and epistemic paradigms. Even if “learning” in its elementary form may relate to individual interests and aspirations, it is always carried out in a context with all its specific resources, social bonds and beliefs (Björklund Boistrup and Selander 2022);

2. the importance of materiality, and of the way different material resources, tools and materials, actually become part of our “normal” interaction with the world – as does the axe for the carpenter, the stick for the blind man or the cell phone and computer for the knowledge worker. This would also include the blurred borders between the body and its surroundings, including the role of performative learning, bodily experience and learning, as well as emotional aspects of learning (Knappett 2005; Lim 2021; Wulf et al. 2021);

3. a focus on digital media, and how they may operate with distributed information, games and simulations, and make it possible to use new instruments like VR (Virtual Reality) as a learning resource or AI (Artificial Intelligence) as a resource for new ways of organizing multimodal text-production, toys and games (including a renewed discussion of learning and play as well as of regulative norms (Selander 2008b), and as a resource for the supporting individual learning paths;

4. a focus on designs for and designs in learning, and how these are framed in different institutional contexts (Björklund Boistrup and Selander 2022; Dorst 2015; Glawe and Selander 2021; Selander 2015; Selander and Kress 2021).

It is, of course, not possible to do justice to Gunther Kress’s entire work within the frame of a short, personal, and reflective article. But as a kind of conclusion, I would like to emphasise two aspects by means of two quotes that show the influence of Kress’s work on our didactic-, transformative-, interactive-, and design-oriented perspective:

In a design-oriented perspective, learners (or the gamers/visitors etc.) are seen as designers of their learning, which is conceptualized in terms of designs in learning […]. This means that the learner is constantly creating anew his or her own learning paths. A related question concerning learning is what one does when one learns, something which highlights a performative perspective on learning (Selander et al. 2021: 33).

Design of knowledge representations will accordingly be discussed in terms of what is represented for the learner […] How knowledge is represented – by way of its material and semiotic form – gives us clues both to its communicative (cultural, social) and its epistemological meaning […] A representation will therefore not primarily be understood as a ‘true’ or ‘false’ mirror of a phenomenon, but rather as a frozen moment that produces an affordance by way of its (situated) representation. (Selander et al. 2021: 33)