What modes can and cannot do: Affordance in Gunther Kress’s theory of sign making

1 Introduction

The notion of ‘affordance’ has been central to formative discussions of the now well established multimodal approach to the study of meaning making. According to Kress, one of the founders and major theorists of this approach, affordance “points to the potentials and limitations of specific modes for the purposes of making signs in representations” (Kress 2010: 157). Kress himself used the term very frequently, and increasingly so. In Multimodal Teaching and Learning (2001), the term occurs 26 times; in Literacy in the New Media Age (2003), 66 times; and in Multimodality (2010), 95 times.

Affordance has also been taken up widely by others in the field. Consider Visual Communication, a major outlet for multimodal research that has published around 400 articles (4 issues of 5 articles per year) between 2000 and 2020. In nearly half of them (192), the term appears at least once. The two textbooks on mutimodality that have come out so far (Bateman et al. 2017; Jewitt et al. 2016) both use the term (with some modifications/refinements), and mention the term 9 and 69 times, respectively. Jewitt et al. (2016: 72) describe affordance as “the idea that different modes offer different potentials for making meaning. Modal affordances affect the kinds of semiotic work a mode can be used for, the ease with which it can be done and how”. This focus on the affordances of modes reflects Kress’s interest, but he also frequently spoke of affordances of things other than modes, including natural objects and phenomena (body, electricity, light, movement), cultural technologies and artefacts (writing systems, tracings on surfaces, books, screens, smartphones) and the senses (seeing, hearing, touching) – in short, he considered the affordances of all resources that cultures have appropriated and developed to address their (social) needs and demands. Affordance is then used to suggest that something has been designed or used to create distinct possibilities for social action. This broader application is also emphasized by Bateman et al. (2017: 90), who write that “[w]hen seen in terms of affordances, an object is directly perceived in terms of the possibilities for action that it opens up for an agent in an environment”. This leads them to discuss the distinct affordances of materialities, canvasses, media, and platforms, and also of specific modes, such as layout, gesture and language.

Another common articulation of the idea of affordance is to say that mode m is good at p, q and r, while mode y is good at x, y and z: “language is good at some things, but other things may be better covered in other forms of expression” (Bateman et al. 2017: 44), and: “Kress rightly emphasizes that each mode can do some things well […] and others less so, or not at all – its “constraints” (p. 185). The visual mode can excellently present concrete details, for instance, but is bad at rendering abstract concepts – unless by means of strongly coded symbols” (Forceville 2011: 3624).

With affordance having become a major conceptual anchor both in Kress’s work and in the social semiotic literature more generally, I set out to address the following questions: What are the origins of Kress’s idea of affordance? How did he adapt it and embed it in his conceptual frame? How did he integrate it in his theory of sign making and communication? What did it do or add to his theory? How did the idea subsequently shape his research programme? How has and can the idea been critiqued? What has it offered Kress, and what, if anything, might it offer current and future multimodalists? In addressing these questions, I document a critical shift in the history of Kressian social semiotics and multimodality, complementing earlier work on the very significant impact of the term and the theory behind it and the extensive accounts of its reconceptualisations and misrepresentations in psychology, Human–Computer Interaction, and other sciences (see e.g., Chong and Proctor 2020; Costall and Morris 2015).

The paper is structured as follows. In Section 2, I explore how Kress articulated his theory of sign making in the early 1990s, just before the notion of affordance appears in his writing. In Section 3, I investigate how he subsequently took up affordance, and how his understanding of the term aligns with the work of James Gibson and Donald Norman. In Section 4, I explore how Kress embedded affordance in his theory of sign making from the early 2000s, and how this changed the theory in fundamental ways. In Section 5, I identify what research questions he pursued, prompted by the revised theory; and how he used affordance in accounts of meaning making. In Sections 6–8, I review and develop critiques of Kress’s use of affordance, and draw my own conclusion about the analytical value of this notion.

2 Kress’s theory of sign making

In Against arbitrairiness: The social production of the sign as a foundational issue in critical discourse analysis (1993), published in a special issue of Discourse and Society on critical discourse analysis, Kress outlines a theory of sign making, providing directions along which to look at the choices that people make in representation and communication. I use this paper as a starting point as it articulates a consolidation of Kress’s thinking just before he brought in affordance. As we shall see, in the following decades he would refine and elaborate on, but not fundamentally change the main premises presented in this paper. As always, he used concrete examples to build his argument. Among the examples he considers in this paper are a drawing consisting of seven circles, made by his then four-year-old son, who had said that “this is a car” (an example that he would return to in many publications to come). The other example is of a sentence spoken by the same child while walking up a hill with his father: “This is a heavy hill”. Kress then asks, Why did the child choose (so many) circles to represent a car, and why did he speak of a ‘heavy’ hill? This question leads him to consider these two communicative events from the following perspectives:

(1.1) The sign maker’s perception, experience, and understanding of (the characteristics of) that which they want to represent (the signified).

(1.2) The sign maker’s assessment of the signified in terms of the characteristics they deem to be “criterial”, i.e., defining features of the signified.

(1.3) The resources and principles for expression that have become available to the sign maker through learning. They include ‘pre-existing’ signs/signifiers, such as words, which can be ‘repurposed’. In the example of the child climbing a hill, the child re-uses the adjective ‘heavy’ to signify what he deems criterial of the action he is describing. In the case of the drawing, the child applies the principle of analogy: circles come to stand for wheels.

(1.4) The sign maker’s perceived “aptness” of fit^[1] between the (criterial features of the) signified and the available signifiers. In the case of the hill climbing, the child “selects an existing sign which in his view most nearly expresses aspects of the signified, namely ‘heavy’” (Kress 1993: 173). That is, an existing sign becomes the signifier for a newly produced signified (Kress 1993). In the case of the drawing,

[…] the child decides that wheels are criterial in representing the signified ‘car’. That decision then determines what may be an apt signifier, in this case circles. Wheels are selected as the criterial aspect of the object to be represented (the signified), and circles are apt signifiers in the sense that they adequately signfy, represent or express the characteristics of the signified. All signs are formed in this metaphoric process. All signs are metaphors. (Kress 1993: 174)

Kress presents the two examples to support his central thesis that “[a]ll signs are motivated in their relation of signifier to signified” (Kress 1993: 180): The child did not pick random signifiers (including words and visual forms), He picked signifiers he knew and deemed fitting. That this is antithetical to de Saussure, as the title of the paper suggests, is questionable. Claiming that the child selected the word ‘heavy’ as it most nearly expresses what he considers to be the defining features of what he is experiencing walking up hill does not contradict de Saussure’s point that entirely different strings of sounds could have equally come to stand for the meaning(s) typically signified by ‘heavy’ in the English-speaking community.^[2] The sounds that represent ‘heavy’ do not provide any clues that would enable sign makers who do not speak English to guess what it means, whereas the circles that represent wheels might, if the sign maker is familiar with the principle of analogy. Kress’s proposition could therefore be taken as an addition to, rather than a rejection of, de Saussure’s idea of arbitrariness.

In Against arbitrartiness, the sign maker and their making of signs becomes the object of study. In Kress’s own words, this view has connections with the works of Hjelmslev (1943), Barthes (1982) and Eco (1976), with “the significant and crucial difference” being his attempt to “locate this process firmly in the social and cultural histories of the producer of the sign” (Kress 1993: 173). Kress further captured his perspective of the individual sign maker with the concept of interest: “‘Interest’ leads the producer of the sign to focus on a particular characteristic of an object or event, that is, make it the basis of the production of a signified” (Kress 1993: 174). Interest stems from “the individual’s social and cultural histories”, and “the context of an interaction with other constitutive factors of the situation which are considered as relevant by the individual” (Kress 1993: 174). Thus it encapsulates both (a) an individual’s journey, their experiences, their learning (what we might call their ontogenesis); (b) social and cultural history (or philogenesis); and (c) the social needs and wants that arise in a concrete communicative situation. This ‘crossing’ of individual and social histories, and the here-and-now would continue to define his conceptual framework and would soon be supplemented by consideration of the material environment and adoption of the notion of affordance.

3 The arrival of affordance and mode

Four years after publication of Against Arbitrariness, Kress worked on a project called “The Rhetoric of the Science Classroom” (1997–1999) with Jon Ogborn, then professor of science education at the Institute of Education. It was Ogborn who introduced Kress to the notion of affordance: “Jon decisively ‘scaffolded’ one of my more stumbling semiotic attempts by throwing the word ‘affordance’ into the conversation” (Kress 2010: xiv). Ogborn recalls:

I do happen to remember the occasion that Gunther mentions, when I suggested during a discussion that he needed the idea of Affordance. It arose because we were trying to think about what different modes of communication made differently possible, or at least easier or harder to achieve. My inclination, from a scientific background was to think of the physicality of different modes, being always impressed by the sheer intransigence of material things. As I used to say to Gunther, “One can imagine whatever one wishes, but one cannot do whatever one wishes; the real world hits back.” (Jon Ogborn, pers comm. Oct 2021).

A book under the project title was to appear two years later in which the term is central to the problem statement. The team write that they considered “the affordances of each of the modes, that is, we asked the question ‘What constraints and possibilities for making meaning are offered by each mode […] and what use is made of them?’ […] ‘what potentials for meaning-making are available in each mode, and how are these potentials arranged (‘structured’) to make meaning?” (Kress et al. 2001: 13). They noted that “different modes take on specialised tasks, broadly along the lines of their inherent affordances” (p. 14) and that “each mode is culturally shaped around the constraints and affordances of its medium – its materiality.” (Kress et al. 2001: 15)

Two things stand out from these quotes. First, while in Against Arbitrariness (1993) there already is mentioning of ‘multi-modality’, the term ‘mode(s)’ is not yet used. By now, mode has (a) become a central term in describing the aims and focus of Kress’s research programme; and (b) is defined in terms of affordance, which in turn (c) is defined in terms of (i) materiality or inherent physical properties; and (ii) social and cultural conventions of using these properties for communication. Where ‘mode’ elaborates on the third premise of the scheme first presented in Against Arbitrariness (what I paraphrased above as resources and principles for expression available to the sign maker), ‘affordance’ elaborates on the fourth part (what I paraphrased as the sign maker’s assessment of the aptness of fit between the available signifiers and the signified).

Secondly, the research question, ‘What constraints and possibilities for making meaning are offered by each mode […] and what use is made of them?’ places modes in the foreground, and sign makers in the background. The researcher is asked to (objectively) describe what a mode can do, and to compare and contrast this with what sign makers actually did with them. This is an important difference with Kress’s original formulations of sign making in Against Arbitrariness, which started from the sign maker’s unique socio-cultural history, and the context and conditions of the sign making event (cf. “Why did the child choose (so many) circles to represent a car, and why did he speak of a ‘heavy’ hill?”). In the new formulation, it is the force of the (material and social) environment that ‘tames’ the individual that is highlighted. The two formulations are not contradictory, but they do point to a subtle variation in the starting point for an account of meaning making, a shift from ‘what did the sign maker do here’ to ‘what can modes generally do’.

That the term “affordance” was coined by the American psychologist Gibson (1979) is perhaps well known, but not routinely acknowledged in Kress’s work or in the social semiotic literature more widely. Kress was aware of the original work, having been introduced to it by Jon Ogborn, and having participated in discussions about it with other colleagues in the Institute of Education.^[3] Yet he does not cite it in Multimodal Teaching and Learning (2001) or Literacy in the New Media Age (2003), and references it only once in Multimodality (2010), where it appears somewhat out of place.^[4] Kress never claimed to follow Gibson’s notion or indeed any subsequent writers’ reformulations of affordance, but we ought to briefly acknowledge their work.

In his final book, The Ecological Approach to Visual Perception, Gibson writes:

The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment. (Gibson 1979: 115)

Jon Ogborn reminisces:

Gibson’s notion of affordance offered us a way to achieve a theoretical complementarity (to use Gibson’s word) between, on the one hand, the endless malleability of human communicative acts, always in flux, and on the other hand, the decisive material constraints put upon them by the physical nature of the means they use. And I think that this Gibsonian complementarity is in general how Gunther went on to use the idea. (Ogborn, pers comm, Oct 2021)

Gibson was concerned with the relation between humans (and animals) and their environment. He argued that humans perceive physical objects and surfaces in terms of the actions that they afford, that is, the opportunities that they provide for action, relative to the action capabilities of the perceiver that are derived from their physiological-biological make-up. An apple on a tree may be ‘grabbable’ for tall animals and humans but not for others. Donald Norman, whose work has helped popularise this take in the 1990s, reformulates it as follows:

[…] the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. […] Affordances provide strong clues to the operations of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing or bouncing. When affordances are taken advantage of, the user knows what to do just by looking: no picture, label, or instruction needed. (Norman 1988: 9)

The idea of seeing things in terms of their action possibilities resonates strongly with Kress’s uptake of the term, but there are important differences. In line with Gibson (and Norman), Kress considers affordances of objects and surfaces, and more generally the affordances of ‘stuff’ that humans can source from the environment.^[5] Also in line with Gibson (and Norman) is that he presumes that affordances exist independent of sign makers, and that the properties of ‘stuff’ and their potential for action can be objectively described (leading to what I will later describe as ‘Project 1’ and ‘Project 2’, respectively). Their potential may or may not be utilised by individual sign makers or groups of sign makers. The degree to which affordances are ‘picked up’ depends on the capabilities of the perceiver/sign maker, and, in Gibson’s account, also on the relative position of their bodies in the environment, which is in constant flux.

Yet, where Gibson and Norman were primarily interested in what kinds of physiological action possibilities sign makers/perceivers see in things (grabbing, passing through, etc.), Kress was ultimately interested in what kinds of semiotic possibilties sign makers see in things, i.e., what social actions are afforded by things, and how this leads to the development of modes (which are not tangible in the way that Gibson’s objects and surfaces are). Kress’s attention to the material environment was an addition to, not a replacement of, his basic premises on the social. There are also important differences in relation to Gibson’s radical claim, then and now still, that affordances can be directly perceived without recourse to some notion of a (socially shaped) process of constructing mental images from visual input, or to Norman’s idea of clues.

4 Kress’s revised theory of sign making

Ten years on, affordance has become the central pillar of Kress’s theory of sign making: “Foremost among the terms integral to this theory is that of affordance”, writes Kress in Multimodality (2010: 157). From his writings of this time we can infer that the concept serves to draw together the following premises:

(2.1) Modes have distinctly different affordances, i.e., potential or possibilities and limitations for meaning making: “[Affordance] points to the potentials and limitations of specific modes for the purposes of making signs in representations” (Kress 2010: 157). This reflects the absolute perspective discussed above, which presumes that it is possible to conceive of (and possibly even describe) ‘meaning potential’ of a mode and/or the materiality it draws on at some general level, independent of individual sign makers and instances of sign making.

(2.2) Affordances include both inherent material properties and social and cultural conventions: “Affordance rests, on the one hand, on the materiality of the stuff, which work in social environments has fashioned into a cultural and semiotic resource on the other hand” (Kress 2010: 157).^[6] More specifically,

(2.2a) Materiality refers to “for instance the material of sound in speech or music, or graphic matter and light in image, or of the motion of parts of the body in gesture” (Kress 2003: 45), and “[t]he materiality of modes […] holds specific potentials for representation, and at the same time brings certain limitations” (Kress 2003: 45).

(2.2b) Socially and culturally made affordances refer to the prior semiotic work that was done with materiality by a specific community in response to their needs and demands:

The social work performed ceaselessly by members of social groups with the affordances of the material, together produce semiotic resources. That is, modes are the product jointly of the potentials inherent in the material and of a culture’s selection from the bundle of aspects of these potentials and the shaping over time by (members of) a society of the features selected. (Kress 2010: 80–81)

(2.3) Affordances can orient sign makers, prompting them in certain directions, thus mitigating their choices for representation:

The modes and the kinds of uses made of mode in a society bring with them certain orientations, a certain ‘take’ on the world. By and large, that ‘take’ becomes ‘invisible’ (inaudible, intangible) – in the sense of explicit awareness – to those who do ‘take’ it. Cultural and social habituation to modes, genres and practices shapes how we represent. In time that habituation to representation can begin to shape our expectations about how we will encounter and engage with the world which we then represent. Within the broad range of modal choices available in a society, there is then the individual’s decision to make choices to use these modes rather than those in this environment for these reasons. (Kress 2010: 76)

(2.4) There is variation in the use of modes across ‘space’/different communities or social or cultural groups, and modes change over time. Different groups make use of different sets of material affordances and they can end up using them for different social purposes:

The uses of mode constantly reshape its affordances along the lines of the social requirements of those who make meanings; that ensures that mode is constantly changed in the direction of social practices and requirements. Modal change tracks social change. Whatever is not a social need does not get articulated nor elaborated in the entities of a mode. As a consequence, the potentials inherent in materiality are never fully used to become affordances of a mode in a particular culture; nor are all the affordances which are available used for similar purposes across different cultures. (Kress 2010: 81)

Thus, there are three forces that Kress aims to account for: the material, the social and the individual. Whereas in the 1993 paper it was the individual that was emphasised (and that was presented as what set it apart from other theories), now the material and social environment is foregrounded. Ultimately, though, Kress (2010: 184) holds that “the social is prior”. The material environment provides a structural basis but the social turns it into cultural, semiotic resources, which can push sign makers in directions without them being aware of it. Individuals can access, choose from, and transform resources, but they cannot escape them. Affordance has thus become a rhetorical device to recognize and straddle the individual, the material and the social forces that shape sign making, a way to recognise and bypass twentieth century debates about agency and relationality by theorists like Giddens (1979) and Latour (2005).

5 Researching affordances

In Multimodal Teaching and Learning (2001), affordance, along with mode, already defined the research questions. But what did it lead Kress to do next? How did it shape the research that Kress subsequently initiated and/or called for? Looking back on the two decades of work that followed, I suggest that he pursued two distinct projects.

6 Critiques of Kress’s notion of affordance

Subsequent multimodalists and external commentators have questioned the tenability or feasibility of these projects, notably the second one aimed at identifying possibilities for expression. Some have sought to narrow the term down, whereas others draw more radical conclusions.

Prior (2005) levelled a direct response to Kress’s paper on Gains and Losses (2005), in which Kress dates and reflects on shifts in the use of image and writing and its implications (‘what might be gained and what might be lost’), comparing a page from The Boy Electrician, a page from the 1992 Institute of Education (IOE) prospectus, and a page from the IOE website from 2004, and exploring several children’s drawings. He suggests that Kress’s notion of affordance is deterministic (also see Street 2004). Prior (2005: 26) sees Kress’s treatment of affordances as “highly determinative, mutually exclusive, and binary”. Indeed, in the 2005 paper Kress makes particularly strong claims about the differences in affordances of writing and image. In Prior’s (2005: 26) words, “Words in his [i.e. Kress’s] account are finite, sequential, vague, conventional, authored, narrative and/or causal, and open to critique. Images are infinite, spatial, specific, natural and transparent, viewed, and available only for design”. He cites from the paper to illustrate this: “With depiction and images the situation is different: that which I wish to depict, I can depict […] I can draw whatever I like whenever I like to draw it. Unlike words, depictions are full of meaning; they are always specific” (Kress 2005: 15). Prior notes that making such general claims about image goes against the Reading Images project (Kress and Theo 1996), which led to a detailed analytical apparatus to differentiate between different types of images, and to recognise both ‘conceptual’ and ‘narrative’ structures in image.

Oliver (2005) comments on Gibson’s original notion of affordance and on the way in which it has been taken up in studies on technology, design, and human computer interaction.^[7] He argues that “…all that could be said then is that a thing afforded something to someone in a specific circumstance, not that it affords it in general. This path of diminishing returns trivialises the concept analytically.” (Oliver 2005: 403). He also critiques the idea of affordance as ‘action possibilities’ (a phrase that he ascribes to Norman (1988/2013):

‘Possible’ leaves nothing substantial to work with, unless a definitive list of possibilities can be constructed from the properties of each element separately, and their various interactions. This is only feasible if people and things can be exhaustively described, which presupposes both an essentialist, positivist epistemology (which is consistent with Gibson’s position) and a finite number of characteristics (which is not: see p. 243). Lacking this, ‘possible’ is a work of imagination and ‘affordance’ becomes speculative rather than analytic. (Oliver 2005: 403)

Is there a way around this? We could take a situated approach, as Oliver suggests, and consider the range of factors that affect sign making in an observed event. Take the mode of gesture. Pointing (with a hand gesture, or with a pointing device, such as a pen) can be a very effective means of drawing attention to an object in the environment, but it is only effective if it is within sight of the addressee, which depends on the relative position of and distance between the gesture maker and the addressee; the visual abilities of the addressee and their willingness to engage with the gesture; the lightness in their surround, and so on. The question is then not what possibilties gesture has, but what possibilities they have for certain sign makers under these specific circumstances (e.g., the possibility to draw attention to a specific spot in the environment; e.g., where to drill a hole in the wall).

Bateman et al. (2017) follow this line, emphasizing that it is only possible to claim that a thing afforded something to someone in a specific circumstance, and that it is not possible to make general claims about what a mode can or cannot do. They make a clear distinction between the project of describing properties of “[t]he particular canvases in which the material distinctions carrying communication may be inscribed” and the possibilities for action that they open up to individual sign makers. The properties do not determine the possibilities: “Note that this is not to say that particular materials can only carry particular kinds of meanings” (p. 250). They share Kress’s ambition to describe basic material properties (‘Project 1’), offering a ‘simplest systematics of communicative media according to the affordances of their involved canvases’, and suggest descriptive dimensions such as static–dynamic, 2D–3D, permanent–fleeting (see also Van Leeuwen 2005). They differentiate between canvases that afford unfolding in time, planning and design, layout, movement, actions and reactions. The authors also support a version of Kress’s idea of ‘aptness of fit’:

[…] while we cannot derive from a particular canvas any statements about what can be represented, we can say much about how it might be represented. A particular canvas will only make certain material distinctions available and not others, and this is what is available for meaning-making. (Bateman et al. 2017: 251)

[…] it needs to be noted that it is very unlikely that a semiotic mode will ‘go against’ the affordances of its material. That is, to take the example of the drawing of the cell again, placing the nucleus spatially within the drawn boundaries of the cell is hardly going to mean, in any semiotic mode, that the nucleus [sic, presumably this should read ‘cell’] is to be understood as not containing the nucleus! (Bateman et al. 2017: 294)

Focussing on actual situations still requires that we move beyond the analysis of artefacts and use the full spectrum of social research methods to understand the conditions of sign making. Commenting on Gains and Losses (2005), Prior (2005: 28) already noted “the focus on semiotic artifacts” and “an almost total neglect of semiotic practices”.

7 The affordance of affordance

The critique discussed so far renders Kress’s second project, to map possibilities and limitations of modes, untenable, unless they are considered in the context of a concrete observed communicative event.

Yet, a more narrow, time- and space-bound, situated, provisional take on affordance still leaves us with other challenges, including Oliver’s (2005) point about its analytical merit. Consider here Kress’s reflections on a science textbook from 2002. Looking at what writing and image are used for in this particular textbook, he concludes that “Image shows those curricular materials which are best – most aptly – represented in image” (Kress 2010: 47). This is not a generalisation about what image can do and it does not suggest that the choice of image has determined the representation. Yet we have no way of disproving the claim about the aptness, or ‘best fit’ between the (properties or possibilities of the) mode of image and the curriculum contents that need to be represented. The principle of aptness appears to simply always apply. If all choices that sign makers make can always (only) be explained in terms of the affordances of modes (‘a person p uses a mode m here to do y as m generally affords (p to do) y’), then what new insights does it generate? For the concept to have analytical value, we ought to also be able to identify examples that cannot be explained in terms of affordances and their aptness for particular (presumed or investigated) representational purposes.

When aptness is always presumed, affordance risks closing down a line of inquiry that Kress had previously followed. In Against Arbitrariness (1993), Kress’s driving question was not yet, What are the affordances of modes that can explain the choices of sign makers? It was, How can this or that signifier come to stand for this or that signified in this situation? What is it about the signifier that makes it apt to the sign maker for (‘afford’ representation of) the signified? Thus he actively explored how wheels may have come to represent a car, and ‘heavy’ a feature of a hill, in the lifeworld of his then 4-year-old son. There, the answer was not that the child used circles as they are apt signifiers of a criterial feature of wheels, he went to explain why they are apt, suggesting that the child applied the principle of analogy: ‘wheels are like circles’. We could add that this is about the analogy between the shape of the form that is being represented (the signified) and the shape of the form that is representing it (signifier). Incidentally, the example also demonstrates the value of (auto-)ethnography in addressing this question, as the sign maker’s choices could not have been explained in terms of (the affordances of) modes alone. As Kress rightly suggested in Against Arbitrariness (1993), we also need to consider the sign maker’s lifeworld and the situation, including setting, co-presence of other social actors, and activity being undertaken.

8 A personal conclusion

I draw the following conclusions from my reflections on Kress’s theory of sign making and the role of affordance within it.

As a social semiotician, I aim to understand how a sign is made in concrete artefacts or events, therefore I need to consider what the forms that are used mean according to local conventions, and how the sign maker draws on, transforms or transgresses these conventions. When new (unconventional) forms are used, I also need to consider the qualities of the material from which the forms are fashioned. In all instances I need to seek to investigate, and give consideration to, the unique life experiences of the sign maker and their audience and the conditions of the sign making. Without one or the other, I cannot reliably explain the semiotic choices that the sign maker has made.

To do so I need a language to describe features of modes and principles for building sign complexes or multimodal formations, including grammars and typologies of the material differences between modes (which I referred to as Kress’s Project I). I cannot rely on or help compile an exhaustive inventory of what it is possible to do or not in this or that mode (Project II), but I can, with each analysis of a sign making event, aggregate our understanding of conventions for making meaning with certain sets of material resources, i.e., of the (‘preferred’, ‘common’) use of modes in specific periods, spaces, and social networks. Adopting William Labov’s famous words slightly, the slogan for this larger project of surveying semiotic landscapes could be, Who makes what sign for whom, when, where, why and how? More specifically on the how-question, How does this form produced by this person on this occasion mean? What is it about the form that makes it suitable, in the eyes of the sign maker, for what they want to express? This needs to be explained through a combination of factors, including materiality, convention, (imagined) audience, and the conditions of sign making.

I believe that this can be achieved – producing plausible, situated accounts of sign making, and documenting norms and conventions for specific communities of sign makers – using the scheme that Kress laid out in his writings from as early as 1993, and without recourse to explanations that refer to affordances as possibilities and limitations of modes or semiotic resources. What affordance has brought to social semiotics is further recognition of the material environment, and its intricate relation to the social. It has drawn attention to the ways in which cultures and individual sign makers turn material into cultural resources for meaning making, and to the specific qualities of material, which can help explain how modes have developed and how someone has made a sign in a given situation. We can pursue these ambitions even if we remove affordance out of the social semiotic conceptual framework. Rather than closing anything down, this would open up new avenues for analyses and conceptual advancements.