Transmuting the painterly sign

1 Introduction

In 1959, renowned linguist and semiotician Roman Jakobson published an intriguing essay entitled “On linguistic aspects of translation” in which he categorizes three general kinds of translation:

1. Intralingual translation or rewording is an interpretation of verbal signs by means of other signs of the same language.

2. Interlingual translation or translation proper is an interpretation of verbal signs by means of some other language.

3. Intersemiotic translation or transmutation is an interpretation of verbal signs by means of signs of nonverbal sign systems. (Jakobson 1971: 261)

By “intralingual translation” Jakobson mostly means explaining one word or idiomatic phrase by dint of another which is synonymous in meaning. The example he gives is “celibate,” which can be substituted by “bachelor,” although he cautions that the semantic fields of these two words are not exactly the same. Later scholars have expanded this notion of intralingual translation by moving beyond the level of diction, where large chunks of a linguistic text need to be made accessible with more or less the same number of words which retain the meaning of the original text but are easier to understand. The technical term for such intralingual translation is “paraphrase,” which, in the words of Kelly Mays, “resembles translation” (2013: 1846). A good example of this comes from Professor Mays’ Norton introduction to literature (11th edition), where the first sentence of Jane Austen’s Pride and Prejudice (“It is a truth universally acknowledged that a single man in possession of a large fortune must be in want of a wife”) is paraphrased into “Everyone agrees that a propertied bachelor needs (or wants) to find a woman to marry” (2013: 1846). Here and elsewhere, the two sentences involved are similar in meaning but present different degrees of difficulty, hence the need for intralingual translation.

By “interlingual translation” Jakobson means translation in its conventional sense. As is obvious from its name, the second category of translation involves a transfer of meaning from one natural language to another at the level of word, sentence, paragraph, or text. An important point to keep in mind here is that as long as meanings or messages remain the same across the source language and target language, there is no need to adhere to what Jakobson calls “full equivalence between code-units” (1971: 261). For an illustration of his point, we may turn to the first paragraph of Raymond Williams’ Modern tragedy, which was translated into Chinese in 2007:

We come to tragedy by many roads. It is an immediate experience, a body of literature, a conflict of theory, an academic problem. This book is written from the point where the roads cross, in a particular life. (Williams 1996: 13)

我们通过多条路径接触悲剧。它是一种直接经验, 一组文学作品, 一次理论冲突, 一个学术问题。这些路径在一个特殊的人生中相互交叉, 构成本书的写作视角。(Williams 2007: 3)

Grammatically speaking, the last sentence of the English text above is written in the passive voice, with “the book” serving as its subject, but in its Chinese version, the subject of the last sentence becomes “the point where roads cross,” which constitutes the perspective of the book. The change of grammatical voice in this translation is a better fit to the Chinese way of speaking but does not alter the core meaning of the source text. Nevertheless, the two texts involved correspond more or less to each other in length and overall structure, which is not the case with Jakobson’s third category of translation.

By “intersemiotic translation,” Jakobson means interpretation of verbal signs by means of nonverbal signs. Unlike the other two categories of translation, there is little structural isomorphy that exists between texts of different semiotic systems, which explains why Jakobson also uses “transmutation” for this category of translation. As the main focus of his 1959 article is “translation proper,” Jakobson offers little elaboration on intersemiotic translation except for a brief definition. In another context (1987: 63), however, he does return to the same topic and cites the adaptation of Wuthering Heights into film, the conversion of medieval legends into frescos, the transformation of Mallarmé’s “L’Après-midi d’un faune” into music and ballet, and the transposing of Iliad and Odyssey into comics as examples of intersemiotic translation. The impression one gets from the discussion therein is that Jakobson’s intersemiotic transmutation seems to be merely a one-way operation that interprets verbal signs by means of their nonverbal counterparts, but the fact of the matter is that the relationship between verbal and nonverbal sign systems is not only reciprocal but also dominated by linguistic conversions of nonlinguistic texts, a fact that is hardly surprising given the dominance of verbal language over other types of sign systems in human life.

2 Painting and semiotics

The present paper, therefore, is an attempt to redress the inadequacy in Jakobson’s translation theory by illustrating how visual signs in general and painting in particular are converted into critical comments in the form of verbal language. As was mentioned earlier, unlike intralingual and interlingual translation, where the source text and target text more or less correspond to each other in overall structure, the results of intersemiotic translation vary a great deal, especially in terms of textual size. A painting, for example, may be given just one word or phrase as its central meaning or caption, but it may also generate a piece of art criticism that takes the form of one paragraph, one book chapter, or even one book. This lack of structural isomorphy, however, should not put us in despair, because, as we shall see in the rest of this paper with a bit of help from Peircean semiotics, no matter how long or short a critical comment is, it always involves a two-step interpretation which is determined by the unique nature of the painterly sign.

As a special form of human communication, painting or pictorial art has a very long history. As early as in the Paleolithic Age, our ancestors started drawing pictures of various animals on the walls of caves they dwelt in or visited. The same cannot be said about semiotics as an independent discipline of humanities, which was not firmly established until the turn of the twentieth century. What this means is that trying to understand painting with the assistance of semiotics is only a relatively recent intellectual phenomenon. Many art critics have turned to semiotics for inspiration because it is their belief that the art of painting consists of signs or at least includes elements that could be considered as signs (Morris 1964: 66).

Those who do not study painting in relation to semiotics are, of course, much greater in number, and the best known among them in our time is arguably Ernst Gombrich, who has exerted a huge influence on contemporary theories of visual art. According to him, a painter is no different from a scientist in the sense that they both attempt to arrive at the truth about their subject through “trial and error.” The following comment is taken from his book Art and illusion: a study in the psychology of pictorial representation:

This description of the way science works is eminently applicable to the story of visual discoveries in art. Our formula of schema and correction, in fact, illustrates this very procedure. You must have a starting point, a standard of comparison, in order to begin that process of making and matching and remaking which finally becomes embodied in the finished image. The artist cannot start from scratch but he can criticize his forerunners. (2000: 321)

On a quick reading, Gombrich appears to be proposing a constructivist theory of painting which views “the innocent eye” as mere illusion because the painter, instead of operating from a state of tabula rasa, is always under the constraint of a particular cognitive schema. However, the word “correction” that immediately follows “schema” reveals a remnant of the traditional representation theory, which argues that, even though the painter is constrained by a pre-existing cognitive schema, she can still modify it in relation to an object that confronts her. Looked at from this perspective, the value of painting depends very much on how closely it “matches” the object under observation. To put it in Gombrich’s own words: “What matters to us is that the correct portrait, like the useful map, is an end product on a long road through schema and correction […]. [S]uch a model can be constructed to any required degree of accuracy” (2000: 90).

The emphasis on “accuracy” of depiction, be it only in accordance with a certain cognitive schema, has dominated much of contemporary art criticism in the West, so much so that, in the eyes of some of its practitioners, Gombrich’s influence has nearly stultified the entire intellectual enterprise. Norman Bryson, for one, is quite blunt in his criticism of this stagnancy: “To the question, what is a painting? Gombrich gives the answer, that it is the record of a perception. I am certain that this answer is fundamentally wrong […]. [W]hat is suppressed by the account of painting as the record of a perception is the social character of the image, and its reality as sign” (1983: xii). As can be seen from the key term he uses, Bryson wants us to study paintings not as (accurate or inaccurate) representations of external objects or situations but as vehicles that carry particular social meanings, and hence the relevance of semiotics.

What complicates the situation is that not all semioticians agree on what a sign is and how it is be interpreted, thus rendering some of them more pertinent than others to the understanding of painting. Generally speaking, semiotics can be divided into two grand traditions, one characterized by structural analysis in the manner Ferdinand de Saussure and the other by logical inference in the manner of Charles Sanders Peirce. According to the former, the meaning of a sign depends largely on the system in which it is a member, and “it is from the interdependent whole that one must start and through analysis obtain its elements” (Saussure 2011: 113). This priority on the structural relationship between signs has led to a negligence of the sign itself, which connects us to the social world. Norman Bryson has the following to say:

If it is to Gombrich that we owe the theory of painting as a mode of cognition, our ideas of what signs are and how they operate are the legacy of the founder of the ‘discipline of signs’, Saussure. This, too, is a problematic inheritance. […] The element lacking in Saussure’s conception of the systematic nature of signs, I maintain, is description of how signs interact with the world outside their internal system. Painting is an art of the sign, but the particular signs it uses, and above all its representations of the body, mean that it is an art in constant touch with signifying forces outside painting, forces that cannot be accounted for by ‘structuralist’ explanations. (1983: xii–xiii)

There are two major points that have been made in this quote. The first point is obvious, that is, we need to investigate how signs interact with the world outside their internal system. The second point is that painting involves “particular signs” that are different from verbal language. Although Saussure was aware of nonverbal signs such as “the alphabet of deaf-mutes, symbolic rites, polite formulas, military signals” (2011: 16), as a linguist, his focus of investigation was always the synchronic system of verbal signs.

In contrast to the “problematic” Saussurean legacy, the theory of logical abduction and semiotic trichotomy proposed by Peirce seems to shed greater light on our understanding of painting as an act of social communication. This is so because the Peircean tradition not only concerns itself with linguistic signs, where a representamen (signifier) points to its interpretant (signified) through habit, but also pays attention to other types of signs that involve either indexical or iconic reasoning. The following is Peirce’s own explanation of his trichotomous division of signs:

It has been found that there are three kinds of signs which are all indispensable in all reasoning; the first is the diagrammatic sign or icon, which exhibits a similarity or analogy to the subject of discourse; the second is the index, which like a pronoun demonstrative or relative, forces the attention to the particular object intended without describing it; the third [i.e. symbol] is the general name or description which signifies its object by means of an association of ideas or habitual connection between the name and the character signified. (1991: 181)

The first sentence in this quote is a bit confusing. While it is true that we need all three types of signs in our thinking about the world, not all of them involve reasoning in terms of the sign interpretation. To illustrate the differences among them, let us go back to Peirce’s triadic model of the sign from which his semiotic trichotomy is derived (Figure 1).

Figure 1:

Peirce’s semiotic triad.

Index, for one, invokes an act of inference from A to B on the basis of spatial or temporal contiguity between A and C; icon, on the other hand, requires the interpreter to move A to B on the basis of similarity between A and C. When it comes to “symbol,” however, the movement from A to B is automatic as a result of habit or convention. In other words, neither indexical nor iconic reasoning is involved in the interpretation of the linguistic sign.

3 Sign emergence and transformation

What needs to be pointed out is that the Peircean concept of “icon” has nothing to do with the “iconography” and “iconology” proposed by another influential twentieth-century art critic Erwin Panofsky. In his 1939 book Studies in iconology: humanistic themes in the art of the Renaissance, Panofsky characterizes our investigation of painting as consisting of three stages. First, we need to identify pure forms of a picture, which is technically called “pre-iconographical description”; then we try to ascertain artistic motifs or themes expressed through pictorial images and narratives, for which Panofsky’s formal term is “iconography”; finally, on the level of “iconology,” our aim is to “reveal the basic attitude of a nation, a period, a class, a religious or philosophical persuasion – qualified by one personality and condensed into one work” (1972: 7). Peirce, on the other hand, uses the concept of “icon” strictly in the sense of similarity, an important cognitive feature that has to be present if one wishes to convey a pre-existing indexical sign to another person. Take for example the artifice of a candle (initially made of animal fat) that the ancient Chinese burned at night to produce light. For those who knew how a candle worked, setting this object alight could serve as an indexical sign of the light which ensued. However, if a person wished to convey this indexical relationship to a fellow being, she would have to resort to drawing a picture which resembled a burning candle, such as in Figure 2.

Figure 2:

Chinese character for “white.”

In terms of sign evolution or transformation, this act can be described as the “iconization of index” which gave birth to human language. For a long period of time, when a Chinese person wanted to convey the meaning of “light” or by extension “bright and white color,” she could draw the iconic sign , which looked similar to a burning candle. Language evolution, of course, did not stop at iconization of index. After repeated use of the iconic sign , for example, the connection between its representamen (signifier) and its interpretant (signified) became automatic in the mind of the user, producing a Peircean “symbol,” such that communicators no longer relied on the similarity between the representamen and the object it referred to. We can call this stage of sign evolution “symbolization of icon.”

4 Transmuting the painterly sign via iconic and indexical reasoning

Painting came into existence through approximately the same principle of iconization of index (pictorial imitation of a meaningful object or life situation), but there is a huge difference between the two. As is clear from the illustration in Figure 3, ancient Chinese characters start their semiotic life as pictures (on the right) which gradually evolve into various pictographs (on the left), and because of their repeated use, these graphs eventually become symbols in the Peircean sense, that is, the interpretation of their meaning no longer requires iconic reasoning.

Figure 3:

Symbolization of icon (image source: Jiang 2008: 16).

The production of a painting, however, is a one-off event, which prevents the iconic sign from being “symbolized” through its repeated use. For that reason, we can say that a “painterly sign” is an “unsymbolized” or “unsymbolizable” icon.

Unlike a pictograph which, after being symbolized, could develop into a “meta-icon” (an icon of an icon) to be used in the formation of new characters as a meaningful sub-unit of language (Ding 2010: 75), a painted object or situation has to be interpreted as a whole with no independent sub-units of meaning. Nelson Goodman thus explains: “Nonlinguistic systems differ from languages, depiction from description, the representational from the verbal, painting from poems, primarily through lack of differentiation – indeed through density (and consequent total absence of articulation) – in the symbol scheme” (1968: 226). To illustrate Goodman’s concepts of “density” and “absence of articulation,” we may turn to Figure 4, depicting a very convincing example used by Goran Sonesson in one of his recent articles (2016: 52).

Figure 4:

Sonesson’s (2016) illustration of semiotic density based on the artwork Le Viol by René Magritte (1934).

In the extreme left column, the painted object looks like a button; in the second left column, it looks like part of a door handle; in the middle column, it becomes a female nipple; in the second column on the right, it can be interpreted as either a female nipple or an eyeball; in the column on the extreme right, the painted object can be interpreted more definitively as an eyeball in the actual painting done by the twentieth-century Belgian artist René Magritte.

Another example (Figure 5) that well illustrates the uniqueness of the painterly sign comes from Laurie Schneider Adams’ A history of Western art (2010).

Figure 5:

Alexander Calder’s Cat (Adams 2010: 15).

In Figure 5, three alphabetic letters of English have been incorporated by Alexander Calder into a painting where “C” serves as the head of a cat, the upper part of the “A” serves as its hunched back, the lower part of the same letter serve as its feet, and the middle line of “A,” tilting upwards, serves as its belly and tail. The letter “t” seems less well absorbed into the body of cat than “C” and “A,” but its presence on the righthand side is important because the three letters together form the English word “cat,” thus removing any possible doubt about the species identity of the painted animal. Roland Barthes thus comments on the importance of linguistic signs used alongside their nonlinguistic counterparts:

All images are polysemous; they imply, underlying their signifiers, a ‘floating chain’ of signifieds, the reader able to choose some and ignore others. […] Hence in every society various techniques are developed intended to fix the floating chain of signifieds in such a way as to counter the terror of uncertain signs; the linguistic message is one of these techniques. At the level of the literal message, the text replies – in a more or less direct, more or less partial manner – to the question: what is it? […] The denominative function corresponds exactly to an anchorage of all the possible (denoted) meanings of the object by recourse to a nomenclature. (1977: 38–39)

Barthes’ metaphor of “anchorage” here is very appropriate, because without the word “cat,” our interpretation of the picture could float into any number of small mammals that look similar. Nevertheless, the way Calder incorporates this linguistic sign is quite ingenious; in most other places, linguistic expressions simply appear in the form of a “caption” or “title.”

What we also learn from Calder’s semi-abstract painting is that the higher the iconicity of a pictorial work, the lower its dependence on the use of linguistic anchorage. For an illustration of this point, we once again turn to Laurie Adams’ A history of Western art, which on page 22 provides the set of pictures depicted in Figure 6.

Figure 6:

Theo van Doesburg’s illustration of decreasing iconicity (Adams 2010: 22).

If asked, most people would probably say that the degree of abstraction of these five pictures of a cow increases from left to right, with the one on the extreme right being the farthest from everyday life, and hence the need for linguistic anchorage.

The subject of painting does not have to be an object like a cow all the time; more often than not, it is a life situation (real or imaginary) that conveys a moral or philosophical lesson. Take, for example, the well-known Greek myth of Leda and the Swan, in which Zeus (under the guise of a swan) rapes Leda, who later gives birth to Helen and Clytemnestra. These two beautiful women in turn are responsible for the death of Greek general Agamemnon and the destruction of Troy respectively, forever changing the course of Greek history. To capture this gloomy sense of human destiny being pre-ordained, Nobel Prize winner William Butler Yeats wrote the following poem (Mays 2013: 1038):

Leda and the Swan

A sudden blow: the great wings beating still

Above the staggering girl, her thighs caressed

By the dark webs, her nape caught in his bill,

He holds her helpless breast upon his breast.

How can those terrified vague fingers push

The feathered glory from her loosening thighs?

And how can body, laid in that white rush,

But feel the strange heart beating where it lies?

A shudder in the loins engenders there

The broken wall, the burning roof and tower

And Agamemnon dead.

     Being so caught up,

So mastered by the brute blood of the air,

Did she put on his knowledge with his power

  Before the indifferent beak could let her drop?

As important contributors to either state ideology or mass media, painters certainly would not want to miss such a sexually interesting and philosophically profound subject matter for their works. In the long history of Western art, there have been numerous artworks that are titled “Leda and the Swan,” and Figure 7 shows just five of them that can be easily downloaded from the internet.

Figure 7:

Five paintings on “Leda and the Swan.”

What is obvious from this list is that there is a gradual decrease in iconicity from left to right, ending with the most abstract piece drawn by Otto Dix. Despite the hint that can be derived from the title, it is still extremely difficult for viewers to ascertain a picture of the famous rape scene, which once again shows the importance of iconicity to our understanding of painting.

More importantly, even if a reader has no difficulty linking a picture with its corresponding object or life situation, she has only completed the first step of interpreting the painterly sign. As was mentioned earlier, the aim of an artist is not merely to depict something accurately but to communicate a point that she feels can be indexically or iconically (metaphorically) related to it. Many classical Chinese painters, for example, were fond of drawing pictures of buffalo, and they did that mostly because this animal played an important role in their agricultural life. In that sense, a buffalo is a noteworthy part of the whole way of life, with the animal being an indexical sign of the latter. With particular physical and behavioral features of its own, buffalo can also be iconically or metaphorically related to certain human virtues (e.g. humility, patience, strength, and endurance) that are valued by society at large.

The same applies to the painting in Figure 8, discussed by Gombrich himself in The story of art (1951: 216–217).

Figure 8:

Leonardo da Vinci’s Last Supper (source: https://www.britannica.com/topic/Last-Supper-fresco-by-Leonardo-da-Vinci#Overview).

Like most later critics who are influenced by him, Gombrich is mostly concerned about how Leonardo da Vinci makes ingenious use of the perspectival technique, the symmetric arrangement of two groups of three on each side of the central figure, and the bright color of red that draws attention to Jesus Christ. What impresses him the most is that the painter “did not find it necessary to sacrifice correctness of drawing, or accuracy of observation,” which resulted in a vivid rendering of the different gesticulations and movements made by the twelve disciples in reaction to the announcement. As to the “the basic attitude of a nation, a period, a class, a religious or philosophical persuasion” that lies behind the painting, Gombrich remains reticent, which led to the following remark by Panofsky:

As long as we limit ourselves to stating that Leonardo da Vinci’s famous fresco shows a group of thirteen men around a dinner table, and that this group of men represents the Last Supper, we deal with the work of art as such, and we interpret its compositional and iconographical features as its own properties or qualifications. But when we try to understand it as a document of Leonardo’s personality, or of the civilization of the Italian High Renaissance, or of a peculiar religious attitude, we deal with the work of art as a symptom of something else which expresses itself in a countless variety of other symptoms, and we interpret its compositional and iconographical features as more particularized evidence of this “something else.” (1972: 8)

Panofsky received no systematic training in modern semiotics, but his expression – “a symptom of something else” – is not much different from the Peircean concept of index. For him, in addition to a vivid representation of the variant postures of thirteen diners, The Last Supper by Leonardo da Vinci also points to the pious attitude of the artist, as well as the general religious milieu of the Renaissance period.

5 Conclusions

All the examples we have examined seem to lead to the conclusion that the painterly sign is a semiotic complex that consists of two levels of signification. The first level is that of an iconic sign, where the viewer is invited to identify an external object or life situation through the similarity of the sign to the representamen on canvas. The second level is that of an indexical sign, where the viewer is led from the painted subject to a particular aspect of human life of which it is a part. Interestingly, the second level of a painterly sign can also involve another iconic sign, for which the viewer makes a metaphorical connection between a painted object/situation and a particular human value. A similar conclusion to this one was reached by Roland Barthes (1991), albeit in his own poststructuralist terminology, when he commented on the front cover of a magazine that was in circulation in the middle of the twentieth century (Figure 9).

Figure 9:

Cover of Paris Match 326 (25 June–2 July 1955), with commentary by Barthes (1991: 115).

To convert Barthes’ statement into Peircean terms, we can say that readers of issue 326 of Paris Match see on its front cover an iconic sign which enables them to identify a young black soldier saluting to the French flag, but this is only the first step in their understanding of the picture. The second step in understanding this painterly sign involves an indexical sign that directs its readers to a higher level of meaning on the basis of a part–whole relationship, that is, readers are invited to make a connection between the painted subject (a soldier in a French colony) and his loyalty to the French empire, of which he is a proud member. In comparison to the iconic sign on the first level, the indexical sign (sometimes an iconic sign) on the second level is much more difficult for readers to interpret because, as Daniel Chandler puts it, “anything can be a sign as long as someone interprets it as ‘signifying’ something – referring to or standing for something other than itself” (2002: 13). This, perhaps, constitutes the greatest challenge that painting poses to art critics, who have to transmute it into verbal comments of various length.