Morris’ Lost Pragmatics

1 Preliminary remarks

“The modern usage of the term pragmatics,” Levinson (1983: 1; bold original) observes, “is attributable to the philosopher Charles Morris (1938).” To refresh our memory, Morris’ original text is quoted below.

By 'pragmatics' is designated the science of the relation of signs to their interpreters. […] signs have as their interpreters living organisms, it is a sufficiently accurate characterization of pragmatics to say that it deals with the biotic aspects of semiosis, that is, with all the psychological, biological, and sociological phenomena which occur in the functioning of signs. (Morris 1938: 30; italics added)

The italicized texts are worth stressing here, since they are the parts that are skewed in linguistics pragmatics. First Morris’ “living organisms” embraces both humans and animals, even plants included.^[2] This cross-species scope of subjects is subsequently constricted to the human species only, and furthermore, it is the hearing members of the species that play a role in the conceptualization. Manual signers, Braille users — these visual and tactile users — are marginalized at the mercy of applicability of the theory conceptualized on audial-oral modality only.

Second, Morris’ “all the psychological, biological, and sociological” is, however, found too wide by some philosophers and linguists, as “especially within analytical philosophy, the term pragmatics was subject to a successive narrowing of scope” (Levinson 1983: 2; italics original). Mainstream pragmatics, driven by analytic philosophy, is monomodal in the sense that both its conceptualization and practice is based on (1) viewing language as an object instead of viewing it as living experience, and (2) reduction of the language object to written form for visual access. Recall that even recorded live speech is transcribed into static symbols for eyes to look at. In terms of multimodality, the original oral-auditory modality is transformed into visual modality. “It is instructive,” Locke (2011: 6) observes, “to consider the things that readers do not encounter on the printed page. They see no prosody, no voice quality, no tone of voice, no rate of speaking, no loudness, no vocal pitch, and no formant structure.”

The objective of this paper is twofold. First, it reviews, briefly though, Morris’ behavioral semiotics. The review will show that the pragmatics envisaged by Morris is essentially what may be termed multimodal semiotic pragmatics (hence the title of this paper). Surely in Morris’ time, multimodal semiotic pragmatics was only in its embryotic form, yet to grow and develop. Very encouragingly over the last few decades, researches in biosemiotics (section 2), multimodality, pre-speech children’s development in Piaget’s theory of sensorimotor intelligence (sections 3 and 5), total saturated experience in situated discourse studies (section 4), pragmatic impairments in clinical linguistics (section 6) – all these, admittedly being diversified in academic backgrounds, converge onto the direction of multimodal semiotic pragmatics. Hence the second objective of the paper: to make a plea for advancing the founding father’s original thinking of pragmatics.

Before we go to detail, a word about the key term multimodality is in order here. In view of word formation, it breaks into multi + modality, not multi + mode.^[3] Modality is used in the sense as defined in neuroscience:

We receive information about the world through tactile sensations (body senses such as touch and pain), auditory sensations (hearing), visual sensations (sight), and chemical sensations (taste and olfaction). Each sensory modality has one or more separate functions. (Kolb and Whishaw 2005: 135; italics added)

Multimodality being the latest coinage, the terms Morris uses for it are “distance senses” (sight, hearing, and smell), and “contact senses” (touch, taste) (see Morris 1951 [1938]: 32). Since both classes of senses play a vital role in living organisms’ sign-making behaviors, Morris’ theorization of sign behavior naturally incorporates them as its intrinsic component. This multimodal component is distilled when linguistics pragmatics, formulated in the spirit of analytic philosophy of language, moves away from Morris’ broader sign behavior to focus on verbal behavior only.

2 Morris’ behavioral semiotics: Seeds for multimodal semiotic pragmatics

2.1 Morris’ envisionment of pragmatics: Sign behavior (or semiosis)

It is instructive to review the pragmatics envisaged by Morris in his series of works (1993 [1925], 1951 [1938], 1946, 1962, 1964, 1970). Sign behavior is the kernel of his theory building. It is rooted in Mead’s theory of social behaviorism, which is distinctively different from behaviorism as substantiated in Watson (1998 [1924]) and Skinner (2005 [1953]). Among the various differences, first, mind, to Mead, was not to be reduced to non-mental behavior, as Watson and Skinner did, “but to be seen as a type of behavior genetically emerging out of non-mental types. Behaviorism accordingly meant for Mead not the denial of the private nor the neglect of consciousness, but the approach to all experience in terms of conduct” (Morris 1962: xvii; italics added). Second, human interaction with its environment is not like a “puppet, whose wires are pulled by the physical environment” as assumed in Watson and Skinner, but is dynamic in that only those aspects of the world become stimuli when they effect the release of an “ongoing impulse” (Morris 1962: xvii).

Animals and humans both have impulses, which trigger behavior of various kinds, of which sign behavior is primary. In other words, sign behavior is not the privilege endowed exclusively to human species. Animals also are capable of engaging in sign behavior. For instance, the impulse of feeling hungry is universal in the animal kingdom. It universally triggers the sign behavior of searching for and locating food. What makes the human species qualitatively different is the transition from impulse to rationality, the transition being inconceivable without the aid of human language, the most complicated form of sign behavior ever seen.

From an impulse to its satisfaction there is an action process, which according to Mead (1962) displays a general pattern of three phases: (1) orientation, (2) manipulation, and (3) consummation. These serve as the bedrock for Morris to formulate his theory of sign behavior. To continue the hunger impulse as an example, the actor, triggered by the impulse, launches the orientation phase of searching for food. There are a range of possibilities, one of which is a stimulus occurring in the environment and perceived by distance senses. Now the olfactory sense organ (=nose) registers an odor, which is processed by the brain-mind as the odor of, say, cheese. Thus, a sign is generated that, in the absence of an impulse-satisfying object, causes in an organism a disposition, an “interpretant” in Peirce’s terminology, for producing a sequence of responses of the same type that would be caused by the object itself. The second phase of manipulation, following the disposition, involves contact senses, which are also the modalities invoked in the final consummation phase. The impulse-satisfying object (say, cheese) is the denotatum of the sign. It is worth stressing here that the properties of the sign denotatum — the cheese properties — are correlated with the perceptual sense organs and become part of the overall integrated experience of the whole semiosis.

Morris succeeds in integrating Mead’s three phases of a goal-directed action with three dimensions of signification, namely, designative, appraisive, and prescriptive. In the orientation phase, the actor must “obtain information concerning the situation in which he is to act.” The corresponding semiosis is to make perceptual signs that designate the properties of the situational objects concerned. In the manipulation phase, the actor “must select among objects that he will favor or accord positive preferential behavior,” hence the corresponding appraisive dimension. In the consummation phase, its matching dimension of signification being prescriptive, the actor “must act on the selected object by some specific course of behavior” (see Morris 1964: 7). The integration of the three phases, the corresponding three dimensions, and sensory modalities is graphically shown in Figure 1.

Figure 1

Mead’s action schema and Morris’ adaptation in sign behavior (Note: Smell can be via distance and/ or contact)

The essence of semiosis, as will be shown below, lies in its conceptualization of sign behavior as a process of living experience, unfolding over the here-and-now space-time. The role multimodality plays in such semiosis is twofold. First, it provides living experience with perceptually multimodal input and output. Second, it facilitates the construction of the subject’s experienced environment. Such conceptualization is consistent with Uexküll’s theory of Umwelt, to which we turn.

2.2 The tick semiosis and Uexküll’s theory of Umwelt

Uexküll’s theory of Umwelt is now acknowledged as one of the founding bedrocks of biosemiotics. As shown above, Morris’ behavioral semiotics is meant to embrace sign behaviors of both humans and animals. So it is quite appropriate for us to demonstrate Morris’ semiosis with animal studies by Uexküll (2010 [1934]), although the two giant semioticians were unaware of each other’s works. The case to be cited here is Uexküll’s famous study of a tick in search of its prey in the wild.

A tick, as Uexküll (2010 [1934 ]: 44–45) relates, is an eyeless creature, but with a general sensitivity to light in the skin; it is deaf; it has no sense of taste; and it becomes aware of the approach of its prey through its sense of smell. An adult female tick hangs inert on the tip of a branch in a forest clearing. Its position allows it to fall onto a mammal running past. Once a mammal happens to pass by, its skin glands send out butyric acid, which acts as a stimulus to the hanging tick, whose smell sense organ picks it up as a perception sign, and acts on it by dropping itself down to hit the mammal’s hairy warm skin. The temperature signals to the subject that it has spotted the right prey. The tick then uses “its sense of touch to find a spot as free of hair as possible in order to bore past its own head into the skin tissue of the prey.” For the sake of comparison, a graphic is also drawn to illustrate Uexküll’s narrative (see Figure 2 on next page).

Figure 2

Uexküll’s tick’s meaning-making

The similarity between the tick’s meaning-making behavior and the sign behavior of the hunger–cheese scenario is quite apparent. It is important to note that the tick’s meaning-making behavior is simultaneously both framed and enabled by its perceptual sense organs. It is framed for example by it being eyeless, deaf, and tasteless. In other words, there are no such things as image, sound, or sweetness in the tick’s experienced environment, i.e., the tick’s Umwelt, in spite of the fact that the surrounding may be infiltrated with such things. The tick, however, is enabled by its skin’s general sensitivity to light and its senses of smell and temperature. These modalities facilitate the construction of the tick’s experienced environment in which it lives.

Semiosis is hierarchically organized in view of the complexity of sign behavior. The semiosis, i.e., patterns of sign behavior shared among living organisms demonstrated above, is referred to as primary here-and-now semiosis, to be contrasted with semiosis that transcends here-and-now space-time (see further discussion in section 4 below). It is characterized by the fact that the subject’s living experience and environment are constructed via multimodal interactions within particular physical surroundings, and that the subject’s existence depends on a successful and continuous flow of such multimodal interactions. In an extraordinary case, the hanging tick above had waited for its prey to pass by for 18 years, during which time the tick had been kept in starvation (see Uexküll 2010 [1934 ]: 52).

Now it is time to fine-tune the usage of the term “multimodal semiotic pragmatics.” It first of all designates a commitment to the primary here-and-now semiosis as the ground floor of semiotics. When Morris deals with the sign behavior of language, he takes it for granted that verbal sign behavior builds on the primary here-and-now semiosis. Deely (2001: 7–9) holds that animals remain in, and cannot transcend, their simple Umwelts, whereas humans construct a “linguistic Lebenswelt” on top of the simpler Umwelt. Deely’s view is in total agreement with Morris’.

2.3 Semiotics of Peirce and Morris: Not rival, but complementary

In the literature of both semiotics and linguistics pragmatics Peirce is best known for his theorization of semiotics on the basis of the intuitive notion that a sign “is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign […] which […] I call the interpretant of the first sign” (CP 2.228). It is to be given a more formal definition as follows:

A sign [...] is a First which stands in such a genuine triadic relation to a Second, called its Object, as to be capable of determining a Third, called its Interpretant, to assume the same triadic relation to its Object in which it stands itself to the same Object. The triadic relation is genuine, that is its three members are bound together by it in a way that does not consist in any complexus of dyadic relations. (CP 2.274)

[A sign is] anything which is related to a Second thing, its Object, in respect to a Quality in such a way as to bring a Third thing, its Interpretant, into relation to the same object [...]. (CP 2.92)

Space here does not permit us to elaborate Peirce’s most insightful definition, and readers are advised to consult Liszka (1996) for expert treatment. What concerns us here is the question: Does Peirce’s theory of semiotics, as is assumed by some, pose a rival alternative to Morris? The answer is No. Morris himself observes that his behavioral semiotics “may be characterized as an attempt to carry out resolutely the insight of Charles Peirce that a sign gives rise to an interpretant” (Morris 1946: 27–28). Morris’ resolute devotion to the concept of interpretant is motivated by the fact that an interpretant is ultimately associated with what Peirce calls “habit-change; meaning by a habit-change a modification of a person’s tendencies toward action, resulting from previous experiences” (Peirce CP 5.476; bold and italics added). To Morris, Peirce’s theory of semiotics, though primarily logically constructed, has laid down the genes for his behavior semiotics, viz. the study of signs in terms of sign behaviors of both humans and animals.

Morris (1964: 2) also gives a more formal treatment of semiosis:

For present purposes the basic terms of semiotic can be introduced as follows: Semiosis (or sign process) is regarded as a five-term relation – v, w. x, y, z – in which v sets up in w the disposition to react in a certain kind of way x to a certain kind of object y (not then acting as a stimulus), under certain conditions, z. The v's in the cases where this relation obtains, are signs, the w's are interpreters, the x's are interpretants, the y's are significations, and the z's are the contexts in which the signs occur. (Morris 1964: 2)

Morris warns us that the formulation is not proposed as a definition of sign. It simply gives the conditions for recognizing certain events as signs (Morris 1964). The formation smacks of Peirce in all essential aspects: Morris’ v, x, and y correspond to Peirce’s triadic elements, while Morris’ w, and z only make explicit what is implicitly presupposed in Peirce. For readers who are not very keen on abstract treatment, the dancing bee (in fact Morris’ own favorite animal) is used to demonstrate the five-term relations as shown in Figure 3.

Figure 3

The dancing bee’s sign behavior

The figure is intended to provide a graphic grasp of Morris’ five-term relation of sign behavior. In addition, there is a minor improvement, so the author believes, to Morris in that the separation of the sign (i.e., the dance) and its signification (i.e., the nectar) is explicitly shown to be in two different space-time zones – contrary to the normal, default operation within a here-and-now context. This is by no means trivial! The dancing bee and its fellow nectar-hungry bees, being co-present, interact in the primary here-and-now semiosis. The latter’s interpretants, hence dispositions to act, are immediately associated with the dancing bee’s wagging dance. There is no interval of delay between the disposition and the follow-up action. However, on the dancing bee’s side, the situation is different. To the dancing bee, the separation means that there can be a long interval of delay, which implicates that the dancing bee has a sense of permanence of the nectar absent from the here-and-now context. A long delay interval between a disposition to act and the implementation of the act itself, and a sense of permanence of the object outside the immediate experience, mark a milestone in the human baby’s cognitive development. Morris is aware of the complex issue involved here, but offers no satisfactory solution to it (for the pertinent discussion, see Morris 1946: 24–26).

Admittedly, multimodality, playing a prominent role in Morris’ conceptualization of behavioral semiotics, may seem to be negligible in Peirce. It is not. Peirce differentiates “immediate perception” from “perceptual judgment.” Time, to living experience, is seen as “infinitesimal interval,” in which consciousness is “continuous in a subjective sense, that is, considered as a subject or substance having the attribute of duration, but also, because it is immediate consciousness, its object is ipso facto continuous. In fact, this infinitesimally spread-out consciousness is a direct feeling of its contents as spread out” (CP 6.111; bold original; italics added). The direct feeling of its contents as spread out” is what immediate perception it is. “A fact of Immediate Perception,” Peirce points out, “is not a Percept, nor any part of a Percept” (CP 539). Percept “has for its direct Dynamical Interpretant the Perceptual Judgment […]” (CP 4.540). In other words, from the continuous direct feeling of something to the perception of this “something” as, say, an object, there is an inferential recognition derived from perceptual judgment. When someone walks “through a picture gallery,” if he walks too fast, what he gets is immediate perception – a direct continuous feeling of undifferentiated something;^[4] if instead he walks at a leisurely pace, “[m]uch less can a person prevent himself from perceiving that which […] stares him in the face” (CP 4.540-1). This time what he perceives are pictures – not a blurred something.

It must be kept in mind that Peirce does not deal with perceptual judgment in psychological terms, but in logical terms. Peirce’s semiosis is logically founded: “Logic, in its general sense, is […] only another name for semiotic […] the quasi-necessary, or formal, doctrine of signs” (CP 2.227). Perceptual judgment is an essential component of Peirce’s logical reasoning, for instance, inductive and abductive in particular. Peirce’s complementary contribution to Mead’s three phases of action, and Morris’ three dimensions of signification lies in the fact that they cannot operate properly without invoking perceptual judgment supported by inductive and abductive inferences.

3 Bates’ study of developmental pragmatics: Piaget’s sensorimotor intelligence

Developmental pragmatics by definition is concerned with how children acquire the use of language. Bates’ series of seminal studies (collected in Bates 1976, 1979) attempts to “provide a broad ontogenetic view of the acquisition of pragmatics.” The sampling age period ranges from 9 to 13 months, “a critical period in the emergence of communicative intentions, conventional signaling, and the idea that things have names” (Bates 1979: 315). Intention, communication, convention, and name-object reference are traditional themes of linguistics pragmatics. Bates’ approach to them is distinctive in two ways — its incorporation of Peirce’s semiotics and of Piaget’s theory of sensorimotor intelligence.

Bates (1976: 2) mentions Morris for its “most widely cited definition of pragmatics” as a study of “the relations between signs and their human users.” The definition is found to be flawed with “some weaknesses,” for it “misses the epistemological distinction between content and use, the psychological difference between objects and procedures.” Bates’ critique of Morris is rather hasty and offhand. Morris’ work cited by Bates is Sign, language and behavior (1946 edition by Prentice-Hall), but without indicating the page where the quote is taken. Bates seems to have missed the bedrock of Morris, viz. his theory of behavioral semiotics, which in turn is intimately influenced by Mead’s social behaviorism. When Morris draws the famous tripartite distinctions of syntactics, semantics, and pragmatics, he is not dividing “linguistic science into three areas” as claimed by Bates (1976: 2). Morris is dealing with division of labor in behavioral semiotics, of which linguistics is held to be a sub-part. By dismissing Morris, Bates adopts Peirce’s theory of semiotics based on three types of signs: icons, indices, and symbols. These three types of signs become important metalanguage for Bates to construct her developmental pragmatics. Morris’ behavioral semiotics and its inter-connection with Peirce’s, as discussed above, seem to be unnoticed, or ignored by Bates.

The second distinctive feature of Bates’ approach is the incorporation of Piaget’s sensorimotor intelligence. Piaget regards infant growth and maturation as a process of intelligence development which is constructed incrementally through postnatal experience. He argues that development is structured with distinctive landmarks or stages. There are three major stages, each of which allows for further fine-grained identification of sub-stages. The first, being most relevant to discussion here, is the sensorimotor intelligence (from birth up to one and one-half to two years, see Piaget 1971:17). Piaget (1953) draws the following fine-grained distinctions:

Elementary sensorimotor adaptations

The 1st stage: the use of reflexes

The 2nd stage: the first acquired adaptations and the primary circular reaction

The intentional sensorimotor adaptions

The 3rd stage: The secondary circular reactions and the procedures destined to make interesting sights last

The 4th stage: the coordination of the secondary schemata and their application to new situations

The 5th stage: the tertiary circular reaction and the discovery of new means through active experimentation

The 6th stage: the invention of new means through mental combinations

Bates’ study of infant pragmatic acquisition, in theory formation, embraces the whole range from 0 to 18 months, while in empirical investigation her data set (9 to 13 months) only covers Piaget’s Stages 4 to 6, the choice of which, as pointed out above, is believed to be associated with the “dawn of language.”

What is the “pragmatics” the infant babblers attempt to acquire during the sensorimotor period in preparation for late speech development? Bates and her associates focus mainly on the “three major aspects of pragmatics — performatives, propositions, and presuppositions” (Bates 1976: 113). As we all know, these pragmatic building concepts are all formulated on the basis of the full-fledged cognitively mature adult with a philosophically complicated mind. In what way is it justifiable to apply them to immature babblers? Bates’ ingenuity lies in her interpretation of these concepts in procedural terms. Take performatives for example. “The term performative describes the organization of the child’s communicative goal, e.g., to obtain an object through use of an adult, or to obtain adult attention through the use of an object” (1976: 113; bold original). Carlotta and Marta, Bates’ two infant subjects, were found to have constructed such a “performative without words” prior to their referential use of words. “We are led to the tentative conclusion that the sensorimotor performative is based on the cognitive developments of Stage 5, while the use of words with referents in such sequences is dependent upon the capacity for internal representation characteristic of Stage 6” (Bates 1976: 77; italics added).

The sensorimotor performative is, in our view, the subject matter proper of multimodal semiotic pragmatics. Before looking at it more closely, we need to review Piaget’s conceptualization of “sensorimotor activity schema,” which not only underpins Bates’ sensorimotor performative, but also arguably lays the foundation for multimodal semiotic pragmatics.

The newborn’s first cry declares the beginning of its exploration of the new world. As far as multimodality is concerned, embryological studies find that the somesthetic system (kinesthetic and cutaneous processes) is the earliest sensory system to develop in the human embryo (Stack 2001: 351). The first indication of the developing ear can be found in embryos of approximately 22 days (Sadler 2012: 321), and the auditory system becomes functional to some extent by the sixth month (Fernald 2001: 41). Karmiloff and Karmiloff-Smith remark:

From the sixth month of gestation onward, the fetus spends most of its waking time processing these very special linguistic sounds, growing familiar with the unique qualities of its mother’s voice and of the language or languages that she speaks. (2001: 1)

As for sight, at the time of normal birth, “the peripheral retina of the eye is quite well developed, but the central retina […] is poorly developed and undergoes considerable post-term changes” (Slate 2001: 8). Unlike hearing, visual experience is impossible prior to birth. “It is therefore not surprising to find that the visual world of the newborn infant is quite different from that of the adult.”

It is apparent now that the newborn’s sensory modalities at birth are unevenly developed. Bearing this in mind, we are interested in how the newborn, equipped with immature, but growing multimodality, explores the new world through activities, such as sucking, kicking, touching, grasping, holding, reaching, to name but only a few, which in Morris’ terminology are sign behaviors or, to adopt Piaget’s terminology, sensorimotor activities or behaviors. It is important to note that sensorimotor behavior is no less pragmatics than verbal behavior, since it also involves a triadic relation between the subject, the sign and the object.^[5]

Let us take the sucking reflex behavior for example. It involves, among other things, the sensory tactile stimulus as “external excitant” (Piaget’s term), and the motor responses of mouth and tongue movements. The sucking reflexes show a behavioral pattern which Piaget calls the sucking reflex schema. The sucking reflex in the first instance may be activated to function when the newborn rubs its lips with its own hand, or when the mother’s breast touches its lips. It can even be set in motion when its lips are touched by a cloth or an object like that. This “global” sucking reflex schema can be captured graphically as shown in Figure 4.

Figure 4

The global sucking reflex schema

Piaget characterizes this global schema as “generalizing assimilation,” i.e., the incorporation of increasingly varied objects into the child’s reflex schema, such as sucking his finger, mother’s breast, pillow, quilt, bedclothes, etc. The newborn of course quickly updates the schema by “recognitory assimilation,” i.e., by differentiating the nipple from non-nipple objects. The differentiation is associated with swallowing and satisfaction (i.e., “consummation” in Mead and Morris). Persistent failures in achieving satisfaction lead to crying or rejection, which in turn results in the stop of the reflex behavior — when this happens, accommodation has taken place. The updated sucking reflex schema is shown in Figure 5.

Figure 5

The sucking reflex schema updated

What do we learn from this analysis? It is one of Piaget’s fundamental tenets regarding the child’s development, namely that all mental objects are constructed from the child’s sensorimotor activities upon the external world. Real world objects do not copy themselves onto the passive child. Rather, the child explores the world of objects through varied sensorimotor activities empowered by the framing and enabling, but maturing, hence dynamic, sensorimotor capabilities. During this exploration, the child tries to impose its own “sensorimotor schema” on the world (i.e., assimilatory adaptation) and revises those schemata when the child meets resistance (accommodatory adaptation).

Our analysis of Morris and Uexküll above has rendered it obvious that this fundamental tenet is shared by Morris’ behavior semiotics and Uexküll’s biosemiotics. The infant constructs its Umwelt (i.e., experienced environment) through its sensorimotor activities (i.e., sign behavior or meaning-making behavior). The infant’s Umwelt, due to its immature multimodality, is different from that of adults. What makes a human infant different from, say, a tick’s baby, is that, as far as multimodality is concerned, the former’s sensorimotor activity schema is highly plastic, capable of both assimilatory and accommodatory adaptations, whereas the latter’s capacity is genetically fixed.

Now let us return to Bates. Bates’ developmental pragmatics does not take the infant’s language as something innately endowed, as assumed in the Chomskian paradigm, according to which all the postnatal experience does is simply “trigger” the endowed I-language to grow (Cook and Newson 2000: 106).^[6] Bates, in contrast, adopts Piaget’s epigenesis and shows that pre-speech sensorimotor activities lay a foundation for the development of language use. Bates’ developmental pragmatics is rich, and full justice to her works has to be found elsewhere (see Tomasello and Slobin 2005).

4 The primary here-and-now semiosis: Total saturated experience with total saturated signification

The hunger–cheese sign behavior from orientation to manipulation to consummation results in what Gu (2009b) proposes to call total saturated experience (TSE) with total saturated signification (TSS). Gu uses the scenario of enjoying the roast Peking duck to illustrate the concept. Given the goal, there are a range of possibilities: (1) going to a Chinese restaurant and eating a real roast Peking duck; (2) watching others eating it; (3) watching a video show of how people enjoy eating it; (4) listening to a talk about how a roast Peking duck is being made; (5) reading a recipe about how to make a roast Peking duck. The first one in comparison with the remaining represents a TSE of duck-eating. Its TSS is typically associated with the qualities extracted from multimodal interactions with the real object, for example, qualities such as its taste, color, odor, crispiness, and tenderness, as well as emotional states triggered by them.

In the primary here-and-now semiosis where the TSE-TSS is attained, the subject-object relation, in the subject’s experienced Umwelt, is, first of all, a consummatory relation established through contact senses (taste, touch, etc.). In the Chinese theory of cuisine, sight and smell, Morris’ distance senses, are also regarded as equally contributing to the consummatory relation. The reference relation traditionally held in linguistics pragmatics between the subject and object is obviously inadequate. One can testify this by the fact that one can never enjoy eating duck by simply referring to or pointing at it!

The primary here-and-now semiosis characteristic of TSE-TSS has developmental significance not only at individual personal level, but at the national societal level. Human languages in view of TSE-TSS experience develop unevenly. Each language is like what Wittgenstein calls an ancient city (1997 [1953]: 5e), with old streets and lanes, as well as new buildings, CBDs, etc. Take China for example. There are about 150 or more languages. Mandarin Chinese in comparison offers more modes of experience than the remaining ones. The primary here-and-now semiosis with TSE-TSS, i.e., multimodal semiotic pragmatics, is like the basement floor, on top of which there is what Gu (2009a) calls the land-borne situated discourse (LBSD), i.e., the sensorimotor-constructed Umwelt plus linguistic Lebenswelt — the oldest in history, but the most dynamic, fleeting, saturated mode of experience. Mandarin Chinese over three thousand years ago, on the other hand, invented writing script, thus creating another mode of experience, i.e., experience based on visual interpretation of written sign vehicles. This is the written word-borne discourse (WWBD). Of 150 or so languages, only a minority provide such a mode of experience for the users. Mandarin Chinese is further privileged to be able to provide two more modes of experience: (1) telephones calls, radio broadcasts, TV programs, etc., that is to say, air-borne situated discourse (ABSD) for oral-auditory as well as visual consumption; and (2) the latest Web-borne situated discourse (WBSD), which totally transcends the primary here-and-now live semiosis into the domain of virtual reality.

5 The primary here-and-now semiosis integrated with the Piagetian sensorimotor schema

Now it is beneficial to revisit Morris’ definition of pragmatics cited above. Pragmatics is conceptualized to deal with the biotic aspects of semiosis encompassing all the psychological, biological, and sociological phenomena occurring in the functioning of signs. We have used two concepts multimodality and sign behavior as two exploratory probes to fathom the pragmatics initially envisaged by Morris. In this sub-section we would like to highlight the interactions of the sensorimotor mechanism, which are biological in nature, with the psychological and sociological via the subject’s live sign behavior. Take greeting for example. Greeting in the mode of primary here-and-now semiosis, if fully performed, is a TSE with TSS. The TSE-TSS greeting displays simultaneously multiple layered properties: (1) the well-integrated sensorimotor activity schema; (2) the gestural or verbal schema; (3) the emotional companionship; and (4) the sociocultural appropriateness. Layers 2 to 4 hardly need elaboration here. The first covers a complex phenomenon of bio-psychological nature. A well-integrated sensorimotor activity schema can be graphically shown in Figure 6.

Figure 6

Sensorimotor schema of greeting^[7]

The sensorimotor display of positive emotion is the externalization of the current thought and psychological emotion (e.g., feeling happy to see the greetee) that accompany the gestural and/or verbal behavior of greeting. Gu (2013) labels it as mao (貌 i.e. the embodiment) of thought and emotion, which is observable in facial expression, bodily posture, prosody of speech, and so on. The four components are normally sequentially organized, and well-coordinated. Malfunctions or hiccups will result, with everything else being equal, in failure to greet.

Why bother about sensorimotor activity schema and well-integratedness? Does it have anything to do with linguistics pragmatics? Gu (2013) shows that incongruence between what is said and what is emotionally displayed also triggers implicatures. Children with autism spectrum disorders (ASD) are particularly compromised in reaching mutual eye contact, still less in maintaining it for long. No better justification is given than that which studies of pragmatic impairments provide, to which we now turn.

6 Pragmatic impairment: Perkin’s study

The current mainstream pragmatics, apart from its bias on modeling the mature adult as pointed out above, suffers from another limitation, namely that it is based on the assumption that linguistic communication “typically appears to be a single, seamless process” (Perkins 2007: 8). Little consideration is given to communication disorders in its theory formation. Moreover:

Unlike clinicians, who need to understand a condition in its entirety in order to play appropriate intervention, pragmatic theorists have had the luxury of being able to focus only on the specific features which are of interest to them. (Perkins 2007: 8)

When the current theories are applied to the clinical context, “it is not always well suited to the needs of language pathologists and has led to a great deal of confusion in the clinical diagnosis of pragmatic impairment, and in regard to the nature of pragmatic impairment itself” (Perkins 2007: 8).

Hence Perkins argues for a holistic model of pragmatics, namely “emergentist pragmatics” (EP). Its holistic approach is particularly seen in the inclusion of “sensorimotor systems” as one of the three “elements” of theory construction (the other two being cognitive systems and semiotic systems). There are both theoretical and practical aspects that are compelling for Perkins to make this inclusion.

Apart from obvious examples such as the use of gesture to compensate for linguistic output problems and the use of facial expression and tone of voice to interpret the attitudinal or emotional state of a speaker during comprehension, sensory input and motor output systems are rarely included in discussions of pragmatic ability and disability. However, once pragmatic functioning is seen as an emergent phenomenon it is clear that sensorimo-tor systems provide a range of communicative choices in the same way that cognitive and linguistic systems do; that restriction in choice as a result of impairment is pragmatically constraining and can have a knock-on effect both within the sensorimotor domain and in cognitive and linguistic domains; and that sensorimotor systems are as vulnerable as language and cognition to the effects of compensatory adaptation during interpersonal communication. (Perkins 2007: 139)

The most devastating knock-on effect of sensorimotor impairment on pragmatics (and language in general), perhaps, is the loss of vision (acquired blindness) and hearing (acquired deafness). Alternative modality compensation has to be attained, for example, tactile modality for sign behavior and communication.

Pragmatic impairment due to malfunctions of sensorimotor systems presupposes knowledge about the normal functioning of the system, i.e., knowledge of what Gu (2007, 2015) calls “multimodal congruence.” Perkins touches upon the same phenomenon when he points out that visual and auditory perception play a key role in inferential processing, and misreading of facial expression or voice quality could result in failure to detect irony. Moreover, the “expression of emotion and attitude is particularly multimodal, with meaning being conveyed via articulation, voice quality, prosody, facial expression, gesture, posture and gaze” (Perkins 2007: 140).

The coordination and integration of sensorimotor systems with cognitive systems and semiotic systems, as Perkins correctly points out, “is a relatively unexplored area” (Perkins 2007: 139). There are many hard and pressing problems calling for solutions. Here is an instance taken from our corpus of Alzheimer’s disease patients. As shown in the screen shot (see Figure 7 on next page), the left is the AD patient, and the right is his second son. There is a third person, i.e., the researcher, not shown in the picture. The conversation goes as follows:

Figure 7

Visual agnosia of an Alzheimer’s disease patient

Researcher: (to the AD, while pointing at the son) Do you recognize him?

AD: (Turns to look at his son) er …

Son: (to the AD, while pointing at himself) Do you recognize me?

AD: (Staring at his son) er …

Pragmatically speaking, the impairment can be categorized as person recognition impairment, or visual agnosia. But what is the cause of this impairment? Is it associated with the condition of cognitive memory retrieval? Or is it due to the compromise of associated visual neural pathways? Even if definite answers are found, clinicians are pressed by the patient for a therapeutic solution, which is yet another hard problem that is currently unsolvable.

7 Multimodal semiotic pragmatics: Future directions for Morris’ vision

It is worth recalling the fact that the term multimodal semiotic pragmatics does not refer to a mature established theory, out there, ready to pick up and use. It only points to a research direction initially envisaged by Morris, toward which other researchers, aware or unaware of Morris’ works, happen to converge.

Multimodal semiotic pragmatics, as reviewed above, clearly holds a complementary relation with current mainstream linguistics pragmatics. That is, it does not, and cannot, replace the existing theories. Conversely linguistics pragmatics, as it is, does not, and cannot replace multimodal semiotic pragmatics either. Having said this, multimodal semiotic pragmatics seems to be “more basic,” that is, it can serve as the “ground floor” on which linguistics pragmatics is to be more securely situated. This is the direction toward which Bates’ developmental pragmatics, Perkins’ EP, and Gu’s study of four-borne discourses seem to lead us.

There are, of course, many research issues associated with multimodal semiotic pragmatics itself proper. As pointed out above, Morris would intend multimodal semiotic pragmatics (i.e., primary here-and-now semiosis) to include both animals and humans. So logically there will be multimodal semiotic pragmatics for animal sign behaviors. Very interestingly, the latest growing interest in biosemiotics (Hoffmeyer 1996; Barbieri 2007; Romanini and Fernandez 2014) seems to be moving toward this direction. Biosemiotics also recognizes sign behavior and the sensorimotor mechanism as two domains of properties common to living organisms. Adopting these two as the point of departure to conceptualize pragmatics, the outcome is not just an issue of being broader or narrower, but brings to the fore the fundamental question of where language comes from. The issue of the evolution of language has been literally “out of the question,” for mainstream linguistics pragmatics takes language as given, and pragmatics’ business proper is to see how language is used in communication. This is particularly transparent when a componential view of pragmatics is adopted, with a division of labor between syntax, semantics, and pragmatics.

Ironically, the componential view is believed to have originated in Morris. A close reading of Morris, however, will show that Morris is not at all responsible for such a division of labor in linguistics pragmatics. This is because Morris, drawing the tripartite division, is concerned with behavioral semiotics, not linguistics. Moreover, “syntactics, semantics, and pragmatics,” are “subordinate branches” “dealing, respectively, with the syntactical, the semantical, and the pragmatical dimensions of semiosis” (1951 [1938]: 8; italics added). Morris is not recommending that semiosis be divided into three parts, but rather that it be viewed from three perspectives, as he takes it for granted that “the various dimensions are only aspects of a unitary process” (1951 [1938]: 8; italics added). Later on, he emphasizes that, “after making use of the abstractions involved in this treatment, we will specifically stress the unity of semiotic” (1951 [1938]:13; italics added).

Of the three dimensions abstracted from semiosis, the pragmatic is, in actual use, primary and fundamental; this is because it operates at the level of behavior making something become a sign in the first place. This is implicated in the remarks Morris makes:

Syntactical rules determine the sign relations between sign vehicles; semantical rules correlate sign vehicles with other objects; pragmatical rules state the conditions in the interpreters under which the sign vehicle is a sign. Any rule when actually in use operates as a type of behavior, and in this sense there is a pragmatical component in all rules. (1951 [1938]: 35)

To put it differently, there is a sequential order implicit in semiosis, namely syntactics and semantics will have nothing to do until a sign is made, and it is the sign behavior which fixes a sign vehicle and the object the sign is made to stand for.

Multimodal semiotic pragmatics does not take language for granted. Verbal behavior is a species of sign behavior. In the period of evolution, human verbal behavior is a latecomer. Stokoe observes:

When gesture is defined as body movement that communicates more or less consciously, and Sign is taken as a generic term for natural sign languages, the progression from gesture to Sign is entirely natural. What makes it so is the nature of the hominid phenotype and its attributes — its vision and the physical structure and use of hands, arms, faces, and bodies. (Stokoe 2000: 388)

If one accepts Stokoe’s position, multimodal semiotic pragmatics also has a role to play in the story of language evolution.

Finally, it is worth emphasizing the fact that Morris’ conceptualization of semiotics is intended to have a unifying function:

It is doubtful if signs have ever before been so vigorously studied by so many persons and from so many points of view. The army of investigators includes linguists, logicians, philosophers, psychologists, biologists, anthropologists, psychopathologists, aestheticians, and sociologists. There is lacking, however, a theoretical structure simple in outline and yet comprehensive enough to embrace the results obtained from different points of view and to unite them into a unified and consistent whole. It is the purpose of the present study to suggest this unifying point of view and to sketch the contours of the science of signs. (1951 [1938]:1)

Morris’ “unified and consistent whole” theory of signs is indeed solidly demanded in real-life semiosis, which can be demonstrated by what Gu (2012) calls the “ecological chain” of activities. An individual Mr. Y suffers from hay fever as the result of his interaction with the physical environment. He sneezes like mad. He takes a bus to go to a drug store — he enters into a node on the web of spatial-temporal trajectories framed and enabled by the community. In the drug store, i.e. another node on the web of spatial-temporal trajectories, he talks about his hay fever to a girl assistant, who shows him a few choices and offers him some advice. He makes a choice and pays. This whole transaction would have been impossible without a drug manufacturer producing the drug. The latter, on the other hand, would never have been produced without research on the drug. The drug research, in turn, is motivated by the fact that Mr. Y is not alone in sneezing upon exposure to flowering plant pollens. The whole series of activities, each seeming to happen separately and independently, actually form a coherent whole, as shown in Figure 8.

Figure 8

An ecological chain of activities. Quoted from Gu (2012: 550)

It would be no small achievement to account for this ecological chain in terms of the unifying metalanguage of behavioral semiotics as envisaged by the founding father!

A final note is in order here. There has been a growing body of studies on “embodied mind” and “embodiment” in general that should have been included in this paper. Thanks to Lakoff and Johnson’s writing (e.g., Lakoff and Johnson 1999; Johnson 2007), the literature is well known in the linguistics community. The notion of embodiment, as used in philosophy with an emphasis on the role that “body” plays in meaning-making, and the notion of multimodality, as discussed in this paper, are obviously not equivelant, or interchangeable in usage. But in real-life terms, perceptual sense organs (multi+modalities) are intrinsic components of the body. Hence embodiment studies constitute a viable strand, unwittingly or otherwise, toward the lost Morrisian pragmatics. The author has in fact drafted another paper reviewing the synergetic partnership between muldimodality and embodiment. All the author can do here is apologize for his negligence.