Semiotic Processing in Working Memory

1 Introduction

Using C. S. Peirce’s categories of Firstness, Secondness, and Thirdness as a framework, I demonstrate that the oral elicited imitation (EI) test is a valid and reliable measure of the state of L2 working memory (WM) processing at distinct developmental levels. I first establish its history as a viable empirical method both in L1 and in L2 experimental genres, then attend to its utility to assess language processing in WM; and finally, I discuss its particular application to semiotic inquiry. To exemplify EI’s utility as a semiotic mode of inquiry, I show how responses of L2 Spanish learners at two distinct proficiency levels demonstrate qualitatively different representations. The case is made that while L2 learners with less language experience represent linguistic knowledge as percepts (1903: CP 7.624), learners with substantially greater L2 experience represent linguistic knowledge in the form of perceptual judgments. Accordingly, this inquiry suggests that particular testing procedures are more efficacious (the EI paradigm) than other approaches (drawing from natural speech samples) to measure the meaning accorded to signs across L2 proficiency levels. EI responses further demonstrate the pivotal role of language experience to advance “Interpretant building,” illustrated by alterations made to the respective original stimulus sentence (lexical deletions or lexical substitutions). The following assertion by Hameyer (1980: 21) underscores the need for further research into the types of morphemic modifications across proficiency groups: “…Incorrect inflections and incorrect substitutions are the surest single indicators by which to distinguish proficiency groups.”

2 The elicited imitation procedure

Prior to describing and defending the elicited imitation procedure as such, attention is accorded to the working memory system and its function in language processing, since it is linguistic WM competency which EI best measures. The WM system most relied upon by linguists is that of Baddeley and Hitch (1974). Later, in 2000, Baddeley implemented a modification to the original model. The original model posits that two slave short-term memory (STM) systems (the phonological loop and the visuospatial sketchpad) and the central executive make up the WM system. The modification merely consists in the addition of the episodic buffer, which primarily integrates information from the other subsystems and provides additional storage. Processing within both the loop and the sketchpad is characterized by limited slots or units of information stored for an abbreviated time span – three seconds at best. The number of linguistic units capable of being stored in WM depends on whether learners are processing at the phonological, morphological, or semantic level. WM word count ranges from approximately four for non-speakers (Hameyer 1980: 14) to approximately fifteen for the most proficient speakers (Naiman 1974: 72; Erlam 2006: 477, 2009: 78)). My own findings (West 2012: 216–217) reveal that L2 learners of differing competencies (distinct L2 proficiency levels) typically store between seven morphemes (for elementary learners), nine morphemes (for intermediate learners), and eleven morphemes and beyond (for advanced L2s). My own research sought to measure the number of morphemes (rather than the number of syllables/words), given the greater sensitivity to meaning on the word level; such more accurately (compared to using words as the measure) determines the number of meanings attributed to each EI stimulus sentence – many fewer for the beginning L2s than for more advanced learners.

In view of these recall limitations, it is critical that language learners process information in WM on the morphemic or semantic level, rather than on the sound level; otherwise, far fewer language units are likely to be recalled and shunted to long term memory (LTM). Processing at the phonological level requires far more WM resources than does processing at higher, more meaning-based levels, given the need to assign sound to representation and to integrate abstract sound units into words. Since WM resources are at a premium for beginning L2s, burdening an already limited capacity storage results in decreased recall of the stimulus sentences. Nonetheless, Ellis identifies a functional advantage which WM’s limited capacity affords: “the limitations of working memory drive learners to construct higher order chunks out of lower order chunks” (Ellis 2008: 469). The latter function – sequencing morphosyntactic items in WM – is likely to be less utilized since attentional resources are expended on LTM retrieval and message coordination. This burden on attentional resources is responsible for an increased likelihood of L2 errors or slower “serialized articulation” (non-automatic speech) (Temple 1997: 87). Establishing the limits of WM at distinct L2 proficiency levels lays the groundwork for the use of EI to measure competencies in oral memory (cf. Skehan 1998: 78). EI can systematically control the phonemic/morphemic/word count, thereby establishing thresholds for meaningful processing at different language competency levels.

The oral EI procedure developed for this study requires language learners to listen to a recorded contextualized set of twenty-four stimulus sentences individually presented in the target language (TL). These stimulus sentences systematically contained from six to thirteen morphemes (seven to thirteen syllables). Although stimulus sentences were presented as individual frames, they could be constructed to relate to one another and to the discourse at large, since each contributed to the outcome of a story. Learners were tested individually, and were instructed to first provide a word association for the sentence then to repeat each sentence as it was heard. The requirement to supply the word association was implemented (West 2012: 211–212) to ensure that learners were focusing on meaning (not merely form), measuring other than verbatim recall. Focusing on meaning is critical to establish that responses reflect implicit language knowledge, and not explicit knowledge emanating from memorized chunks or from language rules which are taught (Erlam 2006: 466; 2009: 67). The rationale for requiring an interval between administration of stimulus sentences and respective responses is that three seconds represents the threshold after which sentence recall no longer represents mere regurgitation (Eysenck 2001: 160). In fact, if memory is not refreshed within three seconds of presentment via some form of rehearsal (either vocal or subvocal) of the language string, recall declines (since access to non-refreshed information is unlikely after short intervals [Baddeley 2007: 51]). In other words, once WM is cleared of the iconic (phonetic) representation, semantic representations can be accessed and analyzed should they exist in the interlanguage (IL). These semantic alterations are associated with responses of more experienced learners, who increasingly have the means to infuse linguistic signs with more generalizable Thirdness-based interpretants.

An additional advantage of EI over other methods (natural speech corpora, timed grammatical judgments) to test implicit knowledge or, better said, the state of associated interpretants is insurance that the same opportunity exists for all participants to produce the linguistic sign and to access its interpretant. Since this battery of EI consists of twenty to forty identical stimulus sentences presented to all participants in the same sequence, production of the signs is not left to chance (Bley-Vroman and Chaudron 1994: 256); as a consequence, avoidance of the linguistic structure(s) under consideration is a non-issue. To this end, fashioning the stimulus sentences to include particular grammatical structures (two past aspects in Spanish in the present study) and systematically altering the placement of the structure and morphemic count within the stimulus sentence in question is paramount. Implementing these features into the design before presentment to the participants guards against confounding recall of the items with effects of primacy or recency – sequential factors which must be considered (for a fuller account of these design issues cf. West 2012: 210–211). EI creates a level playing field, permitting more valid comparisons across subject performance.

Findings from studies employing the oral EI approach have tapped the state of L1, as well as L2 implicit language knowledge. Lust (1986) and Lust, Flynn, and Foley (1996) have used the EI test to demonstrate children’s syntactic knowledge at different mean length of utterance MLU levels. Lust et al.’s findings unequivocally show that at different developmental levels children’s responses are qualitatively different – indicated by the state of their reconstructions of the original stimulus sentences. Responses at different proficiency levels to: “Mommy drinks milk and Mommie eats cookies,” can elicit an exact echo of the stimulus sentence or a reconstruction: “Mommie drinks milk and eats cookies” (cf.: West in press for an extended discussion of this phenomenon). This reconstructive ability, substituting a meaningful word in place of the non-recalled one, demonstrates that learners are chunking at the semantic level, rather than at the sound level. In this sample, processing at the phonological level alone (chunking sounds) results in word deletions or in producing nonsense words which conform to the sound structure of the language in question (West 2012: 217).

More importantly, in addition to revealing the extent of learners’ underlying syntactic knowledge, reconstructions reflect meaning-based advances – associations which hold between linguistic signs, their objects, and their interpretants. These word–object–meaning connections are a direct consequence of qualitatively different semiotic representations which account for semantic advances, not merely syntactic ones, as Lust et al. (1996: 57) assume. The distinct semiotic representations are sound-based, word-based, or phrase-based. Their signs are characterized either as: primarily phonemic signs, morphemic signs, or semantically derived signs. While all of these signs can entail some degree of chunking (connecting individual units into a composite), the latter exacts chunking at a higher level, such that substantially more information belongs to a chunk. This form of processing, guided by increasingly higher semiotic levels, is responsible for enhanced chunking, hence enhanced linguistic processing.

3 Peirce’s distinction between percepts and perceptual judgments

Peirce determines that percepts are foundational to perceptual judgments, such that the former are grounded in Secondness, while the latter rely on a more Thirdness-based rationale: “There are several other points of contrast between the perceptual judgment and the percept that are calculated to exhibit their disparateness. The judgment, ‘This chair appears yellow,’ separates the color from the chair, making the one predicate and the other subject. The percept, on the other hand, presents the chair in its entirety and makes no analysis whatever” (1903: CP 7.631). Firstness-based components are evident in both forms of cognition; but, in the former case (percepts) Firstness takes the form of affect and attentional preference, while the latter case (perceptual judgments) operates on idiosyncratic, rational propositions. Percepts appear to be foundational to perceptual judgments in the same way that sensorimotor schemes serve as a basis for logical reasoning in Piaget’s system (Piaget and Inhelder 1948/1967: 455) – early processes are used to reconstruct mental constructs, such that percepts (as attention to existent objects) are necessary to build perceptual judgments (propositions involving world knowledge).

For percepts, Firstness entails noticing similarities and nuances among entities and events; hence its effect is couched in replica-building. Noticing gives rise to simple illustrations of haecceity (1897: CP 3.460) – limited reach/access to “thisness” (entities near to self), or to individuality (OED). Firstness-based applications to perceptual judgments take the form of individual insights which appear compulsively, not as a consequence of conscious and contrived arguments. Peirce (1903: CP 5.181) asserts that compulsions/insights are not controllable. Insight and uncontrollability are characterized as Firstness-based issues, by virtue of the possibility and individuality that hold between the sign relations.

Both percepts and perceptual judgments likewise have their roots in Secondness, although percepts are far more bound to Secondness – cognitions which have a significant foundation in material fact and action. According to Peirce, percepts go beyond “first impressions of sense,” (1902: CP 2.141). “Beyond first impressions of sense” does not indicate that sense experience is immaterial or minimized, but that percepts supersede initial exposure to a stimulus, perhaps to include some level of expectation of the meaning/effects of the sign. This basic relationship is never supplanted by perceptual judgments; percepts merely serve as their foundation. In CP 5.538 (c. 1902), Peirce contends that perceptual judgments exhibit a higher level of cognitive intervention, propelled by the element of deliberateness. Deliberateness does not imply consciousness, nor does it intimate any planning. In fact, the perceptual judgment is a necessary component in “recommending a course of action” (1903: MS 637: 12), which constitutes one of Peirce’s primary ingredients in abductive reasoning. What precludes conscious deliberation and developing sequential steps toward a goal is the presence of spontaneity. The fact that a perceptual judgment must be spontaneous permits the judgment to be volitional without the presence of forethought directing a series of related events.

Although some interpretation exists in the percept, its character does not rise to the level of perceptual judgments until reflection materializes and until the “totality” of the object and its context confront the mind. In CP 7.643 (1903) Peirce refers to this reflective mental confrontation as the kernel of perceptual judgments, the “percipuum”:

We know nothing about the percept otherwise than by testimony of the perceptual judgment, excepting that we feel the blow of it, the reaction of it against us, and we see the contents of it arranged into an object, in its totality…. But the moment we fix our minds upon it [the percept] and think the least thing about the percept, it is the perceptual judgment that tells us what we so “perceive.” For this, and other reasons, I propose to consider the percept as it is immediately interpreted in the perceptual judgment, under the name of the “percipuum.” The percipuum then, is what forces itself upon your acknowledgement, without any why or wherefore, so that if anybody asks you why you should regard it as appearing so and so, all you can say is, “I can’t help it…”

In the percipuum the mind becomes “fixed” upon the object and upon the judgment, illustrating an absence of conscious deliberation either to form the percept or the perceptual judgment. It must also be mentioned that the judgment requires some reflection, unconscious though it may be. Nonetheless, this forceful focus is primarily cognitive not affective; hence its foundation is epistemic, not deontic. Peirce, however, is silent regarding how and when percipuum emerges, and its continuing function in measuring the efficaciousness of resultant perceptual judgments.

Other means, perhaps more reliable ones, test whether cognitions constitute percepts or whether they rise to the level of perceptual judgments, namely, children’s early nonverbal responses to nonlinguistic stimuli, especially responses not bidden by instruction. One of these is preferential looking at objects which are recognizable; the other such measure is sorting behaviors to an array of objects by: shape, color, etc. These early nonlinguistic measures can determine the onset and quality of the representation obviated by the quality of response to the objects, for example, automatic, unsolicited object sorting by similar shape, or looking longer at similarly shaped novel objects rather than familiar ones. If indexical features mediate the representation, the response is similarly mediated by shape (diagrammatic) and location (pointing to place issues). In sum, the state of processing in WM or in LTM can best determine whether the nature of the representation is a percept (when mediated by index) or a perceptual judgment (often mediated by symbolic representations).

Because of the necessity for symbolic mediation, Thirdness is far more integral to perceptual judgments than to percepts. Accordingly, perceptual judgments, since they assert a semblance of belief/veridicality, constitute Thirdness-based relations, whereas percepts are not so founded (1903: CP 7.622; 7.631). Because the foundation of perceptual judgments extends beyond any sense impressions in Secondness to rest on “retroductions” (1903: MS 637: 12), they constitute propositions – not mere stark experience. These spontaneous propositions often propose a remedy, taking into consideration a panorama of previous experience. In short, judgments are formulated via a hunch generated from an abbreviated “look back” on similar experiences at large, hence Peirce’s use of “retroduction.” These retroductions are not mere “looks back,” otherwise “retrospection” rather than “retroduction” might have been chosen. Instead, these “looks back” supersede retrospections. They are enriched by Thirdness-based interpretants, drawing relevant past and possible subsequent experiences together into a coherent system. This system establishes functional relationships between and among relevant events, and monitors whether and how any conclusory proposition (recommendations of a course of action) should be altered with intake of subsequent experience. It is obvious then, that more complex meaning underlies perceptual judgments, and that percepts, even though they involve interpretation, largely represent adherence to existing schemes.

4 Application to semiotic genres

EI responses from distinct L2 competency levels reveal that whereas percepts guide beginner and intermediate learners, perceptual judgments underlie advanced learners’ responses (West 2012: 216–217). The percepts of the less experienced L2s consist in processing on the phonological level, such that linguistic sound units constitute the attentional forum. Beginning learners primarily chunk sounds; and any expectation(s) which arises results from awareness of the particular sound patterns of the language, for example, which consonant clusters exist and in which location within words, as opposed to processing on the morphological or semantic level. Beginning and intermediate L2s likewise are more likely than are more advanced learners to delete words or portions of words when recall fades (West 2012: 216). In contrast, more advanced L2s process on the morphemic and/or on the semantic level; and their chunks consist in meaningful pieces of words and/or words themselves. Chunking morphemically/semantically derives from attaching meaning to a lexical item; and EI modifications frequently give rise to word substitutions as opposed to deletions (West 2012: 216). Such substitutions might include a response of: “The children eat shiny apples” to the EI prompt “The children chew shiny apples.” In this case, the substitution of “eat” for “chew” represents a meaningful one; and both consist in a monosyllabic meaning unit. The initial and final words/morphemes are less likely to experience alteration, given primacy and recency effects. In short, alterations made to recall the EI string can indicate at which level learners are processing, phonologically (using percepts) or semantically (relying on higher level meaning chunks). Moreover, when percepts underlie the processing approach, as in phonological chunking, deletions are the more likely modification in the face of non-refreshed EI strings; but, when perceptual judgments support EI stimulus sentences, substitutions (often meaningful) surface to compensate for non-refreshed and partially forgotten items in WM. In sum, the presence of these morphemic substitutions indicates the use of underlying Thirdness-based semiotic interventions.

It is evident that at the beginning L2 proficiency level, percepts prevail, since phonological processing rests upon sense impressions or stimuli which go slightly beyond “first sense impressions.” Learners need not engage judgments, nor need they associate any abstract cognition(s) to the sounds. They merely associate sound patterns/sequences with locations within word structures. For example, accurate repetition of “gr-” and “fr-” in “the green frog jumps,” surfaces consequent to knowledge of obstruent-liquid in initial word position for beginning learners, rather than on any symbolically based word meaning. More advanced L2s, in contrast, employ perceptual judgments to chunk semantically and to substitute meaningful alternatives. To illustrate, a substitution might consist of – “the green frog hops.” The fact that these learners substitute, as well as recall, greater numbers of chunks demonstrates reliance on insightful, deliberate word-finding skills. Still, their word-finding skills are not indicative of conscious or planned approaches, but of a “flash” of retroductive reasoning, infused with a view of the “totality of the object.” In this application, the lexical meanings of the particular EI stimulus sentence (both separately and as an aggregate) constitute larger and more meaningful chunked units. “The green frog jumps,” is processed in WM as two constituents: “the green frog” and “jumps.” This higher level of semiotic reasoning (that frogs can jump, and that they do not jump anything) demonstrates more effective WM processing. In short, these relatively advanced L2 learners would not have the means to insert a word substitution in WM without processing (by means of higher-level chunking) the totality of the sentence.

The upshot is that the kind of semiotic processing selected by learners of different language competency levels governs how it is that they chunk in WM, which in turn, determines the number and size of the units needed to process the sentence. If the number of units exhausts WM limitations for that proficiency group, some units are not processed or recalled, and cannot easily be refreshed (as is the case for beginners’ processing at the sound level). When the number of units is reduced via chunking on a more semantic level, fitting the necessary components of the stimulus sentence into WM for processing is more successful, as is the case for advanced L2s (West 2012: 217). Furthermore, processing on the sound level entails associating the shape, location, and sequence of phonemes to similar previous sense data, and hence is indexical and iconic in nature. It is indexical by virtue of the emphasis on the spatial relations between/among the sounds and their aggregates. The flashbulb-like component of processing here is iconic, in that a mere image/replica of sound patterns within the stimulus sentence is matched to identical sound traces of earlier sense impressions (of previous sound aggregates), whereas semantic chunking in WM draws upon higher semiotic representations – symbolic ones. The level of semiotic representation obviously affects the quality (and perhaps the quantity) of linguistic processing in WM.

The quantity of information which can fit in WM increases with L2 competency, primarily as a consequence of the underlying semiotic representation; and increases in quantity of information directly enhance the quality of processing. Rationale resides in the claim that whereas remembering the location of sound units perpetuates recall of other neighboring sound stimuli, memory of semantic features already associated with words within the EI prompt facilitates recall of the internal meaning intrinsic to the message. More specifically, processing which rests on indexical or iconic representations results in chaining (stringing phonemes sequentially), while processing supported by symbolic representations (especially episodic in nature) gives rise to hierarchically organized chunks in WM – those driven by encapsulated meanings within lexical entries. To illustrate this differential, with underlying indexical and iconic representations, L2 processing in WM is characterized by retrospective memory of only seven or fewer units, while processes motivated by symbolic representations are constructed upon retroductive and prospective episodic memories and permit greater units of information within chunks and greater numbers of chunks (approximately eleven) to enter WM (West 2012: 216–217). The latter materializes consequent to the propensity for associating propositional assumptions with the stimulus sentences, which, in turn, increases the value of each chunk for recall of neighboring units from the stimulus sentences. This latter form of WM processing is characterized by more complex mental networks, which Baddeley (2007: 148) refers to as “dynamic binding,” as opposed to “static binding” (for further discussion of binding in WM as applied to issues of semiosis, cf. Chapter 3 of my forthcoming book).

The upshot of these distinctive representational approaches (which stimulate different levels of processing: static vs. dynamic binding) is the quality of the interpretants which hold between representations and their objects. Any meaning/effects associated with phonological sign units are context driven and context bound. The signs either lack an interpretant (in the case of indexical representations) or they constitute reflections of already heard sound units (as in iconic signs). The former possibility (indexical signs) may well lack an interpretant if the only meaning/effect of the phonological unit is memory of its placement within the EI string, because location alone has merely a circumstantial effect. Even processes driven by iconic representations perpetuate context-bound construction – they replicate mere images of sound data. In fact, Short (2007: 90) characterizes indexical signs largely as “irreducible”; consequently, their signs do not capture any meaning other than co-occurrence or place sharing. Conversely, chunks which are processed symbolically are associated with interpretants which are reducible (Short 2007: 90). Atkin (2005: 178–181) refers to this interpretant-based distinction as reagentive – interpretants, then, associated with degenerate signs (especially indexes) are absent, while those connected with genuine signs (often symbols) serve a reagentive function, namely, to measure the effect(s) of other components within the compound. In other words, symbolic signs are associated with meaning/effects beyond the immediate space and time of the EI utterance, and often a general meaning is applied to similar signs. To illustrate, the interpretant of “recognize” can be nullified if it is processed on the phonological level as “re,” in that it often occurs at the outset of words. Alternatively, on the morphological/semantic level, “re-” is associated with “cognizing again.” In this way the latter exacts fewer chunks and has an interpretant bearing a more generalizable, less context-bound effect.

5 Conclusion

It is unequivocal, then, that the oral EI approach is invaluable as an empirical tool to measure semiosis in ontogeny. It represents a well-crafted testing procedure which can measure qualitative differences in WM processing and ascribe such differences to underlying semiotic competencies. EI responses can indicate the nature of the interpretants which hold between sign and object by tapping how meanings are processed in WM. Proclivity of word substitutions over deletions, together with the degree of boundedness of linguistic form to contextual features, illustrates striking distinctions in sign use, especially with respect to the quality of interpretants. Those interpretants which express a proposition enhance processing, while interpretants which are simply diagrammatic inhibit general meaning-building and decontextualization. Learners who process linguistic signs at the diagrammatic level (ordinarily of lower L2 proficiency) focus attentional resources on the location and reproduction of contextualized phonemic and syllabic sequences, and they exact little hope of meaningful recall, while those processing on the propositional level attend to semantic, morphemic, and even conceptual world knowledge – allowing symbolic meaning to have a regenerative effect on how neighboring words are processed.

The interpretants of learners of the latter group (of higher proficiency) process in a qualitatively more efficient way, given means to draw upon symbolic modes of representation. Conversely, less equipped learners process smaller, more verbatim units, namely, acoustic shape and syllabic contours driven by interpretants of indexical and iconic signs. By comparing the quality of interpretants between the groups, the influence of semiotic advances on L2 processing can be measured. EI responses can determine changes accorded to the sign and how such semiotic changes contribute to enhanced processing in WM. Semiotic use differences illustrate the efficacy of EI in measuring the shift from diagrammatic interpretants in Secondness and Firstness to interpretants infused with Thirdness. These semiotic shifts, in turn, serve as the foundation for qualitative advances in language processing.

EI’s validity and reliability to measure semiotic processing advances is unrivaled by other methods. More particularly, EI testing methods highlight the pivotal influence of semiotic modes of thinking upon processing, storage, and recall of information.