Externalization of the mind in the meaning-making process: An integrated semiotic and cognitive science perspective

1 The dynamics of external and internal semiotic representations

As the research in cognitive science has shown, there are two types of representations, external and internal:

1. External representations are formed by external elements that represent (via reification) ideas and issues that do not have a “natural home” in the brain or that are already stored within it;

2. In terms of neurological activity patterns, internalized representations are re-projections, or recapitulations (learning), of external representations. Through the neurological activity of transformation and integration, they may occasionally be altered internally, just like external objects, and they can also originate new internal reconstructed representations.

This perspective suggests that mental representations are correlated to neurological structures with their electrical activity (neurons firing and providing electrical inputs to other neurons) and chemical functioning (neurotransmitters and hormones). This makes it possible to examine cognition and a notable part of semiotic activities from a neurological perspective. For instance, solving an explanatory problem may be seen as a process where a brain structure that represents an explanatory goal gives rise to another neural structure that serves as a hypothesis for the answer.

This procedure clarifies why humans appear to carry out different forms of information processes, including those requiring representations as:

1. (I Level) patterns of neural activation that emerge from the body-environment interaction, and appropriately molded by the individual history and evolution (e.g., according to Clark (2003), human brain functions primarily as a tool for pattern recognition, pattern completion, and pattern manipulation);

2. (II Level) derived combinatorial syntax and semantics as dynamically shaped by the different external representations and reasoning devices found or constructed in the environment (e.g., geometrical diagrams); they are contingently represented neurologically as patterns of neural activations that sometimes tend to become stabilized structures and to fix, becoming a permanent part of the I Level above.

The I Level generates those sensations (they form a sort of “face” we believe the world has), that gives room for the II Level to semiotically mirror the structure of the environment and, most importantly, that can follow the cognitive structure proposed by these external structures. It is evident that the environment has a significant influence on neurological growth, particularly on synaptic and dendritic formation.

Once the fixation is achieved, the patterns of neurological activation can be triggered without direct stimulation from the outside. They are comparable to fixed internal records of external structures in that they can exist even in the absence of seen external structures. Imagine a pattern of neural activation that represents a sign^[1] expressing a triangular shape, this acquired independent external sensed triangular aspects.

The neural activation patterns that make up the I Level Representations are always associated with the II Level Representation, albeit in a different form – that is, as memory rather than as a vivid sensory experience – because they always retain a record of the experience that gave rise to them. Now, the human agent, via neural mechanisms, can retrieve these II Level Representations and use them as internal semiotic representations or use parts of them to construct new internal semiotic representations very different from the ones stored in memory (cf. Gatti and Magnani (2005)). Sterelny (2004) also lists some of the most important results we can obtain thanks to external representations: 1) reduce memory burden, 2) convert challenging cognitive difficulties into simpler perceptual problems, 3) convert challenging perceptual problems into simpler ones, 4) convert challenging learning problems into simpler ones, and 5) design workplaces to do activities more quickly and consistently.

The above description is a schematic simple illustration of the interplay between internal and external aspects of cognition. This interplay is, of course, more complex, as cognitive activity depends on our interaction with the world around us. In this perspective, cognition (in the broad sense of the term) is a meaning bodily-neuro-social cognitive processing, as in the so-called EEEE Cognition: Embodied, Embedded, Enactive, Extended (For an analysis of these approaches that consider physical interaction with the world as a form of cognitive activity, cf. Baber 2019).

It is worth noting that the interplay between external and internal aspects, while sensible, might also be reconsidered within Peirce’s semiotic framework. This discussion could also be linked to Peirce’s concepts of the immediate versus dynamic object, as well as different types of interpretants, including the final or ultimate interpretant. Although the point about neural correlates is valid, given a materialistic perspective, Peirce might not have fully accepted this without qualifications, such as his concept of synechism. The qualia arguments (knowledge argument) are notable potential counter examples, highlighting experiences that seem beyond the reach of neural activations alone. Contemporary neuroscience is increasingly integrated with psycho-neuro-pharma-endocrine research, reflecting the interconnected nature of electric, chemical, and other biophysical pathways in explaining cognitive phenomena.

Certainly, in light of synechism, Peirce viewed the mind not only as “extended” but also as fundamentally continuous with all aspects of reality. According to synechism, which posits that matter and mind are not entirely separate and that everything, including the future and the past, is continuous, any remnants of separation must disappear. Similarly, this continuity applies to both instinct and inference. We can speculate that the mind tends to generate accurate hypotheses about the cosmos because it is governed by the same metaphysical principles that regulate the universe. Peirce argued that cognition is not solely linked to the brain but can also be found in the activities of bees, crystals, and other essentially physical objects.

Furthermore, the two fundamental semiotic delegations to the environment (mimetic and creative), mentioned above, constitute the two main categories of external representations involved in the so-called externalization of the mind (also called disembodiment of the mind). It should be noted that what we mean by ‘externalization of the mind’ does not imply that it is a purely unidirectional movement. It goes without saying that there is a constant interaction between external data and neurological mechanisms: external data do not have meaning in isolation; a living being is needed to interpret and reconstruct them in its own way. Thus, interaction lies at the heart of semiosis (cf., for example, Groupe µ 2015).

Mimetic external representations play an essential role in semiotic dynamics: they can occasionally generate new meanings because they reflect ideas and issues that are already represented in the brain and require improvement, resolution, further elaboration, and so forth. The following sections will provide examples of how, in the hybrid interaction between human neurological structures and appropriate cognitive environments, mimetic geometric representation can become creative and give rise to new meanings and ideas, resulting in “cognitive niches” that are subsequently appropriately reshaped.^[2]

2 Meaning creation through manipulative abduction

Peirce maintained that all thinking is in signs, and signs can be icons, indices, or symbols, and that all inference is a form of sign activity, including “feeling, image, conception, and other representation” (Peirce 1931–1958, 5.283; for details, cf. Kruijff (2005)), and all “synthetic” types of cognition, to use Kant’s terminology. In this context, it may be argued that model-based processes play a significant role in the creative meaning-making process. Furthermore, a significant portion of the processes involved in creating meaning – not only in science – occurs in the midst of interactions between human neurological structures and outside instruments and things that have been delegated cognitive and/or epistemic roles. Being involved in creating, manifesting, or responding to a sequence of signs and meanings is the same as being occupied in being a mind: we can say that the human brain is a semiotic brain. Acknowledging that a series of material physical occurrences constitute a series of signs indicates the presence of a mind (or of a set of minds): “[…] all thinking is dialogic in form” (Peirce 1931–1958, 6.338), both at the intrasubjective and intersubjective level, so that we see ourselves exactly as others see us, or see them exactly as they see themselves, and we see ourselves through our own speech and other interpretable behaviors, just as others see us and themselves in the same way, in the commonality of the whole process (Brent 2000).

From this perspective, minds are material like neurological structures insofar as they are composed of interwoven internal and external semiotic processes, as we shall better clarify in the parts that follow: it is evident that Peirce foresaw the extended mind theory, saying that “[…] the psychologists undertake to locate various mental powers in the brain; and above all consider it quite certain that the faculty of language resides in a certain lobe; but we believe it comes decidedly nearer the truth (though not really true) that language resides in the tongue. In my opinion it is much more true that the thoughts of a living writer are in any printed copy of his book than they are in his brain” (Peirce 1931–1958, 7.364).

Conversely, manipulative abduction happens when a variety of outside entities, which are typically inert from a semiotic perspective, can be changed into something similar to what has been called “epistemic mediators” in the context of scientific reasoning (see Magnani (2001)) that give rise to new signs, new opportunities for interpretants, and new interpretations. We may use the idea of manipulative abduction to cognitively explain this externalization process. It occurs when we are thinking through doing and not only, in a pragmatic sense, about doing. It happens, for instance, when we build and work with an externally suitably realized icon (such as a triangle) to increase geometrical knowledge and look for additional aspects of it.

It describes an extra-theoretical activity with the goal of constructing communicable narratives of novel experiences in order to incorporate them into systems of experimental and linguistic (semantic) practices that have already been established. This type of tangible manipulative thinking is described by Gooding (1990) when he describes the function of so-called “construals” in the case of science, that embody tacit inferences in processes that in general are apparatus- and machine-based.^[3]

It is challenging to compile a list of invariant behaviors that accurately characterize manipulative abduction, for example in science.^[4] Even if abduction functions in accordance with musement’s aesthetic process, as Peirce states: “a certain agreeable occupation of the mind” which must follow “the very law of liberty” (Peirce 1931–1958, 6.458), the skilled manipulation of entities in a highly semiotically constrained experimental environment certainly derives from the application of old and new templates of behavior that exhibit some regularities.

The process of creating construals is very conjectural and not always or instantly explanatory: these templates are hypotheses of behavior (creative or cognitively preexisting in the scientist’s mind-body system, and sometimes applied) that abductively permit a type of epistemic activity. Therefore, in order to complete the most intriguing creative cognitive feats of manipulative abduction, some action and manipulation templates can be selected from the collection of ones that are already available and pre-stored, while others must be created from scratch.

3 Using external semiotic anchors to dynamically manipulate meanings

Physical alterations to the environment should have a cognitive significance if the structures of our surroundings have such an important influence on our semiotic representations and, consequently, our cognitive processes.

Numerous writers have highlighted the part that physical acts may play in cognitive processes. In this regard, activities are divided into two groups by Kirsh and Maglio (1994): pragmatic actions and epistemic actions. The acts an agent takes in the environment to physically approach a goal are known as pragmatic actions. In this instance, the action alters the environment to give it a configuration that aids the agent in achieving a goal that is seen as physical, or as a desirable condition of affairs. The acts an agent does to relieve the mind of a cognitive burden or to extract knowledge that is concealed or extremely difficult to gain just through internal processing are known as epistemic actions.

In this section, we wish to pay particular attention to the connection that may exist between representations and environmental alterations. Specifically, we aim at investigating whether external manipulations may be seen as a method of creating external semiotic representations that mirror internal ones^[5] or that are new and creative. A manipulative action carried out on the environment with the intention of creating a configuration of signs and meanings conveying pertinent information can be regarded as a semio-cognitive process, and the elements it creates can be regarded as components of an external representation. Any manipulation of the environment intended to create external configurations that qualify as representations is what we refer to as cognitive manipulation.

Figure 1 presents a diagrammatic representation of a cognitive manipulation, commonly referred to as a demonstration or a proof, taken from the field of elementary geometry.

Figure 1:

Diagrammatic demonstration showing that the sum of the internal angles of any triangle is 180 degrees (Euclid’s Proposition XXXII, Book I).

Here, a simple triangle manipulation generates fundamental semiotic information about the internal angles of a triangle anchoring, as a causal relationship, the possibility of obtaining new meanings.

Imagine a child who must “demonstrate” what we call a “theorem” (the one concerning the sum of the internal angle of a triangle related to the example just described) of elementary geometry. The child does not have to demonstrate this theorem for the first time, in the sense that it has already been discovered historically and reported with a deductive proof in every textbook of Euclidean geometry. However, excluding the case when he passively repeats by rote, he can achieve this demonstration by using a series of elementary constructions, based on the appropriate use of basic geometric concepts which are already available to him. In case we are referring to the discovery by pre-Euclidean geometers of this property of the interior angles of any triangle, the example just illustrated instead exemplifies a case of creative discovery, of new geometric meanings, obtained thanks to a complicated semio-cognitive process, which is abductive.

The idea of manipulative abduction helps to explain the full process by which an agent arrives at a physical action that may be considered cognitive manipulation. According to this viewpoint, manipulative abduction is a particular case of cognitive manipulation in which an agent chooses or invents an action that structures the environment in a way that can provide new information that would otherwise be unavailable and that is specifically used to infer explanatory hypotheses when faced with an external situation from which it is difficult or impossible to extract new meaningful features of an object.

By using external representations in place of the internal ones, the semiotic result is obtained in this manner.^[6] Here, the action serves an epistemic function rather than a purely pragmatic one, for example relevant to abductive reasoning. Additionally, the procedure shows a synthesis between a constructive procedure of motor nature (putting the new segment end to end parallel to one side in the externally represented given triangle), followed by a sensory process, specifically the “visual” one (calculation of the sizes of the now clearly – externally – “seen” angles).

Following Thom’s perspective, we assert that this example from ancient Greek geometry already embodies the core ideas of the scientific method, which is “[…] replacing a non-local operation (for example, taking the intersection of two lines in a plane) by a verbal description the formal analysis of which became the demonstration that it was virtually autonomous, that is, able to be rendered independent of the non-local intuitive approaches which described it” (1980: 135). The axiomatic method combined with the employment of literary symbols, which are empty of sense, realizes the localization of the non-local intuition of the plane (and, more generally, of space).

4 Geometric construction as an example of dynamic creative semiosis

Geometric and mathematical reasoning, as articulated by Peirce, “[…] consists in constructing a diagram according to a general precept, in observing certain relations between parts of that diagram not explicitly required by the precept, showing that these relations will hold for all such diagrams, and in formulating this conclusion in general terms. All valid necessary reasoning is in fact thus diagrammatic” (Peirce 1931–1958, 1.54). This passage blatantly alludes to a circumstance similar to the one we described in the preceding section. Peirce also referred to this type of reasoning as “theorematic”, that it is a type of deduction required to deduce important theorems, which he termed “Necessary Deduction”:^[7] it “[…] is one which, having represented the conditions of the conclusion in a diagram, performs an ingenious experiment upon the diagram, and by observation of the diagram, so modified, ascertains the truth of the conclusion” (1931–1958, 2.267). The procedure is executed with the help of “[…] imagination upon the image of the premiss in order from the result of such experiment to make corollarial deductions to the truth of the conclusion” (Peirce 1976, IV, p. 38). The “corollarial” reasoning is mechanical (Peirce thinks it can be performed by a “logical machine”) and not creative: “A Corollarial Deduction is one which represents the condition of the conclusion in a diagram and finds from the observation of this diagram, as it is, the truth of the conclusion” (Peirce 1931–1958, 2.267; cf. also Hoffmann (1999)).

The goal of theorematic reasoning, to put it briefly, is to change an issue into something intriguing and potentially novel by developing an unnoticed point of view. Theorematic deduction is also related to mathematical proofs of “new” theorems: in this last case it cannot be called anymore “deduction”: to acknowledge this problem Peirce furnished a complicated argumentation on what he calls “theoric” aspects of theorematic reasoning, which we do not have time to go into here.^[8]

Hence, in our perspective, it is simple to understand theorematical deduction in terms of manipulative abduction. As we have seen, manipulative abduction is a type of abduction that primarily uses models to take advantage of external models – diagrams in the present case – endowed with delegated (and often implicit) semio-cognitive functions and characteristics:

1. The model (diagram) is external and the strategy that structures the manipulations is unknown a priori;

2. The outcome achieved is new (if we, for example, refer to the early creators of pre-Euclidean geometry) and adds properties not contained before in the concept.^[9]

A key component of theorematical reasoning is iconicity. In the wake of Kant, Peirce argues that “[…] philosophical reasoning is reasoning with words; while theorematic reasoning, or mathematical reasoning is reasoning with specially constructed schemata” (Peirce 1931–1958, 4.233). In addition, he uses the terms “schematic” and “diagrammatic” as synonyms, connecting his ideas to the Kantian tradition in which schemata mediate between intellect and phenomena.

We know that the type of reasoned inference involved in creative abduction goes beyond probabilistic reasoning, which merely increases or decreases the probability of the conclusion, as well as beyond the relationship that exists between premises and conclusions in valid deductions, where the truth of the premises assures the truth of the conclusions. Rather, we must see creative abduction as the result of using heuristic procedures, which encompass a variety of good and bad inferential behaviors, rather than just following rules mechanically. The only way to ensure the acquisition of “new” truths is through these heuristic techniques. Peirce’s sophisticated perspective on creative abduction as a type of inference, as seen above, also tends to emphasize the strategic nature of reasoning.

Deductive reasoning, according to numerous researchers in the fields of philosophy, logic, and cognitive science, also entails the heuristic – and so expert – application of logical rules while upholding their truth-preserving qualities. This is because the rules are applied in a way that makes it possible to suggest one course of action over another. Furthermore, iconicity plays a key role in the model-based abductive stages that are frequently used to carry out the heuristic procedures of deductive reasoning. After all, also strictly symbolic (and not diagrammatic) proofs typical of deductive reasoning (for example in mathematics) possess an iconic character and can be for example considered more or less simple and elegant. Moreover, some of them can be so poorly constructed – due to the adoption of inappropriate heuristics – that they are unable to arrive at the desired proof of the theorem in question. This means that also in the case of the building of deductive proofs, we face abductive steps related to the capability to choose the good heuristics.

We have seen that the most common example of manipulative creative abduction is the usual experience people have of solving problems in geometry in a model-based way trying to devise proofs using diagrams and illustrations: of course the attribute of creativity we give to abduction in this case does not mean that it has never been performed before by anyone or that it is original in the history of some knowledge (they actually are cases of Peircean corollarial deduction).^[10]

The epistemological situation of our geometrical example can be summarized as follows:

1. The process of inference is a type of abduction that is both manipulative and model-based, mostly visual, and it possesses an explanatory quality. This means that the abduced mirror diagrams related to a triangle furnish an external visual (and iconic) explanation/description of our starting internal representation of it;

2. The external image and the construction that follows provide the opportunity for an additional multimodal and distributed abductive step, now primarily non-explanatory and instrumental, based on both internal and external representations and on both visual and sentential aspects. This additional abductive procedure allows for the derivation of the new “indisputable” result/theorem on the sum of a triangle’s interior angles.

5 What cognitive consequences?

For Peirce as well, more than a century before the novel concepts came from the field of distributed cognition, the two facets are combined in the semiotic view, a view that goes beyond the schematism of the Kantian approach. Diagrams are icons that manifest in a material and semiotic form in an external environment endowed with, on the one hand, constraints related to the specific cognitive delegation executed by human beings, and, on the other hand, the particular intrinsic constraints that depend on the materiality at play.^[11]

This procedure has several intriguing semiotic features that make for a subtle analysis. Imagine that there is a suitable fixed internal record that, at the level of neural activation, arises from the cognitive exploitation of the prior suitable interplay with external structures. For example, it contains an abstract concept with all its features, such as the concept of a triangle. The human agent may now re-embody that concept through neurological mechanisms and physical acts making an external perceivable sign, for instance, directed to the attention of other human or animal senses and neurological structures. As an illustration, the human agent may employ what is known as a symbol (with its conventional character: ABC, for example), but also an icon of relations (a suitable diagram of a triangle), or a hybrid representation that will take advantage of both. In Peircean words:

A representation of an idea is nothing but a sign that calls up another idea. When one mind desires to communicate an idea to another, he embodies his idea by making an outward perceptible image which directly calls up a like idea; and another mind perceiving that image gets a like idea. Two persons may agree upon a conventional sign which shall call up to them an idea it would not call up to anybody else. But in framing the convention they must have resorted to the primitive diagrammatic method of embodying the idea in an outward form, a picture. Remembering what likeness consists in, namely, in the natural attraction of ideas apart from habitual outward associations, we call those signs which stand for their likeness to them icons.

Accordingly, we say that the only way of directly communicating an idea is by means of an icon; and every indirect method of communicating an idea must depend on its establishment upon the use of an icon (Peirce 1966, 787, 26–28).

It is important to highlight that for Peirce a concept “[…] is not properly a conception, because a conception is not an idea at all, but a habit. But the repeated occurrence of a general idea and the experience of its utility, results in the formation or strengthening of that habit which is the conception” (Peirce 1931–1958, 7.498).

Habits are typically thought of as physical states insofar as they are similar to “dispositions”, according to Peirce, as beliefs, cognitive processes, and anticipatory opportunities for action.

In the perspective of the cognitive interplay internal/external, as described above, it is better to see them as forms of interaction with the appropriate circumstances involved in the related action, as Määttänen (2010) stresses. From this vantage point, perception and action are closely linked, simply because they involve varying degrees of interaction with the environment: in action, “our modification of other things is more prominent than their reaction on us” as compared to perception “where their effect on us is overwhelmingly greater than our effect on them” (Peirce 1931-1958, 1.324).

Naturally, what is external to the body need not be external to the process of realizing cognition, which essentially uses sensorimotor representations that are created or re-activated (if stable and developed) during the interactions with the physical environment. That being said, we have to re-emphasize that a significant portion of the cognitive process frequently takes place outside thanks to the suitable materiality endowed with contingent cognitive delegations. In this way, the potential development of a habit is in itself also the institution of new meanings. According to Peirce, acquired habits are meanings in themselves, as a result of interactions with the environment, they are not literally only in the head but also interrelated with motor action, that is embodied. Sensory inputs, that are abductively correlated with a good habit, are associated not only with each other but also with neural mechanisms regulating motor action (see Määttänen 1997, 2010).

Re-embodiment is the process by which internal mental representations are formed (as simple neurological states) and closely linked to motor characteristics on both a neurological and somatic level. In this sense, the world is seen as providing a range of chances to engage in habitual activities, or what Gibson (1979) refers to as affordances: habits of action abductively “reveal” affordances.^[12]

Due to the limitations of the human mind, such as the inability or extreme difficulty of performing internal computations in many problem-solving scenarios, human beings delegate cognitive features to external representations through semiotic attributions. In the previous sections we have seen that, initially, a type of alienation is carried out, and then, a recapitulation is achieved at the neuronal level by internally expressing what was discovered or simply recovered externally. As a result, we only later on execute internal cognitive processes on the data structures that synaptic patterns have “picked up” from the external world, in an analogical way. We may uphold that experiences encountered in the semiotic environment serve as a deeper source for internal representations like many events of meaning creation. Hutchins (2005: 1,575) provides more clarification on this recapitulation process: “[…] when a material structure becomes very familiar, it may be possible to imagine the material structure when it is not present in the environment. It is even possible to imagine systematic transformations applied to such a representation. This happened historically with the development of mathematical and logical symbol systems in our own cultural tradition”.

In this interplay of re-embodiment diagrams that afford certain behaviors as feasible (for instance, a chair affords an opportunity for sitting, air breathing, water swimming, stairs climbing, etc.), the embodied outcome may be seen as the formation of a habit, which is, in a Peircean sense, not just a theoretical outcome but also a form of “know-how”: “We imagine cases, place mental diagrams before our mind’s eye, and multiply these cases, until a habit is formed of expecting what always turns out the case, which has been seen to be the result in all the diagrams. To appeal to such a habit is a very different thing from appealing to any immediate instinct of rationality. That the process of forming a habit of reasoning by the use of diagrams is often performed there is no room for doubt” (1931–1958, 2.170). Finally, what would be the consequence of the description we sketched for this process of externalization of the mind? Mimetic external representations mirror concepts and problems that are already represented in the brain and need to be enhanced, solved, further complicated, etc. Creative external representations are capable of dealing with concepts and problems that are already represented in the brain with the aim of finding something new and so they are fundamental in the processes of meaning formation.^[13] From this vantage point, it is now clear that the mind extends beyond the boundaries of the individual and incorporates elements of the surroundings. It follows that the mind is semiotic by nature (cf. also Tiercelin (1993) who discusses the “semiotization of the mind”).

6 Conclusions

In this article, we tried to make explicit the role of the mind, in conjunction with neurological structures, in the implementation and development of semiosis. This approach, through the analysis of the externalization of the mind, provides a novel interpretation of the semiotic role of external models and representations as anchors of meaning. Following an examination of the relationship between major internal resources and devices, and their dynamic interactions with the externalized materiality, we argued that this connection is crucial, because minds can be thus described in an extended semiotic framework. The method by which the mind externalizes also reveals a new cognitive understanding of the mechanics behind the semiotic development of abductive processes of meaning-making, particularly manipulative abductions that are able to both produce and anchor meanings. In sum, a significant portion of human abductive thinking involves a process akin to reification in the outside world, followed by re-projection and reinterpretation through new configurations of neural networks and chemical processes.