The body says it all: Non-verbal indicators of choice awareness
-
Alexandra Mouratidou
Alexandra Mouratidou is a Doctoral Candidate in Cognitive Semiotics, Division for Cognitive Semiotics, Center for Languages and Literature at Lund University, Sweden. She was a public relations officer in the board of the International Association for Cognitive Semiotics (IACS), from 2022 to 2024, and a co-organiser of IACS-5. Her research interests lie in polysemiotic communication, especially when choice-making and choice manipulations are concerned, as well as in phenomenology and consciousness. She is assistant editor of
Public Journal of Semiotics .
,
Jordan Zlatev
Jordan Zlatev is Professor of General Linguistics and Director of Research for the Division of Cognitive Semiotics at Lund University, Sweden. He was the first president of the Swedish (currently: Scandinavian) Association for Language and Cognition (SALC), from 2006 to 2009, and of the International Association for Cognitive Semiotics (IACS), from 2013 to 2014, and he is currently a board member and IACS-5 organizer. He is the author of
Situated Embodiment: Studies in the Emergence of Spatial Meaning (1997), and of over 100 articles in academic journals and anthologies. He co-editedThe Shared Mind: Perspective on Intersubjectivity (2008), Moving Ourselves, Moving Others: Motion and Emotion in Intersubjectivity, Consciousness, and Language (2012) andMeaning, Mind and Communication: Explorations in Cognitive Semiotics (2016). His current research focuses on polysemiotic communication, and more generally on the nature of language in relation to other semiotic systems like gesture and depiction. His approach to cognitive semiotics is strongly influenced by phenomenology, the philosophy and methodology of lived experience. He is editor-in-chief ofPublic Journal of Semiotics .Joost van de Weijer is employed at Lund University Humanities Lab in Sweden. He collaborates with various researchers on the topics of animal communication, prosody and factivity, and motivation in highschool students to learn a foreign language. He teaches courses in psycholinguistics and in the statistical analysis of experimental data, provides methodological support for ongoing research projects within various areas of language and speech, and on psychophysiological measurement of behavioural and cognitive processes. He is editorial secretary for the journal
Studia Linguistica , and is member of the editorial board of the journalBilingualism, Language and Cognition.
Abstract
Recent cognitive science research suggests that occasional “blindness” to choice manipulations indicates a lack of awareness in choice making. This claim is based on participants’ tendency not to detect choice manipulations and the similarity between their justifications for choices they made and those they were tricked into believing they made. Using a cognitive-semiotic framework, we argue that such conclusions underestimate the embodied, intersubjective nature of human meaning-making. We support this by investigating choice awareness beyond language to include non-verbal behavior. Forty-one participants were asked to choose from pairs of photographs of human faces the one they found most attractive and then to justify their choices, without knowing that for some of the trials they were asked to justify a choice that they had not made. Verbal responses were categorized as (i) non-manipulated, (ii) detected manipulated, and (iii) undetected manipulated trials. Bodily expressions, assessed using five different Categories of Bodily Expression (CBE): Adaptors, Torso, Head, Face and Hand expressions, revealed differences in: (a) duration, (b) rates of occurrence and (c) variety of the CBEs across trials. Thus, even when manipulations were not verbally detected, participants took longer to assess choices, showed increased bodily expressions, and engaged more body parts in undetected manipulations compared to non-manipulated choice trials. This suggests a degree of awareness to the choice manipulation, even if pre-reflective, manifested in participants’ bodily expressions.
1 Introduction
Research in cognitive science influenced by physicalism often shares the premise of the illusory nature of conscious will, focusing its scientific inquiries on the mechanisms that underlie our purportedly false experiences of agency (Bargh and Ferguson 2000; Dennett 1991, 1996; Johansson et al. 2013; Libet 2005; Wegner 2018). For instance, our occasional tendency to justify a choice that we were tricked into believing we made has been treated in so-called choice blindness experiments (e.g., Johansson et al. 2005) as evidence of our unreliability as conscious agents. Such findings have been seen as supportive of a body of research, prevalent over the last half of the century, on the limits and fallibility of introspective reports and conscious experience. The reliability and use of first-person reports remains, however, a focal point of an ongoing debate within cognitive science (Jack and Roepstorff 2003, 2004; Overgaard 2006; Petitmengin 2009a, 2011; Varela and Shear 1999).
An alternative approach, based in cognitive semiotics, posed the question of whether it is justified to draw inferences about our lack of choice awareness just because participants in “choice blindness” experiments end up verbally justifying choices they had not made (Mouratidou 2020; Mouratidou et al. 2022). A different, and arguably more adequate term was proposed, manipulation blindness, since what such observations can at most show is unawareness concerning the switch of one item with another, not of the choice itself. Mouratidou (2020) explored potential factors influencing manipulation detection in a choice experiment, focusing on memory, affectivity, and consequence.[1] The first two of these factors had significant effects, as participants detected most of the choice manipulations when they remembered their original choice. In the experiment, 43 Greek participants were asked to choose from 20 pairs of photographs of human faces and from 20 pairs of abstract figures the ones they preferred (see Figure 1A and B). After going through all the pairs, they were asked to confirm whether a face or figure was the one they had chosen (C). Finally, they were asked to justify the choice they made in B. Without the participants’ knowledge, four face trials and four abstract figure trials were manipulated by asking participants to justify why they had preferred an item which they actually had not (D).
The choice manipulation procedure. (A) The participant is presented with two alternatives and, (B) asked to make a choice; (C) then asked if they had indeed chosen this, and (D) finally asked to justify their choices, where in a number of trials (manipulations) the non-chosen alternative in B was presented as their choice.
Further analysis of the verbal justifications of both actual (i.e., non-manipulated choice trials) and manipulated choice trials showed differences in terms of choice investment: the “mattering” or meaningfulness of a choice in an act of choice making (Mouratidou et al. 2022: 606). Acknowledging the relationship between pre-verbal consciousness and language, a distinction was made between pre-verbal and verbal choice investment. Participants’ verbal responses were analyzed along with a set of semantic categories, formulating a matrix of eleven markers of choice investment. A greater degree of investment was predicted when motivating (a) choices of faces than figures, due to their different status in terms of affectivity; (b) manipulated than actual choices, since participants’ conflicting experience in such choice trials would influence them to be more invested in explaining a choice that was not their own than when it was (e.g., by explaining the reasons for not making such a choice when the manipulation was detected, or by reconsidering it anew through a comparison of both alternatives); and (c) detected than non-detected manipulations, due to the assumption that choices that mattered more for participants would be more often detected than those that did not (see Section 3.3). These predictions were confirmed, but with various strengths, leading to conclusions for both consilience and differences between pre-verbal choice investment and the corresponding verbal motivations of the choices made, and thus for (degrees of) conscious awareness of choice making.[2]
The discipline of cognitive semiotics, within which these investigations of choice making and manipulation detection have been carried out, acknowledges correspondences – but not identity between non-verbal experience and language, in line with research in phenomenology (Merleau-Ponty 1962; Sokolowski 2000). By integrating concepts and methods from linguistics, cognitive science and semiotics (Konderak 2018; Sonesson 2009; Zlatev 2015), cognitive semiotics focuses on the study of aspects of meaning and their interrelations, with consciousness (intentionality, in the broad phenomenological sense of the term) and meaning-making as two closely interrelated aspects of our interaction with the world and others (Zlatev 2018; Zlatev and Konderak 2022).
One of the principles of cognitive semiotics is that of phenomenological triangulation (e.g., Mendoza-Collazos 2022; Pielli and Zlatev 2020; Zlatev 2009; Zlatev and Mouratidou 2024), where any given phenomenon is methodologically approached by a plurality of perspectives. Starting with a first-person perspective and by reflecting on our own experiences on the matter, the phenomenon is then studied from an intersubjective, second-person perspective (e.g., through interviews, observations, etc.), and finally (and optionally) also through a more detached, third-person perspective (e.g., systematic coding, physiological measurements, etc.). What such phenomenological triangulation implies is that third-person methods are never sufficient on their own, nor, strictly speaking, necessary. Rather, they must be preceded by, and complemented with, first-person and second-person methods. Thus, no matter the ontological status of the phenomenon, the perspectives of the first and second-person are always at play, even if backgrounded, in line with the phenomenological dictum of the primacy of consciousness as the means through which everything is given to us, including scientific knowledge.
Another methodological principle of cognitive semiotics is that of the conceptual-empirical loop (e.g., Devylder and Zlatev 2020; Mendoza-Collazos 2022; Stampoulidis et al. 2019; Zlatev 2015), postulating the need to integrate philosophical and empirical investigations. The dialogue between “what” and “how” questions involves numerous iterations of the loop, where theoretical constructs are formulated, and further operationalized as appropriate for the empirical study in question (see Section 3.2). Thus, both of these tools are required for investigations that are committed to remain true to the original phenomenon and to elucidate it further.
One key concept of cognitive semiotics is that of polysemiosis: communicating by the simultaneous use of two or more semiotic systems, which could be sign systems like language, gesture and depiction, or signal systems like bodily postures and spontaneous facial expressions (Zlatev et al. 2020). Such semiotic systems involve different degrees of awareness both from the producer’s and the interpreter’s side. A gap in the studies mentioned above (i.e., Mouratidou 2020; Mouratidou et al. 2022) was that we did not consider any other semiotic system but language as indicative of choice awareness and investment. To address this, we here propose a novel approach to the investigation of choice awareness and manipulation detection. In the present context, this implies the analysis of the polysemiotic utterances, consisting of both verbal, and bodily signs and signals. The objectives of the investigation are thus (a) to go beyond language into other bodily sign systems (e.g., hand gestures) and signal systems (e.g., adaptors, i.e., movements that satisfy personal needs and respond to environmental triggers) that involve different degrees of awareness both from the producer’s and the interpreter’s side, and (b) to expand our understanding of the interaction between different semiotic systems in two conditions: manipulated choices (when participants are asked to justify choices they did not make), and actual choices (when participants are asked to justify non-manipulated choices, i.e., the choices they actually made).
The structure of the remainder of the paper is as follows. Section 2 offers the theoretical background concerning polysemiotic communication, with focus on language, gestures and adaptors. Section 3 describes the methodology. The results and discussion are presented in Section 4, followed by the conclusions in Section 5.
2 Theoretical framework
Human communication is in most circumstances polysemiotic, relying on the interaction of different semiotic systems, as well as multimodal, relying on the combination of two or more sensory modalities like vision and hearing. These two dimensions are in fact orthogonal, even if they are often conflated in the literature (for a discussion, see Green 2014; Zlatev 2019; Zlatev et al. 2023). Semiotic systems can be either sign systems, including (most of) language, gesture and depiction or signal systems like postures and adaptors (see below). Both kinds consist of pairings of expression and meaning. However, only signs can denote things, properties or events, while the meanings of signals are not denotational (Zlatev et al. 2020). Thus, most (but not all) words and hand gestures qualify as signs. Various kinds of bodily movements (e.g., scratching, yawning, etc.) that do not denote, and in general are more constrained in what they can express, function as signals. Finally, other bodily expressions involving the head and face have somewhat intermediary status: if produced with communicative intent and/or denotational meaning they count as gestural signs (Andrén 2010; Kendon 2004). Else, they count as signals, as we explicate below.
The sign system of language is uniquely characterized by double articulation: phonemes or graphemes combine systematically to form meaningful morphemes, and by high degrees of conventionality (Clark 1996) and normativity (Itkonen 2003) of simple and complex linguistic signs (e.g., sentences). The syntactic relations are characterized by compositionality, where the meaning of a composite sign is built up (at least in part) from the meanings of its constituent signs and the rules for combining these, though not in a mechanical “building block” manner, given the context-sensitivity of linguistic meaning (Goldberg 1995, 2006; Zlatev 1997).
When bodily expressions qualify as signs, they have denotational meaning. This can be categorized by the predominant, but not exclusive, type of semiotic ground (Sonesson 2010) between expression and denoted object (Zlatev et al. 2020): iconic (resembling the object), indexical/deictic (bringing the object to attention), and symbolic (denoting the object on the basis of a socially shared convention). In addition, such expressions may also have non-denotational meaning, expressing emphasis, modality (uncertainty, rejection, etc.), and affect (surprise, repulsion, etc.) (Ekman and Friesen 1969; Kendon 2004; Streeck 2009). In such cases the non-denotational meaning contains information about the speaker’s attitude.
The ordinary language term “gesture” is highly ambiguous, but here we understand it in the sense of one of the pioneers of the field, Kendon (2004: 14), as denoting bodily “movements that (…) manifest deliberate expressiveness to an obvious degree”. We can usefully complement this with the proposal of Andrén (2010) that the semiotic system of gesture can be distinguished from bodily signals and practical actions on the basis of one or both of the two criteria: sign function or overt communicative intent. The border between gestures and non-deliberate bodily signals is fluid, especially for articulators like the head, face, and body trunk.
An intermediary phenomenon with respect to conventionality are so-called recurrent gestures: a type of co-speech gestures that are to some extent conventional in the sense of commonly repeated, and thus to a degree socially shared. Although recurrent gestures are context dependent, they have relatively stable expression-object relations, albeit not as conventional as emblems or words (Bressem and Müller 2014; Kendon 2004; Ladewig 2014a; Muller 2017). While emblems (like the OK-gesture) are apt in replacing speech, recurrent gestures are usually part of a polysemiotic utterance and may have both denotational and non-denotational meaning. A good example is that of the shrug, which according to Debras (2017), qualifies as a recurrent gesture complex – although it has been customarily classified as an emblem (Kendon 1981; Efron and Van Veen 1972; Ekman and Friesen 1969, 1974; Ekman et al. 1976). What characterizes the shrug is a form of “compositionality” or in other terms: compound enactment (Streeck 2009), shrug complex (Morris 1994) or shrugging composite (Given 1977). The point is that the different bodily “components” of the gesture can be autonomous, but still interdependent.[3] This extends to other emblems, usually performed with a hand gesture and a facial expression (e.g., Calbris 1990; Poggi 2002, etc.).
A class of sensory-motor movements that mostly fall on the signal side of the sign/signal divide are affect displays, expressing spontaneous information about the nature and the intensity of affect like surprise, indifference, or repulsion (Ekman and Friesen 1969). Another type of bodily signals are adaptors, which function as part of a total adaptive system (e.g., to satisfy bodily needs, manage emotions, learn instrumental activities, etc.). Unlike gestures, they are responses to environmental triggers that are not intended to communicate a message, and (generally) performed without (focal) awareness (Ekman and Friesen 1969: 84). Adaptors have been categorized differently according to their form and function (Ekman and Friesen 1969; Freedman 1972), but a major distinction is whether they are geared towards one’s body (self-adaptors) or an external object. It has been claimed that in communicative settings where ambiguous, interfering, and conflicting cues are involved, “the speaker is likely to turn to soothing, grooming, rubbing, or scratching, as ways of confirming the boundaries of the self at the time when the sharing of thoughts is also required” (Freedman 1977: 114). Based on psychotherapy and forensic research (e.g., Ekman 1991, 1999, 2009; Ekman and Friesen 1969, 1976; Ekman et al. 1991), adaptors have been analyzed as indicators to deception “leakage” (i.e., the unintentional betrayal of the truth through demeanor) or more generally of high emotional and cognitive load (Ekman 2009).
Both affect displays and adaptors have been observed to be sometimes “discordant” with the message people explicitly convey - through speech and possibly also (some) gestures – in deception interactions. This is consistent with the sign versus signal distinction, given that signs require at least potential consciousness of the denotational relation between expression and intentional object, while signals do not (Zlatev et al. 2020).[4]
In sum, gestures that usually involve different parts of the body (such as a shrug or a nod) can be regarded as both denotational, since their form and meaning remain relatively fixed in different contexts and for different speakers, as well as non-denotational, reflecting participants’ attitude (e.g., certainty, surprise, etc.). Other bodily expressions, which can have relatively stable meaning but also differ in form across contexts and speakers, have only non-denotational meaning (e.g., lips downwards and eyebrows lifted, puckering the lips when saying hmmm). Finally, simple movements usually performed without speech (e.g., participants licking their lips) lack both denotational and non-denotational meaning. Thus, signs always have denotational and sometimes also non-denotational meaning. Signals can at most have non-denotational meaning, even if some of these expressions can have some degree of conventionality. Finally, adaptors and purely practical actions, mostly connected with the head and the upper body, serve practical aims rather than purposes of communicative expressions. These conceptual distinctions have been implemented in the coding system described in the following section.
3 Methods
3.1 Participants and procedure
To explore manifestations of choice awareness in different semiotic systems, we adapted the choice manipulation experiment used by Mouratidou (2020) that was briefly described in the introduction. 41 Greek participants (roughly balanced for gender) were shown 20 pairs of photographs of human faces (10 pairs depicting female faces, 10 pairs depicting male faces) and chose the person they found most attractive (see A and B in Figure 1). After that, the participants were presented with the initial pairs and were asked to justify their choice, without knowing that four trials were manipulated by deliberately asking them to justify their non-preferred choice (see D in Figure 1). The verbal responses to these manipulated trials were categorized according to the type of detection and type of response, following Mouratidou’s (2020) categorization of participants’ response patterns, into three levels of detection: (a) clear, when participants rejected the manipulated item as their own, (b) possible, when participants either questioned or did not justify the presented item as their choice but did so for its alternative (i.e., their actual choice), and (c) none, when they accepted the manipulated choice but without offering a justification, when they evaluated it as of equal weight with the alternative, or when they justified it as their own. Thus, each level of manipulation detection included one or more patterns that characterized participants’ responses (i.e., denial, uncertainty, ignorance, indifference, and acceptance) given for manipulated choices. All verbal interactions took place in Greek, the native language of the first author, who led the experiment.
3.2 Coding and analysis
Developing a coding system for polysemiotic communication is a complex methodological process. While the literature offers information about annotation schemes, detailed descriptions of the process of their development are lacking. As Stec (2015: 61) points out:
Annotation work is treated as a final product rather than as a process which involves watching the data, creating and piloting an annotation scheme with multiple passes through one’s data, improving that scheme as one makes new observations or encounters unexpected difficulties, implementing it and obtaining (if desired) measures of inter-rater reliability. However much we would like annotation schemes to be “objective” measures (…), they reflect theoretical choices and interpretations at every step of the way.
Our coding system concerns the semiotic systems described in Section 2, implemented in the ELAN video annotation software (Wittenburg et al. 2006). Methodologically, we used the conceptual-empirical loop (see Section 2): we began by distinguishing between different semiotic systems on the basis of phenomenologically grounded theoretical concepts, and established fine-tuned distinctions as we went along. On the empirical side of the conceptual-empirical loop, decisions on which semiotic systems were relevant for the analysis and further operationalizations relied on the specifics of the collected material. The coding system shown in Appendix A (Table A.1) is the result of several iterations of this loop. The collected video-recordings were watched multiple times throughout the coding process and the system was revised as a result of subsequent viewings of the data.
Schematically, the process can be summarized in four major plateaus, with multiple smaller steps in between. First, we developed a coding system based on the cognitive semiotic concepts defined in the previous sections as well as our observations of the data and then used it to code seven participants. Through this testing process, elements that needed to be revised or further considered and adapted became obvious. We noted them down during the coding of each participant and addressed them in subsequent meetings. At this stage, most changes were made with respect to (a) the form of all bodily expressions (see Table A.1), mainly adding or specifying values that were missing; and (b) the structure of the ELAN template. Second, a new template was created, which we used to code the rest of the material. Again, new observations and difficulties occurred and these were noted down. These concerned mostly cases where it was not clear where specific bodily expressions fall, implying that further decisions needed to be made (e.g., to distinguish between when an expression serves only a practical aim or reflects some kind of participant’s attitude). When the first complete round of coding was made, we addressed these issues in multiple meetings among us. Third, we adapted the template accordingly and re-coded the material. Fourth, a final pass was made, checking for consistency across the coded files.
The polysemiotic utterances that were coded had been generated as responses to the question “why did you choose this picture card”, while the participants were presented with the card they had chosen and the alternative they had rejected. Thus, a polysemiotic utterance corresponds to each choice trial that the participants were presented with and were asked to justify. When the presented picture card was the participant’s actual choice (i.e., the non-manipulated one), it was coded as an Actual choice trial; when the presented picture card was the alternative that the participant had initially rejected, it was coded as a Manipulated choice trial. Eight polysemiotic utterances (4 for Manipulated and an equal number of 4 for Actual choice trials) for each participant (in total 37 participants) were coded.[5] We, thus, labeled the first tier, Trial, as Manipulated or Actual, and the transition from one to the next as Transition, and transcribed speech in the next tier (i.e., Speech).
The remaining tiers were ordered as five different Categories of Bodily Expression (CBE) on the basis of a tentative cline from signals to signs (see Section 2): (A) Adaptors, (B) Torso expressions, (C) Head expressions, (D) Face expressions and (E) Hand expressions, explained below.
Each one of these was preceded in the ELAN template by a tier specifying (if possible) which picture was being focused on, using, besides the video, the speech as main evidence: Chosen, when they talked about the chosen item; Non-chosen, when they referred to the non-preferred item; Pair, when both cards were referred to as a pair, and Other, when participants provided generalized references irrelevant with the specific choice trial, or comments that were vague in regards to which item they referred to (such as let’s see, that’s it, etc.).
The category (A) Adaptors were treated as clear signals, involving movements of Hand to hand, Hand to body, and Hand to object, with no denotational meanings. Hence, their meanings were listed as three different affect-related functions (Gentle self-ministration, e.g., stroking, soothing, etc.; Punitive self-ministration, e.g., squeezing, scratching, biting, etc.; and Adjustive, e.g., adjusting one’s glasses). The value Unclear was used when the quality of self-ministration could not be specified.
Categories (B-E) can, but need not, have sign status (see Section 2), and when they did have sign status, they always received a value for denotational meaning: Deictic, Iconic, Symbolic, or Unclear (when more than one value was possible). Additionally, they can also have non-denotational meaning Attitude, Other and Movement. Attitude was used for coding participants’ expressions of emphasis, modality and affect, while other was used for spontaneous and idiosyncratic gestures (often produced when participants tried to come up with a specific word). Movement was used to code subtle physical movements in space rather than purposes of communicative expression.
For (B), Torso expressions, we coded Forward, Back, Shrug and Other (e.g., turning right and left in their chair). (C) involved Head turning from Right/left, Down/up, and Forward/back (and vice versa). Some of these expressions were combined into one (e.g., head tilted to the right while pushed forward), and in such cases, the expression that seemed to prevail more was coded. Category (D) included face expressions with the Eyebrow(s) (e.g., up/down), Mouth (e.g., lips downwards), Both (e.g., eyebrows up and mouth downwards, like in surprise), and Other (e.g., when the whole face was involved in a particular expression; when the participant wrinkled their nose, like in repulsion, etc.). Finally, for category (E), Hand movements performed with one or more Finger(s) were coded; one Hand or Both hands; and Move, when participants moved one or both of the picture cards (e.g., pick up, push away, bring forward, etc.). To sum up, if categories (B-E) were judged to have signal status, then no value for denotational meaning was given, but they could still have non-denotational meaning. Thus, each instance of categories (B-E) could have values for both the denotational and non-denotational tiers, or for only one. Finally, the last two tiers were those of Manipulation status, specifying the type of detection and Comments, for any additional notes the annotator wished to make.
Since the goal of the study was to explore manifestations of choice awareness in the interaction of language with other semiotic systems, decisions on coding for the meaning of the bodily expressions relied to a great extent on how these appeared in coordination with speech. That was the case even for emblems, which, despite claims to the contrary (e.g., McNeill 2005), tend to co-occur with speech (Kendon 2008; Poyatos 1981). Likewise, headshakes and nodding, that can be more or less translatable into words (e.g., response particles like “no” and “yes”), most often interact polysemiotically with speech (e.g., Andrén 2014; Boholm and Allwood 2010).[6] Two independent raters coded 10 % of the data. The degree of interrater agreement was 90.2 %.
3.3 Hypotheses
Mouratidou et al. (2022) somewhat paradoxically argued that participants’ verbal utterances for manipulated choices would be marked by a higher degree of investment than justifications for actual choices, especially so when manipulations were detected (and the original choice needed to be re-confirmed). The reasoning for that was that even when participants did not detect the manipulation, they would still treat these trials “as more demanding, and therefore would require more effort to (a) assess the presented alternative as a potential choice, (b) provide reasons that resonate with their experiential life and (c) communicate them to the experimenter” (ibid: 710). The higher degree of cognitive and experiential complexity of such trials was proposed to be reflected in higher rates of verbal choice investment. Here, we extend the claim made in regard to investment expressed in language to non-verbal production and hypothesize that since participants will in general be required to “work harder” and either imagine a situation in which they will make such preference, or else reason as to why they would maintain their original preference:
H1:
Higher rates and a wider variety of bodily expressions will be present in the polysemiotic utterances produced for the manipulated than for the actual choice trials – at least for some of the semiotic systems.
Further, Mouratidou et al. (2022) hypothesized that the detected manipulations would provide the most conflicting experiences, requiring effort from the participants, which would be manifested in their verbal expressions. This implies that as the experiential complexity of the situation decreases from detected manipulations to non-detected and to actual choice trials, the language used to justify the choices would be simpler (ibid: 709). Again, extending this reasoning beyond language, participants’ bodily expressions can be expected to be influenced by the complexity of the task: while in the detected manipulations the inconsistency between the presented manipulated item and the initially chosen one would be verbally and bodily expressed, in non-detected manipulations, the verbal expression of detection would be lacking, but the presented item (the manipulated choice) could still be perceived as odd, surprising, or confusing, albeit without these impressions being verbally communicated. In such cases, the manipulation would be considered as (verbally) non-detected, but there could still be indicators of participants’ perplexity manifested in their bodily expressions. Thus, we formulated the following hypothesis:
H2:
Higher rates and a wider variety of bodily expressions will be present in the polysemiotic utterances produced for (a) detected than non-detected manipulations and (b) non-detected manipulations than actual choice trials.
Finally, based on our theoretical framework and previous research on the sign versus signal distinction, we hypothesized that since signs require (reflective) consciousness of the denotational relation between expression and object, while signals do not, there will be differences in how the five different Categories of Bodily Expressions (CBEs) are used in manipulated and actual choice trials:
H3:
Higher rates and a wider variety of signs will be present in the polysemiotic utterances produced for (a) detected than non-detected manipulations and (b) non-detected manipulations than actual choice trials. Inversely, a higher proportion and a wider variety of signals will be present in the polysemiotic utterances produced for (a) non-detected than detected manipulations and (b) detected manipulations than actual choice trials.
The motivation of the hypotheses reflects the methodological application of phenomenological triangulation, where the phenomenon under study is approached from first-person, second-person and, potentially also the detached third-person perspective (see Section 1). Thus, the claim of higher experiential complexity during manipulated trials rests first on our own experiences, thinking how puzzling it is when we face a choice we had not made (e.g., being served a different dish to the one we have ordered in a restaurant, opening the bag from the grocery store to find out a different kind of yogurt than the one we usually buy, etc.). And even more so, during a forced choice task, where someone of (assumed) authority is physically present and claims it to be ours, “forcing” us to defend our choice and restore the mistake, as opposed to situations where things progress smoothly without tensions or surprises (e.g., the non-manipulated trials).
In line with the second-person perspective, having conducted previous experiments on manipulation blindness, we have noticed differences in participants’ behavior (verbal and non-verbal) when they were presented with manipulated choices as opposed to their own, motivating the shift of our focus of research to non-verbal expressions. We thus extend the claim of our previous work to bodily expressions, drawing from cognitive linguistics (e.g., Langacker 1987, 2006) and phenomenology, and supported by work on various fields involving people’s verbalization of their experiences, such as psychotherapy and micro-phenomenological interviews (e.g., Goldman 2005; Hendricks 2002, 2009; Petitmengin and Bitbol 2009). These suggest correspondences between experience and language and that when there is congruence between them, the language used by the subjects tends to be simpler.
Finally, even third-person research on non-verbal behavior supports such claims: as stated in Section 2, adaptors have typically been considered as indicators of deception “leakage”, and more generally of high emotional and cognitive load, predicting their occurrence in communicative settings where ambiguous, interfering, and conflicting cues are involved. Likewise, affect displays have been claimed to be “discordant” with the message people explicitly convey through speech and gestures in deception interactions. Hence, it is only natural to assume that during the manipulated trials, participants either being ambivalent between the two choice alternatives or detecting the manipulation would for example (a) use more pointing gestures to argue for “this picture” over “that picture”; (b) would produce more adaptors in this conflicting situation; and (c) express their surprise, puzzlement, etc. with their facial expressions.
In the following section, we present the results of the three hypotheses in a logical order that can be likened with a funnel. H1 focuses on the differences between the polysemiotic utterances produced for manipulated and actual choice trials. H2 concentrates on the differences between detected versus non-detected manipulations, and non-detected manipulations versus actual choice trials. Finally, the H3 narrows down to explore potential differences between signs and signals in manipulated (both detected and non-detected) and actual choice trials. Our third-person method combined inferential and descriptive statistics. We used mixed effects regression models to compare the outcome variables between the manipulated and the actual choice trials, and then once again between the detected and non-detected manipulated trials and the actual trials. The analysis was performed in R version 4.4.0 (R core team 2024), using the package lme4 (Bates et al. 2015). However, in accordance with the lesser importance of third-person methods in cognitive semiotics than in cognitive science in line with phenomenological triangulation, we present and discuss the results of both kinds of statistics on an equal basis.
4 Results and discussion
4.1 General findings
There were 148 responses to actual choice trials, and 143 to manipulated choice trials. Participants’ responses to the manipulated trials were almost equally divided between detected 52 % (Clear 41 %, Possible 11 %) (see Section 3.1), and non-detected 48 %. Table 1 shows the average numbers of the five CBEs and the average trial durations for the conditions which our hypotheses below compare: manipulated choice trials as a whole, with detected and non-detected subtypes, as well as actual choice trials.
Average numbers with standard deviations in parentheses of instances of each Category of Bodily Expressions (CBE) and trial duration for manipulated choice trials, detected manipulations, non-detected manipulations, and actual choice trials.
| Manipulated | Manipulated detected | Manipulated non-detected | Actual | |
|---|---|---|---|---|
| CBE | ||||
| Adaptors | 1.62 (1.95) | 1.36 (2.01) | 1.94 (1.84) | 1.06 (1.41) |
| Torso | 3.36 (3.28) | 3.84 (3.58) | 2.88 (2.84) | 2.54 (2.46) |
| Head | 8.99 (7.93) | 10.17 (8.72) | 7.79 (6.76) | 7.05 (6.36) |
| Face | 3.12 (2.77) | 3.31 (2.74) | 2.96 (2.79) | 2.34 (2.36) |
| Hand | 6.18 (4.67) | 7.12 (5.36) | 5.22 (3.49) | 4.95 (4.11) |
| Trial duration (seconds) | 24.79 (13.82) | 27.34 (14.97) | 22.16 (11.93) | 18.66 (11.67) |
Table 1 shows that manipulated choice trials were on average more than 5 s longer than actual choice trials. An intuitive way to interpret this is that in the manipulated choice trials participants took more time to either imagine a situation in which they would have made such a preference, or to elaborate on why they would maintain their original one. The detected manipulations were also significantly longer (see Table 2) than the non-detected, and so were the non-detected when compared with the actual choice trials. Thus, when participants’ initial preference matched the presented item, the trials were shortest. In other words, even participants who did not (verbally) detect the manipulation still took more time to assess what was presented to them than when evaluating choices which they had initially made. Due to such differences, we included duration as a covariate in the analysis when comparing the different kinds of trials.
Mixed effects regression analysis of Categories of Bodily Expressions (CBE) and trial duration for manipulated and actual choice trials, with significant differences indicated in boldface.
| Estimate | Standard error | t | p | |
|---|---|---|---|---|
| CBE | ||||
| Adaptors | −0.570 | 0.173 | −3.300 | 0.001 |
| Torso | −0.888 | 0.267 | −3.322 | 0.001 |
| Head | −1.858 | 0.607 | −3.059 | 0.002 |
| Face | −0.751 | 0.260 | −2.881 | 0.004 |
| Hand | −1.222 | 0.461 | −2.649 | 0.009 |
| Trial duration (seconds) | −6.176 | 1.234 | −5.005 | 0.000 |
4.2 Manipulated versus Actual choice trials (H1)
The average numbers of CBEs in the manipulated and the actual choice trials are shown in Figure 2. As can be seen, the average numbers are higher in the manipulated choice trials than in the actual choice trials. The relative difference is largest in the Adaptors. In other words, when participants faced inconsistencies between the presented and the initially preferred item, and had to reconfirm their initial choice or explore anew the presented alternative, they more often engaged in body stimulation than when the presented item matched their actual preference. This is in line with previous findings, where adaptors act as a means of regulating sensory input (Freedman 1977: 117). Face expressions were also more often produced during manipulated trials, and so were Torso, Head and Hand expressions. In general, participants used their bodies more often to express themselves during manipulated trials than when they justified actual choices.
Average numbers of instances of Categories of Bodily Expressions (CBE) for manipulated and actual choice trials.
Further, we examined differences in the variety of bodily expressions between the two conditions by looking at how many of the five CBEs were present in each trial. As shown in Figure 3, a greater number of different CBEs occurred for the manipulated trials. This suggests that the tension between participants’ original choice and what was presented to them was manifested in their bodily expressions by engaging more semiotic systems to either refute or reconfirm the presented item as the preferred one.
Average numbers with standard deviations in parentheses of types of Categories of Bodily Expressions (CBE) per trial.
The results of the mixed effects linear regression analyses of the five CBEs and trial duration for manipulated and actual choice trials, without controlling for the duration, showed that all differences for the five CBEs were statistically significant, as shown in Table 2. To perform the analysis so is justified, given the considerable differences in length between manipulated and actual choice trials, as shown in Table 1. Even when controlling for duration, the differences between manipulated and actual trials in Adaptors (EST = 0.36, SE = 0.23, t = 1.97) and Variety (EST = 0.18, SE = 0.09, t = 2.04) were significant. In sum, the higher rates and the wider variety of the CBEs found in the polysemiotic utterances for the manipulated trials support the first hypothesis (H1).
4.3 Detected versus non-detected manipulations, and non-detected versus actual choice trials (H2)
As shown in Figure 4, Torso, Head, Face, and Hand expressions occurred on average more often in detected manipulations, followed by non-detected, and, finally, actual choice trials, as predicted by H2. The inferential statistics for detected versus non-detected, and non-detected versus actual choice trials are summarized in Table 3, with significant differences marked in bold. The table shows that differences between detected and non-detected manipulation trials were significant for only Head and Hand.
Average numbers of instances of Categories of Bodily Expression (CBE) for detected manipulated trials, non-detected manipulated trials, and actual choice trials.
Mixed effects regression analysis of detected and non-detected manipulated trials, and non-detected manipulated trials and actual choice trials for the different Categories of Bodily Expression (CBE) and trial duration, with significant differences indicated in boldface.
| Detected vs. Non-detected manipulations | Non-detected manipulations vs. actual choices | |||||||
|---|---|---|---|---|---|---|---|---|
| EST | SE | t | p | EST | SE | t | p | |
| CBE | ||||||||
| Adaptors | −0.211 | 0.269 | −0.784 | 0.434 | −0.681 | 0.223 | −3.046 | 0.003 |
| Torso | 0.717 | 0.414 | 1.706 | 0.089 | −0.512 | 0.346 | −1.478 | 0.140 |
| Head | 2.390 | 0.955 | 2.504 | 0.013 | −0.604 | 0.783 | −0.771 | 0.441 |
| Face | 0.507 | 0.405 | 1.251 | 0.212 | −0.485 | 0.336 | −1.444 | 0.150 |
| Hand | 2.031 | 0.703 | 2.889 | 0.004 | −0.156 | 0.586 | −0.267 | 0.790 |
| Trial duration (seconds) | 5.102 | 1.915 | 2.664 | 0.008 | −3.500 | 1.581 | −2.214 | 0.028 |
Adaptors, on the other hand, had on average more instances in non-detected manipulated trials than in detected manipulated and actual choice trials, and the difference between non-detected manipulated trials and actual was statistically significant (see Table 3). In other words, when participants were presented with the non-preferred alternative as their choice, and without verbally expressing anything that could indicate that they had noticed such inconsistency, they resorted to self or object-stimulation more often than when they detected the manipulation or justified their actual choices. This is consistent with the role of adaptors as “substitutive maneuvers for what is not verbalized” (Freedman 1972: 168), signaling a form of disengagement from a communicative setting (ibid, 174). Thus, the increased rate of Adaptors in the non-detected trials suggests that this category is a possible indicator of choice awareness, albeit of a marginal kind (e.g., Gurwitsch 1974; Koffka 1935).
As far as the variety of bodily expressions is concerned, there were on average more categories present in the polysemiotic utterances produced in detected and non-detected manipulated trials rather than in the actual choice trials as shown in Figure 5, where the difference between non-detected manipulated and actual choice trials was statistically significant (EST = −0.357, SE = 0.119, t = −3.008, p = 0.003). On the other hand, the average numbers of categories present in detected and non-detected manipulated trials were similar, with a non-significant difference (EST = 0.085, SE = 0.142, t = 0.599, p = 0.550). This similarity and the difference between non-detected manipulated and actual choice trials indicate that the mismatch between participants’ preference with the presented item was manifested in the ways participants engaged their bodies during the subsequent choice justifications: equally many categories for detected and non-detected manipulations, and fewer in actual choice trials. Note that this holds independently of whether this difference was verbally expressed or not, showing once again the importance of looking for evidence for choice awareness beyond language.
Average numbers with standard deviations in parentheses of types of Categories of Bodily Expression (CBE) per trial.
In sum, there were overall more instances for four of the five CBEs and a higher variety in (a) detected than non-detected manipulated trials, and for all five CBEs and a higher variety in (b) non-detected manipulated than actual choice trials. For (a), differences in trial duration, as well as Head and Hand expressions were significant, while for (b), differences in trial duration, and the categories Adaptors and Variety. The findings reported above support the predictions of the second hypothesis (H2).
4.4 Signs and signals in detected versus non-detected, and non-detected versus actual choice trials (H3)
To address the more specific third hypothesis concerning differences between signs and signals, we looked at the values assigned for denotational (i.e., Deictic, Iconic, Symbolic) and non-denotational meaning (i.e., Attitude, Other, Movement) for all CBEs. Those that occurred less than five times were excluded from the analysis. We base the discussion below on the descriptive statistics.
Table 4 summarizes the average numbers of CBEs with denotational meaning. As can be seen, there were on average more signs in detected manipulated trials, followed by actual, and, finally, non-detected manipulated trials, which is only partly aligned with H3 (i.e., more sign use in detected manipulated trials, followed by non-detected manipulations and actual choice trials.). On the other hand, when looking at the highest rates in each column, the descriptive statistics are in line with the hypothesis.
Average numbers of different kinds of denotational meaning (deictic, iconic, symbolic) for the different Categories of Bodily Expression (CBE), with highest rates per category indicated in boldface.
| Denotational meaning | CBE | Detected manipulated | Non-detected manipulated | Actual |
|---|---|---|---|---|
| Deictic | Hand | 3.32 | 1.95 | 1.99 |
| Iconic | Hand | 0.21 | 0.44 | 0.20 |
| Face | 0.05 | 0.14 | 0.09 | |
| Symbolic | Hand | 0.17 | 0.28 | 0.20 |
| Face | 0.24 | 0.19 | 0.09 | |
| Head | 0.81 | 0.64 | 0.60 | |
| Torso | 0.23 | 0.16 | 0.18 | |
| Total | 5.03 | 3.08 | 3.35 |
Notably, Deictic hand expressions had higher rates in detected manipulated trials than in non-detected manipulated trials and nearly the same in non-detected manipulated and actual choice trials. This can be interpreted as showing that participants used deictic gestures to differentiate between the two alternative cards when arguing for their initial preference.
There were many more Iconic Hand and Face expressions for the non-detected manipulations than in detected manipulated and actual choice trials. For Hand, there was no difference between detected manipulations and actual choice trials, while for Face, there were higher rates in actual than non-detected manipulated choice trials. Notably, the differences of Iconic Hand expressions between detected and non-detected manipulations, as well as between non-detected and actual choice trials may indicate that participants used iconic gestures to help imagine reasons for supporting a choice they had not initially made. In the study’s choice task, involving human faces assessed on attractiveness, Iconic Face expressions may suggest that participants adopted an affective stance expressing their verbal justification (e.g., he looks mild) with a matching, iconic face expression.
Symbolic Hand occurred more frequently in the non-detected manipulated trials, followed by actual choice trials and least often in the detected manipulated choice trials. As expected, Face (e.g., ignorance) and Head (e.g., nodding) expressions occurred more frequently in the detected manipulated trials, followed by non-detected manipulated and least often in the actual choice trials. This suggests that participants used such signs to express epistemic modality (uncertainty, rejection, etc.) for the detected and non-detected manipulated trials, and much less so when they justified their actual choices. Symbolic Torso expressions occurred more frequently in detected manipulated trials too, but, followed by actual choice trials and non-detected manipulations.
Turning to non-denotational meaning, Adaptors were the only of the five CBEs that always had signal status (i.e., non-denotational meaning). Table 5 shows the average numbers of Adaptors and their affect-related functions across trials.
Average numbers of Adaptors for Detected, Non-detected and Actual choice trials, with highest rates per category indicated in boldface.
| Function | Detected manipulated | Non-detected manipulated | Actual |
|---|---|---|---|
| Gentle | 0.37 | 0.37 | 0.3 |
| Punitive | 0.33 | 0.40 | 0.24 |
| Adjustive | 0.17 | 0.10 | 0.11 |
| Unclear | 0.36 | 0.84 | 0.48 |
| Total | 1.23 | 1.71 | 1.13 |
As expected, there were more Adaptors during non-detected manipulations, followed by detected manipulations and actual choice trials. Looking at the affect-related functions, the majority, in the non-detected manipulated trials, was coded as Unclear due to the difficulty of deciding the force by which the expression was conducted by solely looking at the video recordings (e.g., whether the participants were rubbing or squeezing their arms).
Gentle adaptors occurred roughly similarly in all choice trials, however, Adjustive adaptors were more frequent in detected cases than in non-detected and actual, while Punitive adaptors were more frequent in non-detected choice trials, and the least for actual. This could be interpreted as follows: when participants verbally detected the manipulation, they either “rewarded” themselves with a gentle self-stimulation or adjusted an object on their body, in a similar manner that they verbally “adjusted” what was presented to them with what they had actually chosen. On the contrary, the fact that most Punitive adaptors were performed in non-detected manipulation trials could be seen as an implicit form of participants “punishing” themselves for not verbalizing that something might be wrong with what has been presented to them, channeling the non-verbalised thought back to them with punitive self-stimulation. Consistent with the literature, self-stimulation can be viewed as providing feedback action that regulates the arousal created by the specific condition (e.g., interference, conflict, tension, etc.) (e.g., Freedman 1977; Grand et al. 1975) (see Section 4.4).
Finally, looking at the non-denotational meaning for the CBEs with signal status (see Table 6), against our prediction, there were on average more expressions reflecting Attitude (i.e., emphasis, modality, affect) in detected manipulated choice trials, followed by actual choice trials and, finally, non-detected manipulations. Torso and Head expressions were more frequent in detected manipulations, followed by actual choice trials and then non-detected manipulations, while for Face and Hand expressions, the reverse occurred: slightly more in non-detected manipulations, and then in actual choice trials. Similarly, expressions of Movement did not follow the prediction. Torso and Hand Movement rates were higher in detected trials, followed by non-detected and actual trials, while there were more Head Movement, following the detected choice trials, in the actual manipulations than in non-detected, and most Face Movement occurred in actual trials. However, the differences between these were minor.
Average numbers of non-denotational meaning for Categories of Bodily Expression (CBE) for detected manipulated, non-detected manipulated and actual choice trials, with highest rates per category indicated in boldface.
| Non-denotational meaning | CBE | Detected manipulated | Non-detected manipulated | Actual |
|---|---|---|---|---|
| Attitude | Torso | 0.63 | 0.38 | 0.53 |
| Head | 2.01 | 1.31 | 1.46 | |
| Face | 2.00 | 1.59 | 1.54 | |
| Hand | 2.65 | 2.04 | 1.89 | |
| Movement | Torso | 2.55 | 2.13 | 2.04 |
| Head | 3.82 | 3.26 | 3.71 | |
| Face | 0.50 | 0.58 | 0.69 | |
| Hand | 0.24 | 0.15 | 0.14 | |
| Other | Hand | 0.29 | 0.47 | 0.31 |
| Total | 14.69 | 11.91 | 12.31 |
In regards to non-denotational meaning, Other expressions were almost double in non-detected manipulations: this category included participants idiosyncratic hand expressions performed when they were searching for a specific word (e.g., he looks more….more….how to put it…sensitive). The difference of such expressions between non-detected manipulated, detected manipulated and, even more so, actual choice trials could indicate that while for the latter, participants had easier access to their vocabulary to verbally express their choice justifications, in non-detected manipulations, they strove to find specific words and phrases manifested with an increased use of Other hand expressions. This trend could indicate an implicit form of manipulation awareness, or at least of experiencing something wrong with the presented choices, expressed with participants making more effort to articulate their thoughts due to the inconsistency between what they had chosen and what was presented to them.
We also looked at the form of the CBEs between different trials. In general, there were few differences apart from the following. For Adaptors, the vast majority for detected manipulated and actual choice trials consisted of self-stimulation of Hand to body, with only a few instances of Hand to object. However, for the non-detected manipulated cases, Adaptors were roughly balanced between Hand to body and Hand to hand, but there were only half as many Hand to object cases. However, Hand to object adaptors were still more than twice as common than in the other conditions (see Table 7).
Adaptors form in detected manipulated, non-detected manipulated and actual choice trials, with highest rates per category indicated in boldface.
| Detected manipulated | Non-detected manipulated | Actual | |
|---|---|---|---|
| Hand to body | 0.48 | 0.30 | 0.27 |
| Hand to hand | 0.08 | 0.27 | 0.14 |
| Hand to object | 0.06 | 0.13 | 0.05 |
This is relevant, since distinctions between the form of adaptors have been associated with differences in their function. Freedman (1972: 172) considered adaptors in dyadic communication as “body-focused movements”, involving “a depletion of communicative effort and a splitting of the speaker’s attention”, and linked different adaptors to “different constellations of conflict between what remains unverbalized and is expressed only in motor form, and what is verbalized”. Specifically, Hand to body adaptors were associated with withdrawal from the communicative interaction, functioning as direct need gratifiers; Hand to hand with acute discomfort; while, Hand to object with suppressed or repressed thought (when the thought is verbalized, the movements drop out), leading to exploratory and problem-solving behavior. Likewise, Mahl (1968) identified Hand to object adaptors as “anticipatory”, associated with suppressed or repressed thoughts.
Further, Hand to hand adaptors were considered to involve activity which is non-gentle (e.g., squeezing, rubbing, scratching), while Hand to body to usually involve soothing, stroking, etc. This is in fact aligned with our findings on the affect-related function of adaptors, where most adaptors were coded as Gentle for the detected manipulated cases, rather than in non-detected manipulated and actual choice trials, and Punitive occurred more in non-detected manipulations, and the least often for actual choice trials. In sum, adaptors have been linked to different ways of relating to the listener and different levels of verbal articulation, since they entail different forms of resistance to articulation: some inhibiting, others facilitating verbal expressions (Freedman 1972).
Another CBE with difference in its form between detected manipulated, non-detected manipulated and actual choice trials was Face. One of the coded features were Eyebrows: when there was no evidence from speech that participants were expressing Attitude or acted as a component of another bodily expression, it was coded as Movement. On average, there were three times more expressions of this sort in non-detected manipulated cases (0.17), than in detected manipulated (0.05) and actual choice trials (0.06). This may indicate that participants without verbalizing affect of any kind, expressed their surprise by using their eyebrows (a typical facial feature expressing surprise). Thus, this could be regarded as a signal of participants’ awareness of something being wrong with the choice that was presented to them as their own, when speech was lacking.
In sum, the third hypothesis (H3) found some support, but only in part. As expected, there were overall most signs (i.e., denotational meaning) in detected manipulated choice trials. However, this rate was followed by the rate of sign use in actual choice trials and not by the non-detected manipulations as predicted, though the differences between these two conditions were mostly minimal. As for signals, when considering only adaptors (see Table 5), the rates of these were indeed highest in the non-detected manipulated trials, followed by detected manipulated and actual choice trials. However, when considering non-denotational meaning in CBEs in general (see Table 6), the rates were highest in detected manipulated choice trials, followed by actual choice trials and non-detected manipulations, against our prediction. In the discussion above, we suggested some possible reasons for why this could be the case.
4.5 Additional findings
Apart from the results concerning the three hypotheses, we made several relevant observations which we summarize here.
4.5.1 Types of detection: possible detection
For the sake of greater clarity, as the relatively low number of cases of Possible detection (16), this sub-condition was merged with that of Clear detection (59) for the main analysis (see Section 4.1). However, when considering Possible and Clear detection separately, there were some potentially interesting differences.
First, Possible detection trials were on average 12.7 s longer than Clear detection trials. This implies that participants took more time to make sense of the discrepancy between the presented and the preferred item, and/or elaborate on this conflicting experience than when this discrepancy was verbally expressed (Clear detection). Thus, in such cases, uncertainty was expressed with both speech and body.
When we considered Possible and Clear detection together, Face expressions occurred more for detected manipulations, followed by non-detected manipulations and actual choice trials. However, this order differs when we look at Possible and Clear detection separately: there were on average more Face expressions in non-detected manipulations (2.9), slightly less for detected manipulations (2.8) and the least for actual choice trials (2.3). This suggests that when participants were presented with the non-preferred alternative as their choice, they formed more facial expressions than when they detected the manipulation or justified their actual choices. The increased number of instances of Face expressions during non-detected trials implies that the CBE Face could be an indicator of (marginal) choice awareness similarly to Adaptors.
4.5.2 Shrug and head shake
We investigated two different bodily expressions expressing (different kinds of) negation: the Shrug and the Head shake (e.g., right to left). Although both could be used alongside participants’ verbal expressions like “I don’t know”, we noticed a difference in the way they were actually used. In general, there was more shrugging in detected manipulations and least often in non-detected manipulations, while Head shake occurred slightly more in non-detected manipulations and the least in actual choice trials (see Table 8). This may indicate that Shrug, at least in this context, was used to convey participants’ expression of completing the assigned task (e.g., “I have nothing more to say”), while the Head shake was used to express genuine ignorance or denial. The multiplicity of meaning of the shrug has been extensively discussed by Debras (2017: 27–28), taking on “dynamic meanings (e.g., incapacity, inaction), as well as affective (indifference, rejection) or epistemic ones (in the sense that the speaker has nothing new to add to what is already known)”. On the other hand, the Head shake can be taken “as part of language in the traditional sense of language as a conventionalized system”, as discussed by Andrén (2014: 141). The difference between Head shake in non-detected and actual choice trials (see Table 8) indicates that participants negated more explicitly when the choices were manipulated than when they were their own. Thus, this could be regarded as an indicator of choice investment, and potentially even awareness of the manipulation, without this being verbally expressed.
Shrug and Symbolic Head shake in detected manipulated, non-detected manipulated and actual choice, with highest rates per category indicated in boldface.
| Detected manipulated | Non-detected manipulated | Actual | |
|
|
|||
| Shrug | 0.22 | 0.16 | 0.18 |
| Symbolic head shake | 0.12 | 0.14 | 0.06 |
5 Conclusions
Addressing the phenomenon of choice making and manipulation detection from the perspective of cognitive semiotics implies the principle of phenomenological triangulation: systematic calibration of methods based on first-person (e.g. intuition-based), second-person (e.g. social interaction-based) and (optionally) third-person (e.g. quantitative) methods (Pielli and Zlatev 2020; Zlatev and Mouratidou 2024). Using such an approach and following previous research (Mouratidou 2020; Mouratidou et al. 2022), we have tried to offer deeper insights into the phenomenon of manipulation detection in “choice blindness” experiments, and correlatively, into the phenomenon of choice awareness. When confronted with the situation of having to justify a choice that we have either made in the past or have not, we face a dynamic, more or less experientially complex situation manifested in our verbal expressions and, even further, in how we express this with our body, the ground for all human meaning making, as argued by Merleau-Ponty (1962; cf. Hass 2008).
We would suggest that claims that “we do not know as much of ourselves as we think we did” (Johansson 2006: 39) are based on rather constrained notions of (self) knowledge, underestimating the embodied and intersubjective character of our existence. While we must express our indebtedness to the scholars that initiated the field of “choice blindness” for providing the field with a controversial phenomenon that challenges everyday intuitions, we would argue that by going beyond the perspectives of traditional cognitive science, we can provide a more nuanced account of this phenomenon. After all, the general proposal for such an extension is not new, for example from “cognitivism” to “emergence” to “enaction”, as suggested already by Varela et al. (1991), and more recently in phenomenological cognitive science (Gallagher and Zahavi 2012) and cognitive semiotics (Sonesson 2020; Zlatev 2015).
Such a perspective gives rise to the nuanced findings that we reported in this paper. And they, in turn, speak against one of the tenets of the discourse against our reliability as conscious agents: the homogeneity between participants’ verbal reports justifying choices they made and choices they did not, with assumptions of “confabulation” even when choice manipulation is not involved. In contrast, we found notable differences in the participants’ polysemiotic utterances justifying actual choices and non-detected manipulated choices. These differed in at least three aspects: (a) duration, (b) rates and (c) the variety of categories of bodily expressions (CBEs) used. Thus, whether or not the detection of the manipulation was verbally expressed, we could show that it was reflected in participants’ longer time of assessing the assumed choices, increased rates of bodily expressions, and engagement of more parts of their bodies during the non-detected manipulation than the actual choice trials.
Based on such findings, we would argue for a degree of awareness of the manipulations, even in apparently “non-detected” manipulations, and even if this awareness is not focal, but a matter of pre-reflective, self-consciousness (Gallagher and Zahavi 2012), manifested in the use of participants bodily expressions, such as adaptors. This is consistent with the rich understanding of different kinds of consciousness in phenomenology (e.g., Sokowlowski 2000; Zahavi 2018), and cognitive semiotics (Zlatev 2018). In particular, it can be argued that bodily awareness of choice manipulation manifests itself with signals, operating on the level of operative intentionality, while bodily signs imply categorial intentionality, while explicit verbal statements express propositional intentionality.[7]
Based on the semiotic distinction between signs and signals (Zlatev et al. 2020), we could explore their presence in different kinds of choice trials, corresponding methodologically to “conditions” or “independent variables”. We found that signs were more often used by the participants when they were justifying detected manipulations, less so in actual choice trials, and the least in non-detected cases of manipulation. However, there were many variations in this pattern. Depending on the category of bodily expressions and their status as signs or signals (depending on whether they could be classified as denotational or not), we observed rate differences when comparing between non-detected manipulations and actual choices trials: more in the former, and the least in the latter. Further, signals in general were again mostly present in detected manipulations, followed by actual choice trials and non-detected manipulations. While contradicting our third hypothesis, this finding highlights even more the role of adaptors occurring mostly in non-detected manipulations, less so in detected and the least in actual choice trials.
Future studies on the topic could focus on differences between the form of bodily expressions, especially of adaptors and deictic gestures, to investigate whether the way an expression is formed reflects the affective status of participants towards a preferred or a non-preferred alternative, and thus provide further evidence of choice awareness expressed implicitly in bodily behavior.
Cognitive semiotics is a science which helps to widen our understanding of (human) meaning making and existence, by unifying first-person, second-person and third-person methodologies. In the paper, we hope to have shown its potential for offering a richer understanding of choice awareness and manipulated detection, re-confirming us as conscious agents with different degrees of awareness manifested both verbally and bodily.
About the authors
Alexandra Mouratidou is a Doctoral Candidate in Cognitive Semiotics, Division for Cognitive Semiotics, Center for Languages and Literature at Lund University, Sweden. She was a public relations officer in the board of the International Association for Cognitive Semiotics (IACS), from 2022 to 2024, and a co-organiser of IACS-5. Her research interests lie in polysemiotic communication, especially when choice-making and choice manipulations are concerned, as well as in phenomenology and consciousness. She is assistant editor of Public Journal of Semiotics.
Jordan Zlatev is Professor of General Linguistics and Director of Research for the Division of Cognitive Semiotics at Lund University, Sweden. He was the first president of the Swedish (currently: Scandinavian) Association for Language and Cognition (SALC), from 2006 to 2009, and of the International Association for Cognitive Semiotics (IACS), from 2013 to 2014, and he is currently a board member and IACS-5 organizer. He is the author of Situated Embodiment: Studies in the Emergence of Spatial Meaning (1997), and of over 100 articles in academic journals and anthologies. He co-edited The Shared Mind: Perspective on Intersubjectivity (2008), Moving Ourselves, Moving Others: Motion and Emotion in Intersubjectivity, Consciousness, and Language (2012) and Meaning, Mind and Communication: Explorations in Cognitive Semiotics (2016). His current research focuses on polysemiotic communication, and more generally on the nature of language in relation to other semiotic systems like gesture and depiction. His approach to cognitive semiotics is strongly influenced by phenomenology, the philosophy and methodology of lived experience. He is editor-in-chief of Public Journal of Semiotics.
Joost van de Weijer is employed at Lund University Humanities Lab in Sweden. He collaborates with various researchers on the topics of animal communication, prosody and factivity, and motivation in highschool students to learn a foreign language. He teaches courses in psycholinguistics and in the statistical analysis of experimental data, provides methodological support for ongoing research projects within various areas of language and speech, and on psychophysiological measurement of behavioural and cognitive processes. He is editorial secretary for the journal Studia Linguistica, and is member of the editorial board of the journal Bilingualism, Language and Cognition.
Acknowledgments
We thank two anonymous reviewers for the many comments, even when we disagreed, and their suggestions for revisions, many of which we have implemented in this final version of this paper. We also thank the editors for their guidance.
Appendix A: Table A.1. ELAN coding template for polysemiotic utterances produced for actual and Manipulated choice trials
| Tier | Name | Controlled vocabulary |
|---|---|---|
| 1 | Trial | AC (actual choice trial) MC (manipulated choice trial) TR (transition) |
| 2 | Speech | Transcription |
| 3 | Picture card | C (chosen) NC (non-chosen) P (pair) O (other) |
| 4 |
(A) Adaptors
Form |
HH (hand to hand) HB (hand to body) HO (hand to object) |
| 5 | Meaning | Gentle self-ministration (stroke, sooth, etc.) Punitive self-ministration (squeeze, scratch, bite) Adjustive Unclear |
| 6 |
(B) Torso expressions
Form |
F (forward) B (back) S (shrug) O (other) |
| 8 | Denotational | Deictic Iconic Symbolic Unclear |
| 9 | Non-denotational | Attitude (epistemic, affective, etc.) Other Movement |
| 10 | Picture card | C (chosen) NC (non-chosen) P (pair) O (other) |
| 11 |
(C)Head expressions
Form |
RL (right/left) DU (down/up) FB (forward/back) |
| 12 | Denotational | Deictic Iconic Symbolic Unclear |
| 13 | Non-denotational | Attitude (epistemic, affective, etc.) Other Movement |
| 14 | Picture card | C (chosen) NC (non-chosen) P (pair) O (other) |
| 15 |
(D) Face expressions
Form |
E (eyebrows up/down) M (mouth) B (both) O (other) |
| 16 | Denotational | Deictic Iconic Symbolic Unclear |
| 17 | Non-denotational | Attitude (epistemic, affective, etc.) Other Movement |
| 18 | Picture card | C (chosen) NC (non-chosen) P (pair) O (other) |
| 19 |
(E) Hand expressions
Form |
F (finger/s) H (hand) B (both hands) M (move picture card: pick up, push away, bring forward, etc.) |
| 20 | Denotational | Deictic Iconic Symbolic Unclear |
| 21 | Non-denotational | Attitude (epistemic, affective, etc.) Other Movement |
| 22 | Picture card | C (chosen) NC (non-chosen) P (pair) O (other) |
| 23 | Manipulation status | C (clear) P (possible) N (none) |
| 24 | Comments |
References
Andrén, Mats. 2010. Children’s Gestures from 18 to 30 months, In Center for Languages and literature. Lund: Lund University PhD Thesis.Search in Google Scholar
Andrén, Mats. 2014. Multimodal constructions in children: Is the headshake part of language? Gesture 14(2). 141–170. https://doi.org/10.1075/gest.14.2.02and.Search in Google Scholar
Bargh, John, A. & Melissa J. Ferguson. 2000. Beyond behaviorism: On the automaticity of higher mental processes. Psychological Bulletin 126(6). 925. https://doi.org/10.1037/0033-2909.126.6.925.Search in Google Scholar
Bates, Douglas, Martin Maechler, Ben Bolker & Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67(1). 1–48. https://doi.org/10.18637/jss.v067.i01.Search in Google Scholar
Bressem, Jana & Cornelia Müller. 2014. A repertoire of German recurrent gestures with pragmatic functions. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeil & Jana Bressem (eds.), Body – language – communication. An international Handbook on Multimodality in human interaction, Vol. II, 1575–1592. Berlin: De Gruyter Mouton.10.1515/9783110302028.1575Search in Google Scholar
Boholm, Max & Jens Allwood. 2010. Repeated head movements, their function and relation to speech. In Michael Kipp, Jean-Claude Martin, Patrizia Paggio & Dirk Heylen (eds.), Workshop on multimodal corpora: Advances in capturing, coding and analyzing multimodality. LREC 2010. Valletta, Malta.Search in Google Scholar
Calbris, Geneviève. 1990. The semiotics of French gestures. Bloomington, IN: Indiana University Press.Search in Google Scholar
Clark, H. Herbert. 1996. Using language. Cambridge: Cambridge University Press.Search in Google Scholar
Debras, Camille. 2017. The shrug: Forms and meanings of a compound enactment. Gesture 16(1). 1–34. https://doi.org/10.1075/gest.16.1.01deb.Search in Google Scholar
Dennett, Daniel. 1991. Consciousness explained. London: Penguin Books.Search in Google Scholar
Dennett, Daniel. 1996. Finds of minds: Towards an understanding of consciousness. New York: Basic Books.Search in Google Scholar
Devylder, Simon & Jordan Zlatev. 2020. Cutting and breaking metaphors of the self and the motivation & sedimentation model. Figurative Meaning Construction in Thought and Language. 253–282.10.1075/ftl.9.11devSearch in Google Scholar
Efron, David & Stuyvesant Van Veen. 1972. Gesture race and culture: A tentative study of the spatio-temporal and “linguistic” aspects of the gestural behavior of eastern jews and southern italians in new york city living under similar as well as different environmental conditions. The Hague: Mouton.Search in Google Scholar
Ekman, Paul. 1991. Telling Lies: Clues to Deceit in the Market Place, Politics, and Marriage. New York: W.W. Norton & Company..Search in Google Scholar
Ekman, Paul. 1999. Emotional and conversational nonverbal signals. In Lynn S. Messing & Ruth Campbell (eds.), Gesture, speech, and sign, 45–55. Oxford: Oxford University Press.10.1093/acprof:oso/9780198524519.003.0003Search in Google Scholar
Ekman, Paul. 2009. Lie catching and micro expressions. In Clancy Martin (ed.), The Philosophy of deception. Oxford: Oxford University Press.10.1093/acprof:oso/9780195327939.003.0008Search in Google Scholar
Ekman, Paul & Wallace V. Friesen. 1969. Nonverbal leakage and cues to deception. Psychiatry. Journal for the Study of Interpersonal Processes 32(1). 88–106. https://doi.org/10.1080/00332747.1969.11023575.Search in Google Scholar
Ekman, Paul & Wallace V. Friesen. 1974. Nonverbal behavior and psychopathology. In R. J. Friedman & M. N. Katz (eds.), The psychology of depression: Contemporary theory and research, 203–232. Washington, D.C.: J. Winston.Search in Google Scholar
Ekman, Paul & Wallace V. Friesen. 1976. Measuring facial movement. Nonverbal Behavior 1. 56–75. https://doi.org/10.1007/bf01115465.Search in Google Scholar
Ekman, Paul, Wallace V. Friesen, Maureen O’ Sullivan & Klaus Scherer. 1976. Body movement and voice pitch in deceptive interaction. Semiotica 16(1). 23–27. https://doi.org/10.1515/semi.1976.16.1.23.Search in Google Scholar
Ekman, Paul, Maureen O’Sullivan, Wallace V. Friesen & Klaus Scherer. 1991. Invited article: Face, voice, and body in detecting deceit. Journal of Nonverbal Behavior 15(2). 125–135. https://doi.org/10.1007/bf00998267.Search in Google Scholar
Freedman, Norbert. 1972. The analysis of movement behavior during the clinical interview. In Aron W. Siegman & Benjamin Pope (eds.), Studies in dyadic communication. New York: Pergamon Press.10.1016/B978-0-08-015867-9.50012-5Search in Google Scholar
Freedman, Norbert. 1977. Hands, words, and mind: On the structuralization of body movements during discourse and the capacity for verbal representation. In Norbert Freedman & Stanley Grand (eds.), Communicative Structures and psychic structures. New York: Springer Science and Business media.10.1007/978-1-4757-0492-1_6Search in Google Scholar
Gallagher, Shaun & Dan Zahavi. 2012. The phenomenological Mind, 2nd edn. London: Routledge.10.4324/9780203126752Search in Google Scholar
Givens, B. David. 1977. Shoulder shrugging: A densely communicative expressive behavior. Semiotica 19(1-2). 13–28. https://doi.org/10.1515/semi.1977.19.1-2.13.Search in Google Scholar
Goldberg, E. Adele. 1995. Constructions: A construction grammar approach to argument structure. Chicago: University of Chicago Press.Search in Google Scholar
Goldberg, E. Adele. 2006. Constructions at work: The nature of generalization in language. Oxford: Oxford University Press.10.1093/acprof:oso/9780199268511.001.0001Search in Google Scholar
Goldman, Alan. 2005. Reason internalism. Philosophy and Phenomenological Research 71(3). 505–533. https://doi.org/10.1111/j.1933-1592.2005.tb00469.x.Search in Google Scholar
Grand, Stanley, Norbert Freedman, Irving Steingart & Charles Butchwald. 1975. The relation of adaptive styles to kinetic and linguistic aspects of interview behavior. Journal of Nervous and Mental Disease 161. 293–306. https://doi.org/10.1097/00005053-197511000-00001.Search in Google Scholar
Green, Jenny. 2014. Drawn from the ground: Sound, sign and inscription in Central Australian sand stories. Cambridge: Cambridge University Press.10.1017/CBO9781139237109Search in Google Scholar
Gurwitsch, Aron. 1974. The field of consciousness. In The Phenomenology of the Social World, 67–88. Evanston, IL: USA: Northwestern University Press.Search in Google Scholar
Hass, Lawrence. 2008. Merleau-ponty’s philosophy. Bloomington: Indiana University Press.Search in Google Scholar
Hendricks, M. 2002. Focusing-oriented/experiential psychotherapy: Research and practice. In D. Cain & J. Seeman (eds.), Humanistic psychotherapies: Handbook of research and practice, 221–252. Washington, DC: American Psychological Association.10.1037/10439-007Search in Google Scholar
Hendricks, Marion. 2009. Experiencing level: An instance of developing a variable from a first person process so it can be reliably measured and taught. Journal of Consciousness Studies 16(10–12). 129–155.Search in Google Scholar
Holler, Judith. 2010. Speakers’ use of interactive gestures as markers of common ground. In Stefan Kopp & Ipke Wachsmuth (eds.), Revised selected papers from the 8th international gesture workshop, GW 2009, Vol. 5934, 11–12. Berlin: Springer.Search in Google Scholar
Itkonen Esa. 2003. What is language? A study in the philosophy of linguistics, Vol. 8. Turku: University of Turku: Publications in General Linguistics.Search in Google Scholar
Jack, I. Anthony & Andreas Roepstorff. 2003. Why trust the subject? Journal of Consciousness Studies 10(9-10). v–xx.Search in Google Scholar
Jack, I. Anthony & Andreas Roepstorff. 2004. Trusting the subject? Part 2. Journal of Consciousness Studies. 11–17.10.4135/9781446278901.n6Search in Google Scholar
Johansson, Petter. 2006. Choice blindness: The incongruence of intention, action and introspection. Lund: Lund University, Cognitive Studies.Search in Google Scholar
Johansson, Petter, Lars Hall, Sverker Sikström & Andreas Olsson. 2005. Failure to detect mismatches between intention and outcome in a simple decision task. Science 310(5745). 116–119. https://doi.org/10.1126/science.1111709.Search in Google Scholar
Johansson, Petter, Lars Hall, Betty Tärning, Sverker Sikström & Nick Chater. 2013. Choice blindness and preference change: You will like this paper better if you (believe you) chose to read it. Journal of Behavioral Decision Making 27(3). https://doi.org/10.1002/bdm.1807.Search in Google Scholar
Kendon, Adam. 1981. Nonverbal communication: Interaction and gesture. The Hague: Mouton Publisher.10.1515/9783110880021Search in Google Scholar
Kendon, Adam. 2004. Gesture: Visible action as utterance. Cambridge: Cambridge University Press.10.1017/CBO9780511807572Search in Google Scholar
Kendon, Adam. 2008. Some reflections on the relationship between ’gesture’ and ’sign. Gesture 8. 348–366. https://doi.org/10.1075/gest.8.3.05ken.Search in Google Scholar
Koffka, Kurt. 1935. Principles of gestalt psychology. London: Taylor & Francis.Search in Google Scholar
Konderak, Piotr. 2018. Mind, cognition, semiosis: Ways to cognitive semiotics. Lublin: Maria Curie-Sklodowska University Press.Search in Google Scholar
Ladewig, H. Silva. 2014a. Recurrent gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill & Sedinha Teβendorf (eds.), Body-language-Communication, Vol. 2, 1558–1574. Berlin: Walter de Gruyter. https://doi.org/10.1515/9783110302028.1558.Search in Google Scholar
Langacker, W. Ronald. 1987. Foundations of cognitive grammar: Theoretical prerequisites, Vol. I. Stanford: Stanford University Press.Search in Google Scholar
Langacker, W. Ronald. 2006. Subjectification, grammaticization, and conceptual archetypes. In A. Athansiadou, C. Canakis & B. Cornillie (eds.), Subjectification: Various paths to subjectivity, 17–40. Berlin: de Gruyter.Search in Google Scholar
Libet, Benjamin. 2005. Mind time: The temporal factor in consciousness. Harvard University Press.10.4159/9780674040168Search in Google Scholar
McNeill, David. 2005. Gesture and Thought. Chicago: University of Chicago Press.10.7208/chicago/9780226514642.001.0001Search in Google Scholar
Mahl, F. George. 1968. Gestures and body movements in interviews. In John Shien (ed.), Research in psychotherapy, vol. III. Washington, D. C.: American Psychological Association.10.1037/10546-016Search in Google Scholar
Mendoza-Collazos, Juan. 2022. Agency and artefacts: A cognitive semiotic exploration of design. Lund: Lund University.Search in Google Scholar
Merleau-Ponty, Maurice. 1962. Phenomenology of perception. London: Routledge.Search in Google Scholar
Morris, Desmond. 1994. Bodytalk: A worldguide to gesture. London: Jonathan Cape.Search in Google Scholar
Mouratidou, Alexandra. 2020. Choice awareness and manipulation blindness. Public Journal of Semiotics 9(1). 1–40. https://doi.org/10.37693/pjos.2019.9.21388.Search in Google Scholar
Mouratidou, Alexandra, Jordan Zlatev & Joost van de Weijer. 2022. How much do we care what we pick? Pre-verbal and verbal investment in choices concerning faces and figures. Topoi 41. 695–713. https://doi.org/10.1007/s11245-022-09807-z.Search in Google Scholar
Müller, Cornelia. 2017. How recurrent gestures mean: Conventionalized contexts-of-use and embodied motivation. Gesture 16(2). 276–303.10.1075/gest.16.2.05mulSearch in Google Scholar
Overgaard, Morten. 2006. Special issue on introspection. Consciousness and Cognition 15(4). 629–633. https://doi.org/10.1016/j.concog.2006.10.004.Search in Google Scholar
Partee, H. Barbara. 1984. Compositionality. Varieties of Formal Semantics 3. 281–311.Search in Google Scholar
Petitmengin, Claire. 2009a. Ten years of viewing from within. Journal of Consciousness Studies 16. 10–12.Search in Google Scholar
Petitmengin, Claire. 2011. Describing the experience of describing? The blind spot of introspection. Journal of Consciousness Studies 18(1). 44–62.Search in Google Scholar
Petitmengin, Claire & Michel Bitbol. 2009. The validity of first-person descriptions as authenticity and coherence. Journal of Consciousness Studies 16(10–12). 363–404.Search in Google Scholar
Pielli, Laura & Jordan Zlatev. 2020. The cyborg body: Potentials and limits of a body with prosthetic limbs. Cognitive Semiotics 13(2). https://doi.org/10.1515/cogsem-2020-2033.Search in Google Scholar
Poggi, Isabella. 2002. Symbolic gestures. The case of the Italian gestionary. Gesture 2(1). 71–98. https://doi.org/10.1075/gest.2.1.05pog.Search in Google Scholar
Poyatos, Fernando. 1981. Gesture inventories: Fieldwork methodology and problems. In Adam Kendon (ed.), Nonverbal communication, interaction, and gesture. Selections from Semiotica, 371–400. The Hague: Mouton.10.1515/9783110880021.371Search in Google Scholar
R Core Team. 2024. R: A Language and Environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.Search in Google Scholar
Stampoulidis, George, Marianna Bolognesi & Jordan Zlatev. 2019. A cognitive semiotic exploration of metaphors in Greek street art. Cognitive Semiotics 12(1). https://doi.org/10.1515/cogsem-2019-2008.Search in Google Scholar
Stec, M. K. Kashmiri. 2015. Visible quotation: The multimodal expression of viewpoint. University of Groningen PhD Thesis.Search in Google Scholar
Streeck, Jurgen. 2009. Gesturecraft: The manu-facture of meaning. Amsterdam: John Benjamins.10.1075/gs.2Search in Google Scholar
Sokolowski, Robert. 2000. Introduction to phenomenology. Cambridge: Cambridge University Press.Search in Google Scholar
Sonesson, Göran. 2009. The view from husserl’s lectern: Considerations on the role of phenomenology in cognitive semiotics. Cybernetics and Human Knowing 16(3-4). 107–148.Search in Google Scholar
Sonesson, Göran. 2010. Rhetoric from the standpoint of the lifeworld. In Göran Sonesson & Maria Gulia Dondero (eds.), Le Groupe Mu. Quarante ans de recherche collective. Paris: Nouveaux actes sémiotiques.10.25965/as.3106Search in Google Scholar
Sonesson, Göran. 2020. That there is more to the medium than the message: Further considerations on the cognitive semiotics of lifeworld mediations. In Lars Grabbe, PatrickRupert- Kruse & Norbert Schmitz (eds.), Bildgestalten: Topographien medialer visualität, 39–56. Darmstadt: Büchner-Verlag eG.Search in Google Scholar
Varela, J. Francisco & Jonathan Shear. 1999. The View from Within. First-person approaches to the study of consciousness. Darmstadt: Imprint Academic.Search in Google Scholar
Varela, J. Francisco, Evan Thompson & Eleanor Rosch. 1991. The embodied Mind: Cognitive Science and human experience. Cambridge, MA: MIT Press.10.7551/mitpress/6730.001.0001Search in Google Scholar
Wegner, M. Daniel. 2018. The Illusion of conscious will. MA: MIT Press.10.7551/mitpress/9780262534925.001.0001Search in Google Scholar
Wittenburg, Peter, Hennie Brugman, Albert Russel, Alex Klassmann & Sloetjes Han. 2006. Elan: A professional framework for multimodality research. In Paper presented at the 5th international Conference on language Resources and evaluation (LREC 2006). Genoa, Italy: European Language Resources Association (ELRA).Search in Google Scholar
Zahavi, Dan. 2018. Phenomenology: The basics. London: Routledge.10.4324/9781315441603Search in Google Scholar
Zlatev, Jordan. 1997. Situated embodiment: Studies in the emergence of spatial meaning. Stockholm: Stockholm University PhD Thesis.Search in Google Scholar
Zlatev, Jordan. 2009. The Semiotic Hierarchy: Life, consciousness, signs and language. Cognitive Semiotics 4. 169–200. https://doi.org/10.3726/81608_169.Search in Google Scholar
Zlatev, Jordan. 2015. The emergence of gestures. In Brian MacWhinney & William O’Grady (eds.), The Handbook of language emergence, 458–476. Malden, Mass: John Wiley & Sons.10.1002/9781118346136.ch21Search in Google Scholar
Zlatev, Jordan. 2018. Meaning making from life to language: The Semiotic Hierarchy and phenomenology. Cognitive Semiotics 11(1). https://doi.org/10.1515/cogsem-2018-0001.Search in Google Scholar
Zlatev, Jordan. 2019. Mimesis theory, learning and polysemiotic communication. In Michael A. Peters (ed.), Encyclopedia of educational Philosophy and theory. Cham: Springer.10.1007/978-981-287-532-7_672-1Search in Google Scholar
Zlatev, Jordan, Przemyslaw Żywiczyński & Sławomir Wacewicz. 2020. Pantomime as the original human-specific communicative system. Journal of Language Evolution. 156–174. https://doi.org/10.1093/jole/lzaa006.Search in Google Scholar
Zlatev, Jordan. & Piotr Konderak. 2022. Consciousness and semiosis. In Jamin Pelkey (ed.), Handbook of semiotics. London: Bloomsbury Press.Search in Google Scholar
Zlatev, Jordan, Simon Devylder, Rebecca Defina, Kalina Moskaluk & Linea B. Andersen. 2023. Analyzing polysemiosis: Language, gesture, and depiction in two cultural practices with sand drawing. Semiotica 253. 81–116. https://doi.org/10.1515/sem-2022-0102.Search in Google Scholar
Zlatev, Jordan & Alexandra Mouratidou. 2024. Extending the life world: Phenomenological triangulation along two planes. Biosemiotics. 17(2). 407–429. https://doi.org/10.1007/s12304-024-09576-9.Search in Google Scholar
© 2024 the author(s), published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.
Articles in the same Issue
- Frontmatter
- Articles
- Distributed cognition in oral poetry improvisation: a semiosis-centered approach
- Externalization of the mind in the meaning-making process: An integrated semiotic and cognitive science perspective
- The virtual habits underlying the behavioural hallmarks of alloanimal episodic memory: a Peircean model
- The body says it all: Non-verbal indicators of choice awareness
Articles in the same Issue
- Frontmatter
- Articles
- Distributed cognition in oral poetry improvisation: a semiosis-centered approach
- Externalization of the mind in the meaning-making process: An integrated semiotic and cognitive science perspective
- The virtual habits underlying the behavioural hallmarks of alloanimal episodic memory: a Peircean model
- The body says it all: Non-verbal indicators of choice awareness
