Multimodal mitigation: how facial and body cues index politeness in Catalan requests

2.1 Politeness and mitigation devices

Since requests are face-threatening acts (Brown and Levinson 1987), mitigation devices have often been studied in this context, and one way in which politeness can be evoked is by mitigating the potential threat. Following Caffi (2007), mitigation is a cover term used for a set of strategies which are devised to soften, attenuate, or alleviate different aspects of a person’s speech. Such mitigation strategies involve the use of not only lexical and morphosyntactic but also prosodic and gestural markers.

According to the pragmatics literature, mitigation strategies can modify a speech act either internally or externally (Blum-Kulka et al. 1989). External modification affects not the utterance as such but rather the context in which that particular speech act occurs. This type of mitigation is carried out through various supportive moves or devices. For example, in the case of a request to borrow someone’s car, the request proper (“Could I borrow your car?”) might be preceded (or followed) by an alerter (“Maria”), a preparator (“last week my car broke down”), and an expression of anticipatory gratitude (“I would be very grateful”). Internal modifications, on the other hand, help to downgrade the force of the potentially negative effect (Faerch and Kasper 1989; House and Kasper 1981). This type of mitigation can be encoded lexically through politeness markers – such as, in English, the word please, mental state verbs such as think, or modal adverbs such as maybe, probably, etc. – or syntactically through conditional or interrogative sentence structure. The softening and downgrading of the negative effects of face-threatening speech acts is the main function of mitigation (Caffi 1999; Fraser 1990). Interestingly, as Blum-Kulka et al. (1989) contend that both external and internal modifiers are context-specific and therefore not inherently polite (contrary to Brown and Levinson’s [1987] theory, which treated mitigation as a synonym for politeness).

In connection with the verbal expression of politeness in Catalan, Hübscher et al. (2017) showed (like Payrató and Cots 2011) that the informal/intimate pronoun tu ‘you’ was used more frequently in conditions where politeness was not required, whereas the more formal/polite pronoun vosté ‘you’ was more frequent in situations requiring politeness. Informal requests also showed a greater use of imperatives and the indicative mood, whereas formal requests were characterized by an increased use of indirect interrogatives. However, as will be seen in the following subsection, more recent studies (as well as Hübscher et al. 2017, itself) have shown that mitigation is also expressed in Catalan through prosody.

2.2 The prosody-politeness interface: prosodic mitigation strategies

Pitch, as one crucial element of prosody, has different functions in speech, and one of them is to signal social and affective functions such as dominance, certainty and importantly for the present research, politeness (see e.g., Culpeper et al. 2003). The close nature of the relationship between politeness and prosody was first set out by Ohala (1984) in his Frequency Code Hypothesis, according to which politeness is universally associated across languages with higher pitch. Though some studies have found confirmation of this hypothesis (Chen et al. 2004; Herraiz and Serena 2014; Orozco 2010; Tsuji 2004), another strand of research has shown that in some languages, such as Korean (Winter and Grawunder 2012), Japanese (Idemaru et al. 2019), German (Grawunder et al. 2014) and Catalan (Hübscher et al. 2017), an overall lower pitch marks politeness-related meanings. These last four studies all used an oral discourse completion task to elicit requests in socially distant contexts (e.g., requesting something from a boss or professor) and socially close contexts (e.g., asking a friend for a favor). Audio recordings of participant responses were then analyzed in terms of: fundamental frequency, duration, intensity (loudness), and voice quality. Across the four studies, in the polite condition the speech rate decreased, there were more hesitations, syllables were longer, intensity decreased, and jitter and shimmer decreased, while the difference in amplitude between first and second harmonics (H1–H2) increased. Regarding F0, the polite condition was characterized by lower pitch values, thus contradicting Ohala’s Frequency Code Hypothesis for these languages (at least in the sorts of contexts where there is an asymmetrical power relationship like worker-to-boss or student-to-professor, as also demonstrated in Winter et al. [2021]).

While the above-mentioned studies have focused on the speaker, it is also important to investigate the effects that the speaker’s prosody has on the interlocutor’s perception. The latter was the focus of Brown et al. (2014), who investigated whether native speakers of Korean could detect the formality level of Korean utterances without any explicit verbal markers, and whether American English listeners without any knowledge of Korean could do the same. The stimuli consisted of identical requests read out loud in both honorific (contaymal) and non-honorific (panmal) registers by eight native Korean speakers. The results showed that when listeners had to detect which request was polite and which one informal, both natives and foreigners were able to distinguish between the two registers purely on the basis of the varying phonetic cues pertaining to these two different conditions.

Regardless of whether it confirms or refutes the Frequency Code Hypothesis, this evidence from production and perception studies shows that prosody must be included among the mitigation strategies characterizing socially distant speech addressed to superiors. The question we will ask in this paper is whether multimodal cues also play a role in mitigation. In the following section we review the current state of the art on the relationship between multimodal cues and politeness.

2.3 Multimodal expression of politeness: gesture mitigation strategies?

Over the last few years there has been growing recognition that multimodal cues play a role in the expression of politeness-related meanings. For example, Kita’s (2009) review of gesture research includes a section which describes how rudeness can be conveyed in many European and American cultures with specific gestures such as extending the middle finger. Indeed, in most cultures, particular gestures are regarded as impolite because of their cultural significance (Kita and Essegbey 2001). For example, pointing with the left hand is considered rude in Ghana and thus avoided in a communicative interchange such as giving route directions. While gesture studies have often focused on the use of manual gestures without taking into account other facial and bodily cues, studies from the field of social psychology – as pointed out by Brown and Winter (2019) – have dealt extensively with a variety of facial and bodily markers of politeness-related phenomena, focusing on psychological notions such as power, status, and intimacy (i.e., social distance). Power refers to the perceived dominance dynamic between speaker and hearer, whereas status refers to one’s positioning in a particular social hierarchy (Ellyson and Dovidio 1985). Though there seems to be no one-to-one match between psychological constructs and the different categories used in politeness research, some key factors have been analyzed in relation to the communication of politeness. Power in particular has been found to be a reliable predictor of politeness-related behavior (Goldsmith 2007: 227). It has been found that power is expressed through a range of multimodal behaviors implying strength or fearlessness, while submissiveness is expressed by means of behaviors implying weakness, tension, and fearfulness (Brown and Winter 2019; Mehrabian 1981). Visual cues associated with submissiveness or uncertainty/hedging, seen as any sort of self-withdrawal or personal submission to the hearer, are central in the dialectic expression of power and humility with respect to the higher-ranking person. Previous studies have shown that speakers of lower status express submissiveness by adopting rigid postures with their head in a downturned position or by leaning forward (Brown and Winter 2019; Harper 1985) and maintaining a direct body orientation facing the interlocutor (Brown and Winter 2019; Burgoon and Saine 1978; Jorgenson 1975; Mehrabian 1968). These expressions of submissiveness constitute what Trees and Manusov (1998) call mitigating nonverbal behavior. These authors describe various such behaviors used by English-speaking Americans to express politeness, including touching, close proximity to the interlocutor, direct body orientation, pleasant facial expressions, raised eyebrows, a softer voice, and a tense and closed position with tight, small gestures. In contrast, aggravating nonverbal behavior, used to display impoliteness, includes no touching, greater distance from the interlocutor, indirect body orientation, unpleasant facial expressions, furrowed eyebrows, a loud voice, and wide gestures. Mitigation is thus a multimodal phenomenon where speakers make use of both body/face/voice and words to lessen their degree of imposition on the listener.

Unsurprisingly, studies in the field of linguistics have traditionally focused predominantly on the lexical and morpho-syntactic devices speakers have at their disposal to reduce their level of commitment or imposition on the hearer. Hedging devices are typically one of these strategies. Hedges are both cautious and interactive devices which allow the construction of a relationship between the speaker/writer and listener/reader by moderating the degree of certainty or imposition with which they present their knowledge claims or requests. Hence, hedging devices have been exhaustively analyzed in scientific writing, but due to their capacity to reduce imposition on the listener, they have recently also received attention in the politeness literature. As a matter of fact, using hedges helps speakers to save face by giving the impression that they are not strongly committed to a proposition, hence softening the degree of imposition on the hearer.

It is thus of interest to analyze markers of low commitment (i.e., uncertainty) – both body/face/voice signals and lexical and morphosyntactic hedging – as potential politeness markers. In other words, if lexical and morphosyntactic hedges can serve as devices to simultaneously convey low commitment/uncertainty and politeness, we must assume that the same body signals that communicate uncertainty can also convey politeness. In this case, the speaker’s goal is not to lower his or her level of commitment to his statements, but to lower the degree of imposition on the addressee. For instance, the verbal marker “I don’t know” can function as a marker of uncertainty/lack of knowledge, but it can also convey a lowering of the speaker’s level of imposition on the hearer. On the multimodal plane, the shoulder shrug can serve the same dual functions. It can convey either lack of knowledge/uncertainty or lack of imposition on the interlocutor. This overlap between signals communicating lack of knowledge/uncertainty and lack of imposition suggest that we should encounter multimodal signals of uncertainty like shoulder shrugs in situations that require politeness. Following Givens (2002) and also Swerts and Krahmer (2005), we would also expect to find other facial and bodily markers of uncertainty such as head shakes and tilts, frowns and pouted or pursed lips.

In the present study we hypothesize that these low commitment gestural strategies can be regarded as mitigating devices that will occur more frequently in polite conditions, together with a less frequent use of body and facial cues. To our knowledge, only one previous study has investigated multimodal politeness, in this case in Korean. Brown and Winter (2019) found that specific multimodal cues were used in deferential situations, together with a reduction in the magnitude of facial and bodily cues. The study analyzed gestural behavior in two different but related relational practices: “doing deference” when interacting with a status superior, and “doing intimacy” when interacting with a status equal. They analyzed 154 interactions from Korean televised dramas which contained both types of situations. Their results showed that the participants displayed significantly different behaviors in the two practices. When “doing deference” with a status superior, inferiors displayed erect but constrained body positions, used a direct bodily orientation, and used fewer manual gestures, fewer raised and furrowed eyebrows or adaptors (self-touches) and fewer haptics (i.e., touching the interlocutor). When “doing deference” there was also a decrease in animatedness and freedom of movement as compared to when “doing intimacy”. Interestingly, the authors also found that head nods and smiles were equally used when performing intimacy and when performing deference. So far little is known about how such multimodal patterns behave in typologically different languages.

2.4 The present study: research questions and hypotheses

As mentioned above, recent research on Catalan has shown that when addressing a person of higher status, not only the verbal domain but also the acoustic/prosodic domain is adapted, through the use of prosodic mitigation strategies (Hübscher et al. 2017). However, very little is known as yet about how multimodal cues work together to convey politeness in socially distant encounters in this language or more generally in Romance languages. In order to be able to provide a comprehensive picture of Catalan speakers’ politeness-related behaviors, in this paper we provide a multimodal analysis of Catalan requests by analyzing in a quantitative way a wide array of body and facial gestural patterns during not only the head act (i.e., the actual request) but also the preparatory and post-request stages. We will contrast the behaviors produced by participants in two simulated conditions: a “polite” condition, in which they imagine they are addressing a much older professor, and a “non-polite” condition, in which the imagined interlocutor is a similarly-aged friend (it should be noted that we use the labels “polite” and “non-polite” here for convenience and are not to be understood as direct interpretations of how the two levels of speech are to be interpreted in context). Importantly, the use of an oral discourse completion task will allow us to manipulate social distance as an experimental variable control of social distance.

First, we will test whether Catalan speakers combine gestural mitigation strategies with the prosodic mitigation strategies reported by Hübscher et al. (2017) for the same database (decreases in speech rate, intensity, or pitch values). Our hypothesis is that Catalan speakers will display a behavior similar to that shown by Korean speakers and adopt a gestural mitigation strategy consisting of a reduction in the number of facial and bodily cues (Brown and Winter 2019). In our view, gesture suppression is a form of gestural mitigation since the production of fewer facial and body movements is a reflection of more restricted and less animated behavior.

Second, we will analyze how the use of body cues indicating speaker’s level of commitment (such as shoulder shrugs, eyebrow frowning, etc.) or submission (such as smiling, raised eyebrows, etc.) varies according to the social distance between interlocutors. Similar to Trees and Manusov’s (1998) study, we hypothesize that Catalan speakers will use a greater number of body cues associated with uncertainty/submissiveness when addressing a person who is socially distant to them as compared to a friend to whom they feel close.

3.1 Participants

Twenty female Catalan-dominant speakers (mean age = 21.00, SD = 3.72) participated in the experiment. Only females were invited to participate in order to obtain a more homogeneous sample. All of them were students at the Universitat Pompeu Fabra in Barcelona and reported Catalan to be their dominant language relative to Spanish (mean percentage of Catalan daily usage = 75.83%, SD = 13.62%). Participants received a small payment for participating in the experiment.

3.2 Materials

As noted above, the speech act under study was the request. In order to elicit requests, we used the same oral Discourse Completion Task (DCT). DCTs present the participants with the written description of a situation which is intended to trigger a response while at the same time controlling for different contextual variables (Billmyer and Varghese 2000; Blum-Kulka et al. 1989; Félix-Brasdefer 2010; Vanrell et al. 2018). Also, since we were interested in comparing our results to previous studies on politeness and prosody in Korean, German, Catalan, and Japanese (Grawunder et al. 2014; Hübscher et al. 2017; Idemaru et al. 2019; Winter and Grawunder 2012), we chose to make use of the same methodology as that used in those experiments.

The DCT consisted of 12 discourse contexts, six in the polite condition and six in the non-polite condition, which were created in order to elicit requests (see Table 1 for an example). To be able to obtain comparable data sets across the two conditions, we controlled for the cost of the action requested, but varied the pragmatic factor (social distance) in the two conditions: in other words, requests addressed to an intimate represented less (−) social distance and requests addressed to a much older status superior represented greater (+) social distance. Furthermore, in the descriptions of the contexts it was made clear that the interlocutors were female in all cases. The complete set of discourse contexts is shown in the Appendix. Each discourse prompt was printed on a card and accompanied by a photo illustrating the situation described.

3.3 Procedure

The DCT task was video-recorded at the Linguistics Laboratory at the Department of Translation and Language Sciences of the Universitat Pompeu Fabra in Barcelona, Catalonia. Participants were invited to read the instructions and then were given the 12 cards containing the contextual prompts and accompanying photos, with the six non-polite situations being given first (to avoid eventual layover effects from the polite condition on the non-polite condition) in random order and then the six polite situations (also in random order) coming after.

After reading each card, the students were asked to place it face down on the table and then to produce a request as spontaneously as possible while facing the video camera (a Panasonic 3MOS HD-AVCCAM). Participants also wore a clip-on microphone. The sound sample rate was 44.100 Hz with 16-bit quantization.

A total of 239 audio-visually recorded requests (6 situations × 20 participants × 2 conditions) were obtained. In one case, one of the 12 situations was accidentally overlooked. The total duration of the requests amounted to 3 h, 31 min and 8 s of recorded speech. The dataset obtained was the same as that analyzed in Hübscher et al. (2017), the difference being that in that case only the audio tracks of the recorded requests were analyzed for prosodic content, whereas here we looked at both audio and visual tracks to analyze gestural as well as verbal behaviors.

3.4 Multimodal labeling

The 238 target requests were labeled for multimodal behavior by the second author using ELAN version 4.9.4 (Lausberg and Sloetjes 2009) (see Table 2). Importantly, the multimodal behavior was labeled for both the head act as well as the modification devices. In order to create a comprehensive profile of politeness-related multimodal behaviors, we assessed an extensive range of categories identified in previous studies, following especially the coding system proposed in Brown and Winter (2019). Apart from that, our coding system was primarily based on the MUMIN (Multimodal Interfaces) scheme developed by Allwood et al. (2007). For those facial gestural cues that are not included in the MUMIN, we used elements from the FACS (Facial Action Coding System) by Ekman et al. (2002), and for certain body postures we used the Body Action and Posture Coding System (BAP) by Dael et al. (2012). More specifically, we took note of facial and body cues which have been reported in previous studies as interacting with mitigation behavior, power, and submission. By the same token, we paid special attention to body postures which have been documented to arise in states of speaker submission and speaker uncertainty.

Table 2:

Annotated facial and body cues.

#	Tier	Gesture	Example	Description
1	Head	Down		Movement of the head downwards.
		Forward		Movement of the head forward.
		Nod		Down-up head movement once or twice in close succession.
		Shake		Repeated rotation of the head from side to side.
		Tilt		Tilting movement of the head to one side or from side to side.
		Turn		Rotation of the head toward the left or right side.
2	Eyebrows	Furrowed		Eyebrows lowered and drawn together, leading to vertical wrinkles between the eyebrows and possibly one or more horizontal wrinkles just under the nose.
		Raised		Eyebrows pulled upwards, resulting in an arched, curved shape.
3	Eyelids	Closed		Closing of the eyes longer than a blink.
		Squinted		Eyelids tightened and the eye aperture is narrowed. The lower lid is raised so it covers more of the eyeball than usually.
4	Mouth + Nose	Smile		Lip corners pulled up towards the cheek bone in an oblique direction.
		Lip corners down		The corners of the lips are drawn down.
		Horizontally stretched lips		Movement of the lips horizontally.
		Pressed lips		The upper lip and the lower lip press against each other.
		Nose wrinkle		The skin along the sides of the nose is pulled upwards towards the root of the nose, causing wrinkles to appear along the sides of the nose and just under the nose.
5	Shoulders	Symmetric shoulder shrug		Both shoulders move upward at the same time.
		Asymmetric shoulder shrug		One shoulder moves upward.
		Slumped shoulders		Shoulders shrug and move forward.
6	Trunk	Forward		Movement of the trunk forward.
		Lateral leaning		Movement of the trunk to on one side or from side to side.
		Side rotation		Rotation of the trunk toward the left or right side.

Facial and bodily cues were annotated in ELAN 4. 9. 4 in accordance with Allwood et al. (2007) and Dael et al. (2012). Table 2 illustrates a total of 21 distinct facial and bodily cues which were coded in ELAN as separate tiers.^[1]

3.5 Statistical analyses

For statistical analyses, we used IBM SPSS v24. First, we were interested in whether the discourses produced during the two conditions differed in length. To determine whether any difference in durations would be statistically relevant, a Generalized Linear Mixed Model (GLMM) was run in which this variable was set as the dependent variable (Gamma distribution, Log link), with Condition (polite vs. non-polite) as a fixed factor. Concerning random factors, the model included a random slope for Condition by Subject, and a random intercept for Item. Indeed, Condition yielded a significant result, F (1, 236) = 68.109, β = −0.509, SE = 0.062, p < 0.001, indicating that discourses in the polite condition were longer than in the non-polite one. Because of this, for each recorded discourse, we calculated the number of instances of every type of visual cue, which was then divided by the duration of the discourse in seconds.

A main GLMM was carried out to compare whether the 21 labeled visual cues occurred in different amounts in the two conditions. As noted above, since the discourses in the two conditions differed in discourse length, the dependent variable was the number of visual cue occurrences per second (Gamma distribution, Log link). The fixed factors were Condition (polite, non-polite), Visual Cue (21 different facial and bodily cues in total; see Table 2), and their paired interaction. A random slope was set for both Condition and Visual Cue by Subject. Consequently, in this model, the main effect of Condition would inform us about significant differences in the number of visual cues produced across conditions, taking into account individual variation and the different visual cues that have been labeled. The interaction Condition × Visual Cue would indicate, in turn, whether each gesture label is more frequently present in one condition compared to the other.