A cross-cultural analysis of the gestural pattern of surprise and surprise-disapproval questions

1 Introduction

In this study, we analyze two types of special questions, i.e., counter-expectational questions that convey surprise and surprise-disapproval questions, from a cross-cultural perspective.^[1] We compare the way these questions are expressed in languages such as Italian, Neapolitan, German, and Spanish, with their equivalents in oriental languages such as Japanese, Vietnamese, and Korean. As a theoretical framework, we rely on the Minimalist program, as elaborated by Chomsky (1995, 2000, 2001, 2008) and scholars, and capitalize on Giorgi’s (2016, 2018) work on the syntax of adversativity.

Surprise and surprise-disapproval questions are considered special questions, in that they are not primarily a request for information and, therefore, do not require a canonical answer. Rather, when asking a special question, the speaker has the double goal of venting their feelings on one side and eliciting an explanation for the surprising and disappointing behavior of the interlocutor on the other.^[2]

An exemplification of surprise question is elicited by the following scenario:^[3]

A possible scenario for surprise-disapproval questions is the following one:

Interestingly, these sentences would be judged as infelicitous and ungrammatical if not conveyed with the correct intonation and gestures. This shows that the non-syntactic components play a crucial role in assigning the correct interpretation.^[4]

As we will briefly discuss in the next section, Giorgi and Dal Farra (2019) designed some experiments to check the prosodic and gestural realization of these sentences, in correspondence to specific cues in the syntactic representation. Later on, Dal Farra, Giorgi and Hinterhölzl (forthcoming) replicated the experiments for German, Furlan (2019) for Spanish and Marchetiello (2022) for Neapolitan.^[5] The result of this experimentation shows that across languages these sentences, despite the syntactic differences, share the same properties in two respects (a) alignment, in that syntax, prosody, and gesture are aligned in all languages and (b) the gestural pattern, in that the inventory of co-speech gestures adopted in these cases by speakers across languages is largely the same.

The research question arising at this point concerns the cultural specificity of these invariants. It is possible to argue in fact that the results obtained in the aforementioned studies are due to cultural uniformity, given that all the languages mentioned above belong to the Western world, and in particular to the European. The issue therefore concerns languages belonging to cultures very far away from Europe: do they exhibit the same invariants? And if so, to what extent?

It is widely acknowledged that there are differences in the way people use gestures across different cultures – for a review, see Kita (2009) and Schröder and Streeck (2022). However, certain aspects, such as alignment (McNeill 2000), seem to be universal. Since gestures and speech are closely linked, gaining a better understanding of the universality of gestures can help us improve our knowledge of language and cognition.

This article is organized as follows: in Section 2, we summarize the results obtained in the research mentioned above, in Section 3, we provide a brief overview of the literature on the relevant gestures, in Section 4 we present the experimental plan in Japanese, Korean and Vietnamese, in Section 5 we present a discussion of the results and finally in Section 6 we draw some conclusions.

2 Previous research

In this section, we briefly summarize the results obtained so far on surprise questions in Italian, Neapolitan, Spanish and German.

From a syntactic point of view, these sentences are in general introduced by an adversative particle: ma in Italian and Neapolitan, aber in German, pero in Spanish. Surprise-disapproval questions are open questions, whereas counter-expectational questions are yes/no questions – cf. examples (1) and (2) above.

As far as the gestural component is concerned, the authors found that the manual gestural component is uniform across languages, showing a predominant presence of the Palm Up Open Hands (PUOH) gesture (see Kendon 2004), exemplified in Figure 1 (from Giorgi and Dal Farra 2019: 346).

Figure 1:

PUOH.

The process of producing gestures can be analyzed in different phases (Kendon 1980): preparation, stroke, and retraction. The preparation phase occurs when hands are moved from a resting position to a more visually prominent position, i.e. the stroke position. The stroke marks the point of maximal expansion and can be identified as such. Following the stroke, there is typically a retraction phase, where the hands return to their original resting position. Additionally, there is a fourth phase called hold, which refers to the moment when the hands remain static in the gestural phase after the stroke and before the retraction. In the production of the hand gesture, the preparation phase tends to precede the uttering of the sentence and PUOH often lasts longer than the sentence itself, in that speakers hold the gesture until the very end of the sentence. The retraction phase starts only when the sentence has been uttered.

Nonmanual gestures are mostly head nod or shake, and raised or furrowed brows, as exemplified in Figures 2 and 3 (from Giorgi and Dal Farra 2019: 348–349).

Figure 2:

Eyebrows raised.

Figure 3:

Eyebrows furrowed.

Notably, in all languages, a significant alignment has been observed between the stroke of PUOH, and/or the head movement and the leftmost pitch accent, which typically characterizes this kind of sentences: the stroke of the gestures is usually realized in correspondence with the pitch on the nuclear syllable of the verbal form and/or on the negation.^[6] Some differences were also observed, as in German native speakers realized PUOH less frequently than the speakers of the other languages and with arms nearer to the body.

In the realization of counter-expectational questions speakers realize one among three different hand gestures: PUOH, as above, the so-called artichoke gesture and the hands in prayer gesture, exemplified in Figures 4 and 5 (from Giorgi and Dal Farra 2019: 355–356).

Figure 4:

The artichoke.

Figure 5:

Hands in prayer.

As observed with surprise questions, hand gestures often last longer than the sentence they are realized with. Interestingly, hand movements in these sentences are characterized by a rapid up-down iteration. In other words, the hands move quickly and repeatedly from a preparation position to a stroke position. According to the authors, the stroke position, in this case, is the moment when the hands assume their final shape, i.e., the moment of maximal expansion, just before the repetition begins. Non-manual gestures in surprise-disapproval questions are less frequent than with surprise questions and are mainly expressed through movements of the eyebrows, which are typically furrowed, and shifts of the head towards the front or side. Similarly to surprise questions, there is alignment between prosody and gestures, with a significant correlation between the stroke and the pitch on the nuclear syllable of the verb and/or on the wh- phrase.

Thus, in all the languages mentioned above, these questions are characterized by a peculiar syntax, a special prosody and a typical gestural pattern, both manual – i.e., movements of the hands – and non-manual – i.e. movements of head, eyes and brows.

This research is elaborated in the theoretical framework of the Minimalist approach to language, as developed by Chomsky and other scholars (Chomsky 1995, 2000, 2001, 2008). According to this theoretical hypothesis, there is no direct link between the interpretation of a sentence, i.e., its meaning, and its phonological and prosodic realization, i.e. the sound corresponding to its representation. The relation between the two is necessarily mediated by a representation of the sentence, called core syntax. Thus, the syntactic representation of a sentence interfaces with the sensorimotor component, which yields its phonological and prosodic form, and with the conceptual system, which gives rise to its interpretation.

Giorgi and Dal Farra (2019) propose that prosody and gesture are simultaneously activated at the interface between syntax and the sensorimotor component and that the input to the sensorimotor component for prosody and gesture realization is unique. Prosody and gesture are both triggered by the same syntactic property, i.e., the presence of a left-peripheral Evaluative syntactic projection, realized in the left periphery of the clause. Simplifying a complex discussion, it is possible to say that the evaluative syntactic projection is what represents in the syntax the emotional value of sentences.

The question concerning cultural closeness naturally arises at this point. Both Italian and Spanish are part of the Romance language family and share similarities due to their Mediterranean heritage. The historical interactions between these two cultures have facilitated cultural exchanges, leading to shared gestures and customs. However, German, which belongs to the Germanic language family and is situated in Central Europe, has a distinct cultural context. Its historical development sets it apart from the cultural tapestries of Italy and Spain. Furthermore, Germany’s historical affiliations lean more towards neighboring Central European countries like France and Poland (Clark 2006; Wilson 2016).

As mentioned above, previous research detected some differences between German and the Romance languages examined, in particular in the frequency and the amplitude of the gestures, but both the alignment properties and the quality of the gestural pattern were the same as in the other languages.^[7]

Finally, consider that, in psychological literature, the proposal is that emotions like surprise and disapproval are universally recognized and experienced across diverse human cultures. Research in psychology (Ekman 1973, 1992, 1993, 2003; Ekman and Friesen 1978, 2003; Ekman et al. 2002; Ekman and Davidson 1994) attributes this universality to shared biological and evolutionary characteristics that govern human emotional responses. Thus, the notion that a universal gestural pattern might be generally recognized and acknowledged finds support in psychological research.^[8]

Our exploration bears the potential to investigate the existence of a fundamental, and possibly universal, pattern underpinning these phenomena. To achieve this goal, our work extends to languages culturally distant from the ones heretofore examined, such as Vietnamese, Japanese and Korean.

3 Surprise and disapproval gesture: State of the art

There is a very rich literature on the gestures associated with surprise and disapproval, in particular on PUOH and the artichoke. In this section we will briefly review the works which are most related to our research.

The PUOH gesture has been described as usually associated with practical everyday actions such as giving, offering and receiving objects (Kendon 2004). Although this gesture may have a referential nature while representing such everyday actions, it is often used as a recurrent gesture, as discussed by many authors, e.g., Cienki (2015: 506–508), Harrison (2018: Section 1.4), Kendon (2004: 227), Ladewig (2014), McNeill (2018), Müller (2017), Schröder and Streeck (2022). In this case, it has the primarily pragmatic functions associated with the delivery of information, as argued by Bavelas et al. (1996), Kendon (2004) and Müller (2004). Studies from different theoretical backgrounds have shown that PUOH is often associated with obviousness and shared knowledge – cf. Bavelas et al. (1996), Calbris (1990: 187), Cooperrider et al. (2018), Kendon (2004), Lopez-Ozieblo (2020), Marrese et al. (2021), McNeill (1992: 198) and Müller (2004). According to Kendon (2004), it can be defined also as part of a question that cannot or need not be answered (i.e., a rhetorical question) and Müller (2004) relates the use of the PUOH to the expression of ‘obviousness’.

From a semantic point of view, recent studies also explored the role that tempo and movement features play in the cases of these gestures – cf. Ippolito (2019) and Ferré (2012) – suggesting that these parameters are to be taken into account for further evaluation.

Previous literature on gestures has pointed out that pro-speech and co-speech artichoke gesture – or Mano a Tulipano (tulip hand) gesture MAT, or Mano a Borsa (purse hand) MAB gesture – seems to have an interrogative component (Diadori 1990; Kendon 2004; Poggi 2007). In syntactic literature, the general idea is that the artichoke gesture marks a wh-operator in the constituent questions (Branchini et al. 2015; Colasanti 2023). Interestingly, the artichoke gesture has been found mostly in pragmatically marked questions (Ippolito 2019; Ippolito et al. 2022). According to descriptive literature, this gesture must be regarded as a ‘pragmatic’ gesture, which indicates speech acts and discourse structure (Poggi 1983, 1987; Poggi and Zomparelli 1987).^[9] In particular, it would convey the illocutionary intent of the spoken utterances. For example, the speakers of a Southern Italian dialect spoken in the city of Naples seem to use the artichoke gesture when confronted with something that undermines their expectations being bound with a request of explanation (Kendon 2004). The Mano a Borsa (‘purse hand’ or ‘artichoke gesture’) and the Mani Giunte gesture (the so-called ‘hands in prayer’ configuration) convey the illocutionary intent of the spoken utterances they accompany (Kendon 1995). When used as co-speech gestures, they typically mark non-canonical questions (Ippolito 2019).

The previous literature is consistent with the hypothesis that artichoke gesture, hands in prayer gesture and PUOH, when used as co-speech gestures, mark non-canonical questions, can convey the illocutionary intent of the spoken utterances and are used in the evaluative domain. Our aim is to investigate the role these specific gestures play in a controlled set of data focusing on surprise and surprise-disapproval questions, paying attention to the alignment among prosody, gesture and syntax.

4 Experimental design

To collect evidence about the realization of special questions in Vietnamese, Korean and Japanese, we designed an experiment partially based on the one devised by Giorgi and Dal Farra (2019).^[10]

Our goal is the evaluation of the cultural differences in prosody and, especially, gestures.

Our initial aim was to elicit these structures by means of an elicitation task. As we will briefly discuss, the sentences spontaneously produced by our consultants showed striking formal similarities with the structures studied in Italian, German and Spanish. The sentences obtained by means of the elicitation task, were then used to build a repetition task to submit to other consultants.^[11]

In the elicitation task, we used four specific contexts to introduce a counter-expectational value and four specific contexts to introduce a surprise-disapproval value. After each of them, the participants were asked to utter an appropriate sentence as a reaction to that context. Other than “say it in the most natural way”, no instruction was provided. During the experiment, no reference was made to prosodic or gestural aspects. The participants were videotaped, and the video material was analyzed with ELAN. The audio files have been extracted from the videos, analyzed and annotated with Praat and ToBI system.

We obtained four sentences spontaneously produced as a reaction to surprise contexts and four sentences spontaneously produced as a reaction to surprise-disapproval contexts.

The special questions thus detected have been studied and analyzed in collaboration with native speakers, who were also expert linguists and recognized as rhetorical questions.^[12] Moreover, the (vast majority of the) sentences showed a significant regularity in their lexical, syntactic and interpretive patterns. For the elicitation task, a group of informants were selected. The group consisted of one Korean native speaker, two Japanese native speakers, and one Vietnamese native speaker. The age range of the participants was between 20 and 45 years old.

In the case of the repetition task, our consultants have been presented with four contexts introducing a counter-expectational value and four contexts introducing a surprise-disapproval value. The contexts were the same used in the elicitation experiment. To standardize the administration of the audio scenarios in Korean, Japanese and Vietnamese, the contexts used were read aloud and audio-recorded by a native expert linguist. Then, all the participants were presented with the (same) recorded contexts. After hearing each context, they were presented with the sentence they had to repeat. The sentences were presented in a written form, and no punctuation was indicated. No reference to gestures or intonation has been made in the instructions. The participants were videotaped, and the video material was analyzed with ELAN, whereas the audio files were extracted from the videos, analyzed and annotated with Praat and ToBI system.

For this experiment, a total of 25 participants took part. There were 10 Korean native speakers (5 females and 5 males), 10 Japanese native speakers (4 females and 6 males), and 5 Vietnamese native speakers (1 male and 4 females). The age range of the participants was between 20 and 58 years old.

5 Results and discussion

5.1 Surprise questions

In this section, we will discuss the outcomes of the repetition task on surprise questions across languages. In the repetition task, the consultants were presented with four scenarios, read aloud by a recorded voice, and asked to repeat a sentence provided in written form, after the screen went blank.^[13] As we will show, the results obtained are in line with what was observed for Italian and the other languages.

5.1.1 Surprise questions in Japanese

The following Japanese example is one of the four scenarios, which were presented to the consultants.

(3)

Scenario: You know that your brother reads only detective stories. One day you see him reading ‘War and Peace’. You are surprised and utter:

As far as the scenario given above is concerned, Japanese speakers spontaneously uttered sentences (4) and (5):

(4)

Hee	suiri-shosetsu-shika	yomanai-nja-nakatta?
Hee	detective-stories-only	read-Neg.PAST.Q
‘But weren’t you reading just detective stories?’

(5)

Tokorode	suiri-shosetsu-shika	yomanai-nja-nakatta?
By the way	detective-stories-only	read-Neg.PAST.Q
‘But weren’t you reading just detective stories?’

In the vast majority of the cases, Japanese speakers introduced surprise questions by means of the discourse particle hee, which conveys the meaning of surprise (Mori 2006). Furthermore, these sentences are characterized by the presence of negation and the past marker, nja-nakatta. According to traditional grammar, this combination of negation and the past marker is typical of rhetorical questions.^[14]

In some cases, in particular when the formal register is preferred, these sentences can be introduced by tokorode, as in sentence (5). Tokorode, can be omitted.^[15] The traditional grammars translate tokorode as ‘by the way’ in the sense of ‘however’, the Italian correspondent particle would be ebbene (though), as discussed in Mastrangelo et al. (2016: 87). At least from a semantic point of view, this particle fits in the spectrum of the adversative/denial-of-expectation meaning (see Petrocchi 2022, Sections 2.1.2–2.1.3).

Figure 6 shows a Japanese consultant while realizing sentence (4):

Figure 6:

Japanese speaker uttering sentence (5).

Here we can see the presence of the following non-manual components: ‘widened eyes’, ‘forward head movement’ and ‘raised eyebrows’. This speaker did not use any manual gestures, like several other Japanese consultants.

Figure 7 shows a male speaker uttering sentence (5). As can be seen, this speaker exhibits the same gestural pattern illustrated above: widened eyes, raised eyebrows and ‘forward head movement’. Moreover, he also produced the PUOH, with one hand.^[16]

Figure 7:

Japanese speaker uttering sentence (5).

In general, it was observed that when Japanese participants used manual gestures, they chose the expected one, i.e., PUOH. However, the fact that manual gestures were almost completely absent in their communication is a topic that requires further attention. Hand gestures can be considered inappropriate in this culture and are therefore avoided. We will return to this point in the conclusions in Section 6.

Note that Japanese speakers often ‘trap’ their hands, for instance by interlocking them, as shown in Figure 8:

Figure 8:

Japanese speaker interlocking hands.

When this happens, an additional non-manual gesture often appears during the execution, namely ‘lifted shoulders’. Moreover, in some cases, the speakers move the trapped hands as in an attempt to produce a hand gesture. This observation strongly supports a view according to which the different cultures are much more similar in this particular respect than expected.

In Japanese, the non-manual gesture of ‘forward head movement’ turned out to be aligned with the leftmost surprise particle, mainly. PUOH gesture tends to start at the beginning of the sentence and lasts until the very end of the utterance.

Consider now the ELAN analysis of the sentences. The various tiers encode the analysis of the position of the head, brows, eyes, mouth and shoulder for the non-manual gestures, and hand movement for the manual ones.

Figure 9 is the ELAN analysis of the sentence in the following example:

Figure 9:

ELAN representation of sentence (6).

(6)

Scenario: You know that your friend John is allergic to cats. One day you see him with a big cat in his arms. You are surprised and utter:
Hee	neko-arerugi	nja-nakatta?
By the way	cat-allergy	have-Neg.PAST.Q
‘But weren’t you allergic to cats?’

The following ELAN represents the sentence produced by a male speaker, in reaction to the scenario provided in (3) above (See Figure 10).

Figure 10:

ELAN representation of sentence (5).

In this case as well the non-manual components are widened eyes, ‘forward head movement’ and/or ‘nod’, and ‘raised eyebrows’, plus the hand gesture.

The non-manual gestures tend to spread over all the sentences, in the majority of the cases. The stroke of the head movement is aligned with the verbal form in most of the cases and/or it shows up after tokorode, or in correspondence with hee. The stroke of the raised eyebrows gesture is usually aligned with hee.

Regarding the prosodic component, we saw that in Japanese the intonation of counter-expectational questions differs from the intonation of canonical questions, in that the F0-contour characterizing them is higher than F0-contour in canonical yes/no questions, as shown in the Praat representation in Figure 11.^[17] This is exactly what happens in Italian and in the other languages. The Praat representation in Figure 11 is produced in correspondence to the following scenario.

(7)

Scenario: Your friend Mary calls you on the phone and tells you that she has a wonderful new red dress to wear at tonight’s party. When you meet her at the party, you see that she has a blue gown, you are surprised and say:
Tokorode	aka-no-doresu	janakatta?
By the way	red-color-dress	Cop-Neg.PAST.Q
‘But wasn’t it red?’

Figure 11 illustrates the comparison between the Praat analysis of the surprise question in example (6) – on the left – and the analysis of canonical yes/no questions in Japanese – on the right –discussed in Maekawa (1991). The stimulus proposed by Maekawa (1991, Figure 1) is the following:

Figure 11:

Praat representation of suprise and canonical questions.

(8)

Nánika	miéru
Something	visible
‘Can you see anything?’

As emerges from the comparison between our Praat and Maekawa’s (1991), the prosodic contour of surprise questions in Japanese turns out to be low rather than high, i.e., rising.

We will not discuss here the prosodic aspects any further, since the focus of this article is on gestures, but it is important to stress the similarity of the characteristic prosodic contour across languages and the presence of a significant alignment of prosody with syntax and gesture, which turns out to be a cross-cultural constant.

Finally consider that, occasionally, the Japanese speakers introduce a feeling of disapproval, besides the surprise.^[18] Across languages, disapproval is conveyed essentially by furrowed eyebrows, as we are going to discuss in a while. As it is clear from Figure 12, in these cases, the speaker uses different non-manual gestures uttering the surprise question in (6):

Figure 12:

Japanese speaker uttering example (7).

Again, the same phenomenon was discussed by Giorgi and Dal Farra (2019) for Italian.

To summarize, in 20 % of the cases, our speakers produced the PUOH gesture. In 40 % of the cases, the speakers use other gestures such as pointing. As far as the non-manual gestures are concerned, the percentages are shown in Table 1.

Table 1:

Percentages of gestures.

Let us focus now on a qualitative analysis of the data. As far as head movement is concerned, all the speakers used ‘nod’ except in those cases of enriched interpretation where only ‘forward head movement’ is employed. ‘Raised eyebrows’ and ‘widened eyes’ always occur together and are accompanied almost always by ‘open mouth’. The gesture ‘trapped hands’, i.e., hands blocked by participants themselves, occurs with ‘lifted shoulders’ in 50 % of the cases. Our speakers gesticulate in 20 % of cases and the youngest people seem to gesticulate more than older people; the gestures are the expected ones. Finally, all the speakers used non-manual components. The head movement is the non-manual gesture more present in absolute – 70 % of the speakers use it and do it in 70 % of the utterances, i.e., 28 cases out of a total of 40. The head is moved in a nod in the case of the surprise interpretation and forward in the case of the disapproval-enriched interpretation.

5.1.2 Surprise questions in Korean

Consider now the non-manual components associated with the realization of surprise questions in Korean. Consider the following example:

(9)

Kundey (ne)	koyangi	alleyluki	issci anh-ass-se?
Kundey (you)	cat	allergy	have Neg.PAST.Q
‘But weren’t you allergic to cats?’

Counter-expectational surprise questions in Korean are introduced by the adversative particle kundey, which conveys a particular concessive nuance. As in Japanese, in Korean these sentences show the obligatory presence of negation and of the past marker on the verb.

In Korean, we found the presence of nod or shake on negation, widened eyes and raised eyebrows. Cf. Figure 13.

Figure 13:

Korean speaker uttering sentence (9).

In these cases, the alignment observed concerns the stroke of the head gesture, the most relevant intonational pitch accent and the verbal form. The generalization is that in Korean gestures are aligned with what follows kundey.

Furthermore, occasionally Korean speakers enrich the interpretation of the sentences, as discussed above, conveying disapproval beside surprise by means of furrowed eyebrows. As an exemplification, consider the following example:

(10)

Scenario: Your friend Mary calls you on the phone and tells you that she has a wonderful new red dress to wear at tonight’s party. When you meet her at the party, you see that she has a blue gown, you are surprised and say:
Kundey	(ne oulpampatios)	ppalkansayk	an-i-ess-se?
By the way	(you tonight-party)	dress red color	Neg-Cop.PAST.Q?
‘But was not it red?’

Figure 14 shows the ELAN annotation for the sentence in (10). In this case, is possible to observe the feature ‘furrowed eyebrows’ which is a typical gesture associated with surprise-disapproval questions.

Figure 14:

ELAN analysis of sentence (10).

As far as alignment is concerned, Figure 15 shows the prosodic realization of the sentence in (9).

Figure 15:

Praat representation of sentence (9).

Even in Korean, the prosodic contour of surprise questions is low, whereas the canonical yes/no questions has a high sentence-ending form definable as H%, as discussed in Yun and Lee (2022).

To summarize the results of the experiment, consider Table 2.

Table 2:

Percentages of gestures.

It is possible to see that in surprise questions, 40 % of our Korean consultants employ ‘eyebrows raised’. In 40 % of the cases, they have ‘head nod’. In the 20 % of the cases the speakers show also widened eyes, ‘forward head movement’ and ‘open mouth’ – see Table 2. Concerning head movement, in general the speakers use ‘nod’ except for those cases of enriched interpretation where ‘forward head movement’ is employed. Raised eyebrows and widened eyes always occur together and almost always are accompanied by ‘opened mouth’ as well.

Almost all the speakers used non-manual components. Head movement is the non-manual gesture more present in absolute – 60 % of the speakers use it. The head is moved in a nod in the case of the surprise interpretation and forward in the case of the surprise-disapproval enriched interpretation. This conclusion is similar to the one reached for Japanese, as discussed earlier. In Korean, 80 % of speakers use a ‘head shake’ to indicate negation, while no hand gestures have been found.

5.1.3 Surprise questions in Vietnamese

Consider now the following example in Vietnamese.

(11)

Scenario: Your friend Mary calls you on the phone and tells you that she has a wonderful new red dress to wear at tonight’s party. When you meet her at the party, you see that she has a blue gown, you are surprised and say:
Sao	không	phải	cái váy đỏ	à?
Why	not	classifier	dress red	Q
‘Why not the red one?’

In Vietnamese, surprise questions are almost always introduced by the adversative/concessive particle sao. This element is translated by our consultant as ‘why’ or ‘by the way’. Notice that exactly the same translation is proposed by Japanese and Korean native speakers for tokorode and kundey respectively. In Vietnamese as well the presence of negation is obligatory along with a verb form which can be interpreted as ‘evaluative subjunctive’ – cf. Petrocchi (2022: 145–149).^[19]

Interestingly, in Vietnamese we could observe the same non-manual components associated with surprise questions, as in the other languages we studied, i.e., ‘raised eyebrows’ and head movements. All the speakers used ‘nod’ except in those cases of enriched interpretation where ‘forward head movement’ is employed. In the case of Vietnamese as well, it has been observed an alignment between the relevant pitch, the sentence-final surprise particle and the gestural component – ‘forward head movement’.

Figure 16 shows a Vietnamese female speaker uttering the sentence provided in (11) above.

Figure 16:

Vietnamese speaker uttering sentence (11).

The ELAN analysis of this sentence is given in Figure 17:

Figure 17:

ELAN analysis of sentence (11).

In Vietnamese, communicative functions and sentence types are primarily conveyed by a variety of sentence-final particles. However, intonation does play a role, albeit the degree to which this is conventionalized in the grammatical system is unclear (Duffield et al. 2019). Our first results are coherent with Tran’s (1969) intuition about the existence of emotional questions in Vietnamese. Tran (1969) proposes that in these cases, intonation is modulated to reveal the speaker’s personal attitudes such as surprise, annoyance, exasperation, etc.^[20]

Consider now the following scenario:

(12)

Scenario: You know that your brother reads only spy story. One day you see him reading ‘War and Peace’. You are surprised and utter:
Em đọc ‘Chiến tranh và hòa bình’	á?
You read ‘War and peace’	Q
‘(But) are you reading ‘War and peace’ ?’

The Praat representation of sentence (12) is seen in Figure 18.

Figure 18:

Praat representation of sentence (12).

The Praat representation in Figure 18 shows that the most relevant pitch in the sentence is aligned with the rightmost question particle with surprise value.

Interestingly, in Vietnamese as well, we observed manual gestures of the expected type. Not all the Vietnamese participants gestured, but when they did it, they used the PUOH gesture as can be seen in Figure 19:

Figure 19:

PUOH gesture of a Vietnamese speaker.

Figure 19 shows a Vietnamese speaker uttering the sentence given below in the following scenario:

(13)

Scenario: You know that your friend John is allergic to cats, one day you see him with a big cat in his arms. You are surprised and utter:
Sao	cậu	lại	ôm	mèo?
Why	you	should	hold	cat?
‘Are you holding a cat?’

Summarizing the results obtained for Vietnamese, we could observe the same non-manual components associated with surprise questions as in the other languages. As far as the non-manual gestures are concerned, the percentages are shown in Table 3:

Table 3:

Percentages of gestures.

Concerning head movement, all the speakers used ‘nod’ except in those cases of enriched interpretation where forward head movement is employed. ‘Raised eyebrows’ and ‘widened eyes’ always occur together and almost always are also accompanied by ‘opened mouth’.

Furthermore, we found at least one instance of PUOH. As in in other languages, speakers often ‘trap’ their hands – for example putting them on their hips, as in Figure 20. However, in these cases, an additional non-manual gesture appears, namely ‘lifted shoulders’, as signaled by the yellow arrows.

Figure 20:

Lifted shoulders.

As we already remarked above, Giorgi and Dal Farra (2019) noted the same movement of the shoulders in Italian, when the speakers were forced to trap their hands by holding a heavy bag.

Again, the pattern observed in Vietnamese closely resembles the ones of the other languages examined here and shares several characteristics with European languages.

5.2 Surprise-disapproval questions

In this section, we discuss surprise disapproval sentences. These are open questions that convey negative feelings of disapproval from the speaker towards the addressee due to an unexpected and surprising situation. From our experiment, it emerges that even in this case, there are striking similarities across languages.

5.2.1 Surprise-disapproval in Japanese

Consider the following Japanese example:

(14)

Scenario: You know that your sister should do her homework, but you see that she is reading a romance novel. You are annoyed and utter:
Chotto	nani	shiteruno?!
Hey	what	do.PROG.Q
‘Hey what are you doing?’

All the surprise-disapproval questions uttered by our Japanese participants are introduced by chotto. It is translated as’ hey’ and has a connotation of blame, reproach, and even irritation (Mastrangelo et al. 2016). The sentences produced are always wh-questions.

Figure 21 shows the non-manual gestures associated with the sentence in (14).

Figure 21:

Japanese speaker uttering sentence (14).

Here we can observe ‘furrowed eyebrows’, and ‘forward head movement’. These non-manual components are the same as those already observed in European languages. We can also see a non-manual gesture not observed in Western languages, i.e. ‘squinted eyes’.

Regarding the prosodic component, in Japanese, the prosodic contour of surprise-disapproval questions turns out to be low rather than high, i.e., lowering and not rising. This observation confirms the rhetorical nature of the constructions at issue.

Consider the following example:

(15)

Scenario: You know that your sister should do her homework, but you see that she is reading a romance novel. You are annoyed and utter.
Chotto	nani	yondeiruno?!
Hey	what	read. PROG.Q
‘Hey what are you reading?’

Figure 22 shows the Praat analysis of the utterance in (15):

Figure 22:

Praat representation of sentence (15).

The highest pitch accent is on chotto. As far as the alignment between gestures and the prosodic component is concerned, the non-manual gestures tend to spread over all the sentences, in the majority of the cases, whereas the stroke of the head movement is aligned with chotto. Cf. the ELAN representation in Figure 23.

Figure 23:

ELAN analysis of sentence (15).

To summarize the experiment for Japanese, consider Table 4 above.

Table 4:

Percentages of gestures.

As pointed out above, we can observe the same non-manual components already observed in European languages, namely ‘furrowed brows’ and ‘forward head movement’. In Japanese, we also noted the presence of squinted eyes. These non-manual components are the same as the enriched interpretations observed in the previous section, except for squinted eyes. In this case, we did not find any expected manual gestures.

Analyzing the data, some correlations are observable. Concerning head movement, all the speakers used ‘forward head movement’ as in those cases of enriched interpretation noted in the previous section. ‘Furrowed eyebrows’ and ‘squinted eyes’ always occur together. With ‘trapped hands’, ‘lifted shoulders’ is also observable in 10 % of the cases.

5.2.2 Surprise-disapproval in Korean

Surprise-disapproval questions in Korean are always introduced by ya. Ya is a pseudo-address term that is usually translated in English as ‘hey’ (Kim 2022). It has been defined as ‘vocative interjection’ and is used to summon an addressee at the same age or younger and with a close relationship to the speaker. Wh-constituent is always present as well. Consider the following Korean example:

(16)

Scenario: You know that your sister should do her homework, but you see that she is reading a romance novel. You are annoyed and utter:
Ya (ne)	mwe-hako-iss-se?
Hey (you)	what-do.PROG.Q
‘Hey, what are you doing?!’

In Figure 24, we see a Korean speaker while uttering the particle ya:

Figure 24:

Korean speaker uttering sentence (16).

In this figure we can observe a transition in the position of the eyebrows from slightly furrowed to raised. This might be interpreted as a case of enrichment, as we discussed above. Here, however, contrary to what we saw in the previous section, the enrichment goes in the opposite direction, namely the disapproval sentence is enriched with a surprise component. This is interesting because this phenomenon takes place in European languages as well.

The Praat representation is provided in Figure 25:

Figure 25:

Praat representation of sentence (16).

In Korean the prosodic contour of surprise-disapproval questions turns out to be low rather than high, as in Japanese. The most relevant pitch is found on ya. As far as the alignment between gestures and prosody is concerned, in the majority of the cases the non-manual gestures spread over all the sentences. The stroke of the ‘forward head movement’ and of the ‘furrowed/raised eyebrows’ is observable on ya.

Consider now the ELAN representation for the same sentence in Figure 26:

Figure 26:

ELAN representation of sentence (16).

Again, we find here the same non-manual components already observed in European languages, namely ‘furrowed/raised brows’ and ‘forward head movement’, plus ‘squinted eyes’, as in Japanese.

The following observations hold in Korean: all the speakers used ‘forward head movement’. ‘Head nod’ is never used. ‘Squinted eyes’, when present, always occur with ‘furrowed eyebrows’. No manual gesture was realized by any Korean speaker.

5.2.3 Surprise-disapproval in Vietnamese

Consider now Vietnamese. Surprise-disapproval questions in Vietnamese are always introduced by sao glossed as ‘why’ and di is a particle expressing the ‘evaluative subjunctive’ discussed above.^[21] Consider the following case:

(17)

You know that John must clean his room, but you see him lying on his bed listening to music. You are annoyed and utter:
Sao (cậu)	không	dọn phòng	di
why (you)	not	clean room	particle.SUBJ
‘Why don’t you clean your room?’

The properties of the prosodic component in surprise-disapproval questions in Vietnamese are currently unclear to us. However, there is one certain observation that the emphatic pitch accent is aligned with the verbal form, which in turn is aligned with head movement. Hence, alignment holds in this language as well.

In Vietnamese, we see the same non-manual components already observed in European languages and Japanese, and Korean, namely furrowed brows and forward head movement The non-manual gestures spread over all the sentences, beginning right after sao and lasting until the end of the sentence. In these cases, we did not observe the presence of squinted eyes. Consider Figure 27 above.

Figure 27:

Vietnamese speaker uttering sentence (17).

As can be seen in Figure 27, one Vietnamese speaker produced a manual gesture, i.e., PUOH with iteration. Even if this happened only in a single case, still it might be significant that the gesture produced was exactly an expected one.

In some cases, an enriched interpretation has been found and the sentence is realized with raised eyebrows. Consider the following example:

(18)

You see your brother wearing his best trousers kneeling in the dirt in the garden. You think that he will ruin his trousers. You are annoyed and utter:
Sao (em)	lại mặc	cái quần	kia	ra	vườn?
Why (you)	should wear	classifier trousers those	in	garden?
‘Why are you wearing those pants in the garden?’

Consider now the ELAN analysis of this sentence in Figure 28:

Figure 28:

ELAN representation of a speaker producing sentence (18).

Figure 28 shows that even in Vietnamese some speakers produce an enriched surprise-disapproval sentence, introducing a non-manual gesture typical of surprise questions, i.e., ‘raised eyebrows’. Again, what is interesting here is the observation that this happens in the same way as in the European languages.

6 Conclusions

After having examined surprise and surprise-disapproval questions in Japanese, Korean and Vietnamese, it can be concluded that both types of special questions have syntactic, prosodic, and gestural properties quite similar to those found in Western languages, even if we have observed some interesting – and we can add minor – differences.

Let us consider surprise questions first. From a syntactic point of view, in Eastern languages, these questions exhibit the same features as in the other Western languages already investigated. They present an adversative introductory particle, tokorode in Japanese, sao in Vietnamese and kundey in Korean. Sao and kundey show a concessive nuance (Malchukov 2004).^[22] Negation is also present, along with a verbal form sharing some properties of the Romance imperfect. In addition, we found the presence of an interjection-like element, the surprise particle hee in Japanese. This lexicalizes the evaluative projection, which represents in the syntax the ‘emotional’ meaning of the sentence. Interestingly, no co-occurrence of tokorode and hee have been found so far. Probably, this is due to the fact that the two items belong to different linguistic registers: hee is highly colloquial whereas tokorode is formal. Speakers can omit tokorode, as is the case for the adversative particle ma. However, no omissions of the adversative elements took place in Korean and Vietnamese. These issues seem very promising from a syntactic point of view, also given the very great typological distance between the Eastern languages considered here, and the Western languages previously analyzed. Further research is indeed necessary to provide an exhaustive analysis of the many phenomena found in these constructions. For the purposes of this work, however, it suffices to say that the constructions in question are similar enough to provide a sound basis for a comparison among the languages analyzed.

The prosody of these structures is different from the one associated with the canonical questions. Moreover, the low contour characterizing these sentences testifies to their rhetorical interpretive value. Again, this observation is coherent with what has been observed in the literature for the same structures in Western languages.

Concerning the non-manual gestures observed, we found that in Korean, Vietnamese and Japanese the head can nod, move forward, or be shaken in correspondence with negation. The brows can be either raised and/or furrowed in different moments of the realization of the sentence. They are furrowed when an enriched interpretation is realized, i.e., when disapproval is added to the surprise. We also found a non-manual feature never observed before in surprise questions, i.e., ‘widened eyes’. This feature co-occurs with ‘raised eyebrows’. Furthermore, in the Japanese language, we do find manual gestures of the expected type, namely PUOH, even if much less frequently than in Western languages. We will get back to this point shortly. Also, in this respect, we noted the ‘trapped hands’ effect. Namely, the speakers blocked their hands and tended to raise their shoulders at the same point of the sentence where speakers of Western languages moved their hands – as for instance in the Italian cases observed by Giorgi and Farra (2019).

As far as the alignment among the components is considered, we found that in Korean and Japanese, the most relevant emphatic pitch is aligned with the stroke of the head gesture and co-occurs with the leftmost position available in the syntactic structure, i.e. the nuclear syllable of the verbal form which follows the evaluative – not phonetically realized – head. In Japanese, when hee is present, the stroke of the head gesture is aligned with this interjection-like item. The manual gestures, when realized, start before the beginning of the sentence and tend to overlap the entire sentence. In these cases, given the fact that the adversative particle is usually omitted, we assume that the hand gesture starts in correspondence with the evaluative projection as well. In Vietnamese, the alignment has been observed on the right of the syntactic structure in correspondence with the à surprise particle when this is present. However, we also found cases where the head movement aligns with the verbal form. It seems to happen when the surprise particle is lacking.

Consider now, surprise-disapproval questions. These sentences exhibit peculiar syntactic, prosodical and gestural properties as well. As expected, the prosody of these structures is different from the one associated with the canonical questions. Moreover, the low contour characterizing these sentences expresses their rhetorical interpretive value, in that they are not standard seeking-information questions.

From a syntactic point of view, surprise-disapproval questions are open wh-questions and do not show negation, as in Western languages. In Vietnamese, they are introduced by the same adversative introductory particle as in the surprise cases, i.e., sao. In Japanese and Korean only the interjection-like elements have been observed, chotto and ya, respectively. These items lexicalize the evaluative meaning – the surprise-disapproval, i.e., their emotional content. In these cases, the participants did not use the formal register. Probably, this is because these constructions are more characterized as colloquial, in that they involve ‘disapproval’, which falls in the spectrum of ‘anger’, an emotion usually not allowed in formal contexts.^[23]

Consider now the gestural component. We found that the head can be moved forward or to the side. No instances of ‘head nod’ have been observed. The brows can be either raised and/or furrowed in different moments of the realization of the sentence. They are raised when an enriched interpretation is realized, i.e., when the surprise component is added to the disapproval. We also found a non-manual feature never observed before in the case of Western languages, i.e., ‘squinted eyes’. These features co-occur with ‘furrowed eyebrows’. No hand gestures have been found so far. One hypothesis could be that in such emotionally connoted contexts, the gestures are more likely to be inhibited, i.e., ‘censored’ or ‘tabooed’.

In surprise-disapproval questions in Eastern languages, the prosodical, gestural and syntactic components are aligned. In Vietnamese, the most relevant emphatic pitch is aligned with the stroke of the head gesture and co-occurs with the leftmost position available in the syntactic structure, in that it appears on the nuclear syllable of the verbal form. In Japanese and Vietnamese the emphatic pitch is aligned with the interjection-like elements. Thus, also in these cases, the alignment seems to take place on the right of the adversative particle, or in correspondence with the evaluative syntactic head.

To conclude, our experiments indicate that the non-manual component is very consistent across languages, in that the repertory of the gestures accompanying these sentences is to a large extent the same. With regards to the manual component, the number of manual gestures used was significantly lower – indeed almost non-existent – than in Western languages, particularly the Romance ones. However, the types of gestures produced were generally similar and were of the PUOH category. Moreover, the ‘trapped hands’ effect associated with ‘lifted shoulders’ was found both in Eastern and Western languages.

Going back to the research questions raised in the introduction, our answer is that co-speech gestures do have some general properties that are possibly universal. The first property is alignment since we presented evidence that prosody and gestures respond to the same syntactic trigger in the left periphery of the sentence. This occurs consistently across languages. The second observation is that non-manual gestures seem to be mostly independent of the speaker’s culture. On the other hand, manual gestures are sensitive to the speaker’s culture in terms of quantity, if not quality.

These findings are noteworthy as they strongly support the idea that natural language is multimodal. Gestures, in particular co-speech gestures, can be considered an integral part of language and not merely an addition to a sentence that a speaker might, or might not, choose to include. Moreover, co-speech gestures, in particular the non-manual ones, are part of human heritage, in that they do not vary as much as might be expected across cultures.