Gesture combinations during collaborative decision-making at wall displays

2.1 Gestures on wall displays

According to Gutwin and Greenberg [8], the main sources of awareness information are (i) people’s bodies, (ii) workspace artifacts, and (iii) conversation and gestures. As for the third point, there is a distinction between intentional explicit gestures and consequential communication. While the former are the stereotypical gestures with clear intention of pointing somewhere or something, the latter is information transfer that emerges as a consequence of a person’s activity within an environment [8]. Pointing gestures have been examined by many scholars, such as linguists, semioticians, psychologists, anthropologists, and primatologists. In psycholinguistics, the most prominent gesture taxonomy is that of McNeill [11], who categorized the gestures into gesticulation, emblems, pantomimes, and sign language. Gesticulation is further classified into iconic, metaphoric, rhythmic, cohesive, and deictic or pointing gestures. The prototypical pointing gesture (analogous to the “stereotypical” mentioned before) is a communicative body movement that projects a vector from a body part, with this vector indicating a certain direction, location, or object [12].

Gestures are an indispensable part of embodied cognition, which is cognitive science that implies that thinking and perception are shaped by interactions with the physical environment [13]. Regarding the relation between gestures and embodied cognition, Soni et al. recently identified four types of gesture interactions that promote scientific discussion and collaborative meaning-making through embodied cognition [14]: (T1) gestures for orienting the group; (T2) cooperative gestures for facilitating group meaning-making; (T3) individual intentional gestures for facilitating group meaning-making; and (T4) gestures for articulating conceptual understanding to the group. In our opinion, T3 is analogous to the intentional gestures and T4 to the consequential communication based on the framework of Gutwin and Greenberg [8].

The relationship between gesture and thought is described in the literature review paper of Goldin-Meadow and Beilock [15]. They state that gesture actively brings action into a speaker’s mental representations, and those mental representations then affect behavior – at times more powerfully than the actions on which the gestures are based. Gestures have been examined specifically in relation to problem-solving and decision-making [16–19]. In the study of Alibali et al. [16], they examined whether gestures play a functional role in problem-solving. In their study, participants in two experiments solved problems requiring the prediction of gear movement, either with gesture allowed or with gesture prohibited. They found that participants in the gesture-allowed condition were more likely to use perceptual-motor strategies in the gesture-prohibited condition. Both rotation and ticking gestures tended to accompany perceptual-motor strategies.

As far as gestures specifically in relation to wall-displays are concerned, Liu et al. introduced CoReach [20], a set of collaborative gestures that combine input from multiple users to manipulate content, facilitate data exchange and support communication. In their experiment on a wall display, they asked participants to find similarities and connections between pictures and to arrange them in a meaningful way that they could agree on. In the experiment of Liu et al. [20], it is mainly touch gestures being analyzed compared to our study, which is about mid-air pointing gestures as initiators for gesture sequences. Gesture elicitation studies are also a common and efficient way to create gesture sets [21] and as for eliciting mid-air gestures for wall displays, it has been observed by Wittorf and Jakobsen [22] that the size or extent of gestures is related to the size of the display. In other words, users make larger and more physically-based gestures in wall displays than in smaller displays [22].

According to Hinrichs and Carpendale [23], gestures should not be considered in isolation from previous and subsequent gestures. They have explored gesture sequences in multi-touch interfaces in-the-wild, but to our knowledge, an empirical study with gesture sequences under realistic collaborative conditions at large wall displays, as described in this paper, has not been explored until now.

Noteworthy is the distinction of our definition of gesture sequences from the so-called cooperative gestures. According to Morris et al. [10], cooperative gestures can be used to enhance users’ sense of teamwork, increase awareness of important system events, facilitate reachability and access control on large, shared displays. However in Morris et al.’s approach [10], the gestures of the users were considered as a single, combined command. The case study, which is presented in this paper, builds upon the work of Maquil et al. [24]. In particular, we look at a pointing gesture produced by a single user, consider this as a single command, and then explore the reaction(s) of the other users towards this initial one-user command. Therefore, we differentiate our work from cooperative gestures as defined in Morris et al. [10]. We follow the categorization of Maquil et al. [24] about pointing gestures (see Table 1) and analyze accordingly the gestures produced by the participants in our user study. All three categories in Table 1 are mid-air gestures and not touch gestures.

The first type is narrative pointing (NP), where according to Maquil et al. [24], a user points sharply with the index finger, but also moves the finger towards a larger area of the display (up/down or left/right). The second type of pointing gesture is loose pointing (LP). Here, the user is not looking at the screen and holds the hand usually open or the palm up. LP gestures often happen when a user describes a concept as a whole. A third type is that of sharp pointing (SP). This is the “stereotypical” pointing gesture with an index finger where a user points to a very specific area of the display (e.g., a specific word or number). Its duration is usually much shorter than NP.

In this paper we extend the taxonomy of pointing gestures to explore which gestures come as subsequent gestures to NP, LP, and SP. The research question that we aim to answer through our studies is “Which are the most frequent subsequent gestures after narrative, loose, and sharp pointing respectively?”

The potential gestural reactions following NP, LP, and SP are:

1. Lack of action;

2. Other pointing gestures of any type: NP, LP, SP;

3. Emblems or adaptors;

4. Touch gestures.

These reactions can be produced by the same user or anybody else and we considered this aspect as well in our gesture analysis (see Section 4.1, Figure 6b). In this paper, we first focus on deictic gestures and actions on the system. Hereafter, we also briefly describe emblems and adaptors as they may also come up as reactions and should be considered at a later stage.

Emblems are those nonverbal acts (a) which have a direct verbal translation usually consisting of a word or two, or a phrase, (b) for which this precise meaning is known by most or all members of a group, class, subculture, or culture, (c) which are most often deliberately used with the conscious intent to send a particular message to the other person(s), (d) for which the person(s) who sees the emblem usually not only knows the emblem’s message but also knows that it was deliberately sent to him, and (e) for which the sender usually takes responsibility for having made that communication [25].

Adaptors are movements first learned as part of an effort to satisfy self needs or body needs, or to perform certain bodily actions, or to manage and cope with emotions, or to develop or maintain prototypic interpersonal contacts, or to learn instrumental activities [25].

As touch gestures, we regard all gestures that include touching a display independently of where or what this touch was targeted at. Our hypothesis regarding our aforementioned research question is that there are more touch gestures initiated by sharp pointing (Hypothesis 1), since this is the more stereotypical intentional type of gesture ([12]) compared to narrative and loose pointing.

2.2 Gaze and visual attention

In the next paragraphs, we review the literature on gaze at tangible and digital artifacts, since it is another important natural awareness cue about visual attention that can provide important insights about collaborative data visualization on wall displays. Gaze can be aligned or misaligned with hand gestures. Our focus is on pointing gestures; most often, gaze and pointing gestures are indeed aligned, since humans usually look at objects in one local area at a time. However, humans shift their gaze to scan the surrounding visual environment, because they cannot process all the information simultaneously. Recently Lystbæk et al. [26] examined what happens when a target is presented in the peripheral visual field and made various experiments by manipulating the participants’ head direction and fixation position: the head was directed to the fixation location, the target position, or the opposite side of the fixation. The performance was highest when the head was directed to the target position even when there was misalignment of the head and eye, suggesting that visual perception can be influenced by both head direction and fixation position [26].

Moreover, some studies proved that more fixations on a particular area indicate that it is more noticeable, or more important to the viewer than other areas. This is in line with “The More You Look The More You Get” paradigm [27] where users focusing their gaze on a specific work of art or part of it can be interested to receive some additional content about that specific item. Others [28] have highlighted wall-sized displays as a viable solution to present artworks that are difficult or impossible to move, and presented a Natural User Interface to explore 360° digital artworks shown on wall-sized displays. That solution allowed visitors to look around and explore virtual worlds using only their gaze, stepping away from the boundaries and limitations of keyboard and mouse.

Recently Cheng et al. [29] summarized empirical patterns of interdisciplinary work in organizational behavior, primatology, and social, developmental, and cognitive psychology to analyze visual attention as a window to leadership. Among others, they highlighted that shared gaze may facilitate teammate coordination and performance and eye gaze provides a reliable, behavioral, and under-utilized source of information about the hierarchical structure and functioning of a team [29]. The connection between eye gaze data and cognitive styles has been examined by Raptis et al. [30]. They revealed that individual differences in cognitive styles are quantitatively reflected on eye gaze data (gaze entropies, fixation duration and count) while users perform visual activities of varying type (e.g., visual search, visual decision-making) and varying characteristics. The authors suggested as the next step of this research to conduct more feasibility studies, considering other cognitive styles, activity characteristics, and application domains.

Moreover, Sharma et al. [31] investigated the causal relationship between individual and collaborative cognitive processes with gaze measures as a proxy to provide more insight into the collaborative learning process. They found that collaborative gaze patterns drive the individual focus when participants are engaged in problem-solving dialogues using an intelligent tutoring system and that the nature of the causal relationship changes depending upon the context of the learning.

Furthermore, as far as gaze on displays is concerned, GazeProjector is a system that combines (1) natural feature tracking on displays to determine the mobile eye tracker’s position relative to a display with (2) accurate point-of-gaze estimation [32]. A related work to our research is that of Lystbæk et al. [26], who suggested gaze-hand alignment as a principle for both modalities for pointing in Augmented Reality, and alignment of their input as selection trigger. Their idea is based on Zhai et al. [33] whose key idea is to leverage that the eyes naturally look ahead to a pointing target, followed by the hands. The results of our study, though, show many misalignments which come to contradict the statement of Zhai et al. [33], when it comes to large wall displays (see Section 4.2).

To sum up, gaze is often researched in relation both to the sensing (i.e. how it is measured) and the cue design aspects, but the literature gap is that gaze research often occurs in isolation and not with regards to its alignment with gestures. The research question that arises from the literature review and we seek to answer through our user studies is the following: “When are pointing gestures misaligned with gaze?” It is noteworthy that in the setting of large wall displays and collaborative decision-making between multiple users, there are certain challenges, such as the large size wall displays, the high amount of data being visualized, and crossing users in visual and peripheral field. Our hypothesis is that the gaze-pointing gestures misalignments happen when a group is split into subgroups (Hypothesis 2) and users have to share their attention between frontal viewing on displays and pointing gestures, because they cannot process all the information simultaneously ([26]).

3.1 Participants

24 users, divided into six groups of four, participated in our study. Twenty of them were male, three female, and one participant preferred not to answer the question. Participants’ ages ranged from 20 to 38 (M = 24.2, SD = 4.5, Mdn = 22). All participants were students in either computer science or geography. They were recruited through the University of Luxembourg and a Technical University in Luxembourg. The study participants acted as a team of decision-makers from a hospital that needs to ensure that the stock of protective equipment meets the hospital’s needs for the next three months. The participants needed to interpret and analyse the data, and identify the best solution given the existing constraints.

The study took place in consultation with our ethics committee. We explained to the participants what type of data we would record and how we would process it. To respect privacy, participants would immediately be assigned pseudonyms. We also informed participants about their rights to withdraw their consent and ask for the deletion of the data at any time and without giving reasons. We provided them with an information sheet and a consent form.

3.2 Apparatus

The experiment took place in our 360° Immersive Arena, 2 m high, composed of 12 screens (4 K resolution each) that are spatially positioned in a circle of 3.64 m diameter (see Figure 2a). Eight screens were used (therefore covering 240°) to display data visualizations. Three fixed cameras (top, front, and back cameras) were used to record the user study and the participants were using clip-on microphones for audio recording. Participants did not wear any motion tracking system or eye gaze trackers.

Figure 2:

The mixed-presence collaboration setup and scenario used for the present study. (a) The 360° Immersive Arena, the multi-display setup we used for the study, consisting of 12 screens of 4 K resolution each, displaying the scenario with data visualizations. (b) A closer look at some of our data visualisations. The corresponding tasks are indicated on the Figure. Note that the black bars represent “bezels” i.e. gaps between adjacent screens in the display setup.

3.3 Decision-making scenario

The decision-making scenario had four distinct tasks: (i): estimate future COVID numbers, (ii) select the protective equipment to restock, (iii) select one offer in the overview, and (iv) select delivery option. Each task included at least one type of (interactable) data visualisation, some of which are shown in Figure 2b. The current task was displayed on the first (leftmost) screen (see Figure 2a). When a group considered that a task was complete, they clicked on a confirmation button to continue to the next task.

3.4 Analysis

We performed our analyses on video recordings from the experiments, adding up to a total of 2 h 23 m 52 s. We analyzed and manually annotated a total of 578 pointing gestures, 201 touch gestures and 86 gaze misalignments made by participants using the ELAN software [34], as shown in Figure 3. Such gestures were then organized in sequences of annotations using the same software. We relied on Python scripts to process these annotations, compute the descriptive and general statistics (sums, percentages, averages) mentioned in this document, and used the Matplotlib [35] library to generate the corresponding plots. Both the annotations^[2] and the notebook that contains the code used for computing statistics and generating visualisations^[3] are available online.

Figure 3:

Our study analysis and annotation setup using the ELAN software.

4.1 Gesture sequences

Generally speaking, most of our annotated gestures are part of gesture sequences. If we include touch gestures, we had a total of 779 gestures, 64 % were part of gesture sequences and 36 % isolated gestures. The gesture sequences are generally short with no more than 9 gestures (see Figure 4).

Figure 4:

Length distributions of follow-up sequences of tracked gestures per initiator type. (a) Starting with a NP initiator gesture. (b) Starting with a SP initiator gesture. (c) Starting with a LP initiator gesture.

In Figure 5a we see the distribution of all annotated gesture types. LP ranked first with 29 % followed by SP (27 %). Touch gestures come at the third place (26 %) and NP is the least common type of gesture at 18 %.

Figure 5:

Distributions of annotated gesture types overall and amongst initiators. (a) Distribution of annotated gesture types. (b) Share of initiator gesture types for sequences of annotations.

Figure 5b shows the share of the gesture types (NP, LP, SP) that served as initiators for the annotated gesture sequences. The results show that SP is the most frequent initiator gesture type (45 % of all initiators were SP gestures, and 72/209 i.e. 34 % of all SP gestures were initiators of actual sequences, containing at least one follow-up gesture). LP and NP were indeed less frequent initiators, respectively accounting for 31 % and 24 % of the initiators, with 49/227 (22 %) of all LP gestures being initiators, and 39/142 (27 %) of NP gestures. This seems to indicate that SP gestures tend to produce more gestural reactions than other types of gestures, which may be explained by the fact that they typically concern a precise choice to be made, which may itself lead to discussions and counter-proposals by other team members.

Figure 4 depicts the length of sequences, which shows that sequences typically do not consist of many subsequent gestures. On average, NP-initiated sequences have 1.97 follow-up gestures, SP-initiated sequences 2.14, and LP-initiated sequences 2.16. Overall, the longest sequences are composed of 9 gestures, and 88 % (NP), 87 % (SP), and 92 % (LP) of the sequences are maximum 4 gestures long. As seen in Figure 5a, while the majority of gestures were part of a sequence, some were isolated. Isolated SP, LP, and NP gestures add up to 172 instances out of the total 578 pointing gestures (i.e. 30 %).

In Figure 6a we see the share of the follow-up gestures for sequences respectively having NP, LP, or SP as initiator. We can observe that there is no particular tendency to keep the type of subsequent gestures the same as the initiator. However, it seems like more precise initiators (SP and NP) tend to produce more LP (that are less precise) follow-ups. This is especially the case for sequences starting with a SP initiator, the most precise type of pointing gesture, that produces 50 % LP follow-ups. Conversely, LP-initiated sequences are more often followed by the more precise gesture types (compared to SP- and NP-initiated sequences).

Figure 6:

Follow-up gesture types and authors, depending on sequence initiator type. (a) Share of follow-up gesture types per initiator type. (b) Share of follow-up gesture authors per initiator type.

In terms of users producing subsequent gestures, we notice in Figure 6b that 24 % of the follow-ups after a NP initiator are made by the same user, which roughly corresponds to the “expected” 25 % (that would correspond to a random distribution of these follow-ups among the four participants). LP- and SP-initiated sequences however reach higher values (respectively 34 %, and 30 %) and therefore show a slight tendency to have more of the subsequent follow-ups made by the same author as the initiator, although a large portion of the follow-ups are still made by other users.

Another element we wanted to quantify relates to touch events. In fact, Figure 7 shows how many of the sequences led to a touch event (the touch does not need to be the last element of the sequence, but at least one touch event must be included for that sequence to be counted as leading to a touch). While sequences starting with a NP or SP gesture respectively led to a touch 46 % and 47 % of the time, LP-initiated sequences only led to a touch event 31 % of the time. We can deduce that more-precise gestures (SP and NP), which are likely associated with concepts being described in more details, tend to lead to subsequent actions on the system more often. LP regularly occurs as an unintentional communication artifact whereas NP and especially SP are typically intentional pointing gestures. This result is in line with our first hypothesis (see Hypothesis 1 in Section 2.1).

Figure 7:

Share of sequences that led to at least one touch event.

4.2 Gaze-pointing misalignment

In our study the participants were not wearing an eye-tracking device, which means that the precise Point of Gaze could not be obtained. However, for the purpose of our research, we can deduce certain results about gaze and pointing gestures by analyzing and manually coding the raw video data. As aforementioned, we manually annotated the head-pointing gestures misalignments. The results showed that out of the total of 578 pointing gestures (including SP, LP, and NP) that were performed during the study, in 86 cases (15 %) was the gaze misaligned with the pointing gesture.

We also observed that most misalignments were produced in relation to the screens with large amounts of data visualizations, such as options to touch/select from, e.g. kinds of protective equipment or delivery options. In these cases, the users sharply pointed (SP) and then touched the display while simultaneously looking at another display (the one closest on the right) to see the impact of their action on data visualizations.

In other fewer cases, users misaligned their gaze with their pointing gestures when they referred orally to the tasks related to the collaborative decision-making scenario (as in Figure 8a). In these cases, the user pointing to the display with the screens reminded the other participants of the current task. The most frequent pointing type when this happened was loose pointing, because the user did not refer to a specific element, but to the whole task-reminding screen in order to support their verbal comment.

Figure 8:

Examples of gaze-pointing misalignments. (a) Misalignment – far from screen. (b) Misalignment – close to screen.

In the rest of cases, the user’s hand was held stretched towards the screen and while the user described a concept, they shifted their gaze towards a participant (example shown on Figure 8b). In these cases, the pointing was narrative. It should be noted that the misalignment here is not what usually happens in the retraction phase of a gesture, where a user shortly keeps the arm stretched, but we talk about a longer in duration, intentional gesture-gaze misalignment.

As a last point, the misalignments happened both close and far from the screen, so the position of the user therefore does not seem to be of much impact to misalignments.