On the Attempt to Implement Social Addressability within a Robotic System

1 Introduction

The need for embedded and embodied robot entities [4] is often emphasised in the literature with regard to the question of how to construct acceptable and sociable robotic systems. Underlying this need are questions pertaining to considerations in building a robot entity with whom a user would like to engage socially, and which qualities of the robot’s appearance or shape are necessary with regard to the task the robot is to perform. Historically, the design of social robots is oriented heuristically towards the human form and therefore the assumption is that social robots – or as Neil Frude puts it: intimate machines – need a human face in order to enhance the human-robot relationship and their interactions is an old one (see Halbach [10]: 77). Meanwhile, the notion of face was displaced by more technical concepts such as interface or point of interaction and more recent functional approaches refer to the hypothesis that suggests the materialised form of a robot should correspond to its function in order to improve interaction (“matching hypothesis”).^[1]

We assume that the abstract functional properties for social interactions do not necessarily need a human-like form and we see a lack of theoretical description of these functional preconditions for an intimate or socially interactive robot in the HRI literature. Because the human-human interaction serves as a reference model for social interactions in general, especially for most of the constructors and engineers in the field of HRI, we see an unexploited potential in the use of social and sociological theory and methodology to think about and beyond HRI. Therefore, we would like to give an overview of existing concepts and approaches that address this issue in Chapter 2, and subsequently we would like to emphasise the theoretical notion of social address.

As the interaction with a robot bears its own needs, we differentiate the term “social address” from terms like “point of interaction” or “interface.” Therefore, we would like to point out the characteristics of social address and addressability (3.2), but also the in situ construction of interaction sequences (3.1), where in an ethnomethodological approach, it is necessary to provide information to the interaction partner constantly. On the basis of these assumptions, in combination with findings from a study conducted in 2015, we propose five hypotheses (3.3) on how the robot interaction could become smoother. As a result of the interdisciplinary discourse on these hypotheses, our research team implemented new LEDs on the non-humanoid robot BROMMI:TAK, which was to serve as a basal form of such an address. During the Lange Nacht der Wissenschaften 2016, a science event that took place at the Technical University of Berlin, we had the opportunity to test the modified robot in a handover-interaction with a large number of visitors to the event (4) and we thus tried to find answers to our hypotheses (5). Hence this paper addresses two objectives regarding the abstract thinking about interaction and – building on these thoughts – the specific experimental interaction between a human and a robot (HRI). In Chapter 5 we discuss the findings regarding our hypotheses. As the hypotheses are inspired by abstract social theory, not all of them can be answered sufficiently, but the data shows the general relevance of such an address for HRI. We can show that it bears the potential to make the interaction with a robot smoother if the robot is able to indicate its general status and, most notably, the further actions that will take place.

2 Point of Interaction and the Social Gaze in HRI

HRI-Research focuses on the interdisciplinary research question of how to design a robot that can deal with the complexity of an interaction with a human – one can describe this as the robot-centred view [3]. Additionally, it is important to consider the perspective of the human interaction partner, as his or her perception of the interaction sequence is directly linked to his or her (re-)actions and therefore also linked to the question as to the sequence as a whole is successful or not. The literature here refers to the term human-centred HRI [3] and is mostly oriented towards the well-known human-human-interaction in order to achieve greater acceptance on the part of the human user.

Our research is concerned with what is often unspecifically called a “point of interaction” (e.g. [18]: 55; [16]), which can be a particular interface or particular action and therefore shows a proximity in content to HCI research. There are also articles emphasizing the importance of the gaze of a robot’s “eyes” for an HRI [19], or articles which reflect on the anthropomorphic design of robots heads [5]. These articles did not reference the phenomenon to existing theoretical approaches but rather discuss single aspects of an address, in this example the use of gazing or anthropomorphic designs. Although we are aware of extensive research in this area from a psychological perspective, we have consciously chosen a sociological approach. The main focus of the paper presented here lies in suggesting an innovative sociological perspective which is grounded in social theory, instead of comparing psychological and sociological insights. Such a comparison must take place systematically at a later point in time. In our conception, the point of interaction is crucial, as every interaction between a robotic system and a human user is centred around and oriented towards it.

We will underpin this theoretically with the notion of an address,^[2] which is a more general concept to describe possible points of interaction between a human and a robot entity. A social address in a human-centred view on HRI entails the human user receiving information on what the robotic system is doing and will do next, which enables the human user to adjust his or her actions in response. We thereby hope to be able to generalise from the specific interaction-forms of a robot entity.

Both former and current research in this field related to the PoI are primarily concerned with the use of gaze and its importance in typical HRI settings. Most of this conducted research came to the conclusion that social gaze has a favourable impact. Fischer et al. [7] for instance summarised their research as follows:

“Our qualitative and quantitative analyses show that people in the social gaze condition are significantly more quick to engage the robot, smile significantly more often, and can better account for where the robot is looking. In addition, we find people in the social gaze condition to feel more responsible for the task performance. We conclude that social gaze in assembly scenarios fulfils floor management functions and provides an indicator for the robot’s affordance, yet that it does not influence likability, mutual interest and suspected competence of the robot.” (204)

The social gaze of the robot was tested in a typical tutor setting, i.e. the human had to teach the robot how to perform a cooperative task. Moon et al. [14] as well as Zheng et al. [20], [19] came to very similar conclusions in a similar HRI test setting regarding handover situations. They were able to “provide empirical evidence that using humanlike gaze cues during human-robot handovers [...] the timing and perceived quality of the handover event” (Moon [14]: 2) can be improved. In almost all the recent gaze research that has been conducted in HRI, the models were similar human-human interactions. The tested setting can be seen as a blueprint of the analogous human-human interaction situation. In addition to work cooperation and handover situations, the function of gaze as feedback pertaining to acknowledgement of content-based exchange was also tested [15]. The overall results were similarly positive: “We argue that a robot – when using adequate online feedback strategies – has at its disposal an important resource with which it could pro-actively shape the tutor’s presentation and help generate the input from which it would benefit most.” (Pitch et al. [15]: 268) Recent research on the topic of gaze in HRI came so far to the shared conclusion that the consideration and the proper implementation of social gaze is of paramount importance for a successful outcome in terms of a satisfying, effective, and efficient interaction between humans and robots.

In a former study conducted during the “Lange Nacht der Wissenschaften 2015”^[3] we observed the coordinative mechanisms during a handover-interaction between a robot and a human interaction partner according to several dimensions (i.e. spatio-temporal, material, social) with a robot that was not able to react to the human’s exact position but had fixed positions for the handover instead. In other words, the robot worked in reference to scripts and was not able to act reciprocally – this fact was well perceived by the human interaction partners. We discovered that the human adjusts to the movements of the robot in order to lead the interaction to a success. As the situation took place in front of an audience, it is important to note that the participants had an even higher motivation to succeed in the experimental interaction than in a laboratory setting. Therefore, the data showed a strong need to implement accountable movements, communicative acts, or symbols to provide the human with more information about the robot’s further actions.

3 Theoretical Concepts

Although it is unusual in common HRI research, we do work with sociological theories and concepts. We propose this theoretically informed perspective with two main purposes in mind: First, we use ethnomethodological assumptions to interpret video data, and second, we would like to emphasise the concept of (social) addressability to achieve a proper general description of an HRI.^[4]

4 Method / Implementation

4.1 Study Design

The study was conducted during the “Lange Nacht der Wissenschaften” (Long Night of the Sciences) – an annual science event in which Berlin’s universities and research institutions open their doors for visitors and present a multifaceted programme. In this context, the presentation “BROMMI:TAK – the biomimetic robot trunk” took place for the third consecutive year – for the first time, however, in the FabLab of our research group. BROMMI:TAK is a pneumatically operated, non-humanoid and biomimetically designed robotic system which conducted a basic interaction (handing over an apple) with the visitors in a Wizard-of-Oz setup.

We conducted further preliminary studies in our FabLab to determine possible positions for the information-providing interface and its implementation as a basal form of address. Although the implementation of the LEDs might appear quite specific, there are in fact not that many possibilities due to the limiting but efficient construction of the robotic arm. As the robotic arm has neither many rigid parts nor the possibility to carry much weight, we decided to use small and lightweight LEDs. With respect to familiar forms of status indication on machines, we chose two bigger LED lights at the trunk as general status indicators and smaller LEDs on the gripper as an account for the progress of the interaction sequence (see Fig. 1). While the status LEDs should provide information about the general status and serve as an account for animacy, the gripper LED should provide more specific information on whether the robot is going to open or close the gripper. Therefore, the LEDs on the trunk were active throughout the whole experiment, while the LEDs on the gripper were only active if the robot was holding an object or using (opening/closing) the gripper.

For this study, the form of the implemented address was chosen in respect to a working scenario where one may not be able to use audio but only visual accounts. Also, we would like to implement the address in a form which is most advantageous (on the manipulator and on the trunk) for the interacting user considering the context of use.

Figure 1

BROMMI:TAK with the basal light address.

The study was conducted with two experimental conditions in a mixed-method-design, using a videographic approach (see 4.3.3) combined with quantitative and qualitative questionnaire data. As we had to deal with a come-and-go audience, we decided to test the robot system half the time with the LED-address and half the time without. The advantage of using such an event for conducting a study on HRI lies in reaching a broader range of participants in a less artificial context as in regular laboratory experiments, generating an HRI situation that is more related to an everyday context. Despite the convenient conditions for empirical inquiry in the FabLab however, disruptive factors are not as controllable as in strict laboratory experiments and one has to take into account the social factor of a (more or less) present audience during the HRI.

5 Findings

Using videography, it is possible to gain information about the relevance of the two LED-light addresses, which were implemented in the trunk and the gripper. As the gripper holds the apple, it is hard to say whether a gaze is directed towards the apple or the LED-light on the gripper itself – however it is easy to differentiate those from gazes on the LED-light in the trunk.

The analysis of the video data suggests a general relevance of the implemented LED-Light on the trunk, as it was perceived by the interaction partners in 50 out of 93 (54%) interactions. Differentiated by the point in the interaction at which the gazing occurs (several gazes were possible), we could observe a lower interest in the LED-light before the interaction started (19 views, 20%), a high interest while the interaction is going on (32 views, 34%), and a medium interest when the apple was successfully handed over (26 views, 28%).

The categorisation of the video data shows a relation between gazes on the address and the amount of formerly observed interactions by the human interaction partner. This makes sense intuitively as observing others interacting with the robot is an important source for gathering knowledge about the robot and the interaction. The gathered knowledge could include the action that is expected of the participant (his or her own role), the sequential order of the interaction, the dimensioning of the relevant space, the purpose of the interaction and – possibly most importantly – the competence that can be expected from the robot interaction partner. The data shows that most of the participants who interacted with the robot tried to get further information before they enter the interaction space of BROMMI:TAK – they tend to act more cautiously in front of an audience. In contrast participants who didn’t observe any former HRI had to gather all information during the interaction itself by searching for accounts more actively and interpreting them – these situations are of special interest for us, as they contain data about the potential relevance of a social address in a more contingent HRI.

Table 1

Interaction Sequence 1 – unsuccessful handover (apple falls). Time designations in brackets (starting with 0 sec. in Frame 1).

1. (t=14sec.): P. focuses on gripper.	2. (t=19sec.): P. observes the second blinking.
3. (t=21sec.): The gripper blinks for the fourth time.	4. (t=22sec.): The gripper opens and the apple falls to the ground.

In total, our data contains 15 such participants who interacted with BROMMI:TAK without previous knowledge about the situation and in 14 (93%) of those situations we could observe a consultation of the LED-light on the trunk. In contrast, we count 72 situations in which the participants had more information and we observed such a consultation in only 34 (47%) of those.

In the following paragraphs, we want to provide two exemplary sequential interpretations of handovers in our video material. The first case shows an unsuccessful interaction sequence, where the apple falls to the ground after a misunderstanding about the time of the handover; the second one is an example of a smooth and intuitive handover with the initiation of a giving-back sequence, which takes mistakable characteristics.

In sequence one (Table 1), the interaction partner makes visual contact, recognises the status change in the light address in the trunk and focuses on the gripper (image 1). After the gripper reaches its final position, it blinks two times, until the interaction partner moves his hand towards the apple (image 2). Meanwhile a third blink occurs. But the anticipated handover takes too long – after the fourth blinking, the interaction partner pulls back his hand (image 3), shortly afterward the gripper opens and the apple falls to the ground (image 4). The behaviour of the interaction partner can be interpreted as correctly anticipating the first blinking as an affordance to take the apple whereupon the subsequent blinkings irritated this interpretation and led the participant to draw back his hand. Additionally, there was a long delay between BROMMI:TAK reaching its final position and the opening of the gripper, which is not in accordance with an intuitive and smooth spatio-temporal order of a handover.

In the second sequence (Table 2), one can observe the man on the left taking the apple being offered by the robot. He is entirely anticipating the purpose of the situation and his own role in it, as he is reaching the specific place for the handover at the exact moment when the gripper has started blinking (image 1). It is important to note that this handover of the apple was very smooth regarding the time that the participant had to wait for the gripper to open. The gripper stays open (in relation to the smooth handover: maybe too long) and the trunk moves forth and back a bit because of the loss of weight of the apple. The little movements, combined with the open gripper were interpreted by P. as an affordance to offer the apple back to the robot in the next sequence. But just before the gripper closes he suddenly pulls back his hand.^[7] The gripper opens again and the man places the apple in the gripper (image 2). After the gripper closes, there are no following movements until the LEDs on the gripper blink again. The man moves his hand slowly to the gripper, but BROMMI:TAK starts moving to the right when the man reaches the handover-position (image 3). The man acts embarrassed by pulling back his hand and leaves the situation by turning away (image 4). We interpret the behaviour of the participant as anticipating the blinking of the gripper correctly, which can be seen in his reaction to the second blinking after BROMMI:TAK has received the apple again: The man expects the gripper to open another time because it starts to blink. But the LED address was never meant to communicatively handle the return of the apple and so it was falsely blinking instead of just shining. Nevertheless, it can also be seen that the forms of communication are lacking in complexity as the participant is possibly irritated by the gripper staying open and by the small movements. We are guessing that the man reacts to the temporal structure of the situation (when the gripper stays open) in a sensible way because of the fluent initial handover he experienced (images 2–4). We therefore conclude that the complexity of the communicative potential was not enough to handle the situation and the control of the LEDs was not precise enough to give the accounts in an appropriate window of time. The example not only shows the functional effects of a basal light address, but also the need for its accurate implementation and coherent usage. Otherwise the path dependency of a situation would dominate the interpretation of any new user of a robotic system and could lead to crises in the interaction.

Table 2

Interaction Sequence 2 – smooth handover with mistakable giving-back sequence. Time designations in brackets (starting with 0 sec. in Frame 1).

1. (t=8sec.): P. reaches out to take the apple while the gripper is still closed.	2. (t=17sec.): Second try to give the apple back.
3. (t=28.5sec.): Second blinking, P. moves his hand towards the handover destination.	4. (t=30sec.): B. moves to the right, P. pulls back his hand completely.

With respect to the open-ended questions asking about the colour concept, we conclude that the colour associations were correct in most cases. The colour green was mostly associated with “Everything OK”, “Affordance”, or “Readiness”. Blue was associated with an active meaning, e.g. an “Action”, “Readiness”, “Expectation”, or “Detection.” Red was mostly interpreted as “Error” or “Warning” and yellow was in most of the cases linked to a “Movement” category.

Figure 2

Semantic differential of the adapted and reduced Godspeed Questionnaire (GQS).^[8]

Our sample consisted of n=135 respondents between the age of 6 and 73, with 37% being female. As younger respondents tended to choose extreme categories more often and might have had difficulties understanding some of the adjective pairs, we dropped them from the sample, resulting in n=100 cases (age range 18–73).

To evaluate the general effect of the LED light address, we first considered the overall rating on the semantic differential scale for the two experimental conditions (with and without light). A factor analysis of our adaption of the Godspeed scales led to a reduction of the original item pool and generated the factors

1. “anthropomorphism” (corresponding items: unnatural-natural, mechanical-human/humanlike, artificial-natural/real; Cronbach’s Alpha 0.7),

2. “animacy” (corresponding items: inanimate-animate, inert-vivid, not interactive-interactive, indifferent/apathetic-responsive, inflexible-adaptable; Cronbach’s Alpha 0,6),

3. “intelligence” (corresponding items: incompetent-competent, irresponsible-responsible, tactless/inappropriate-tactful/appropriate, unintelligent-intelligent; Cronbach’s Alpha 0,8),

4. “sympathy” (corresponding items: unsympathetic-sympathetic, impolite-polite, unpleasant-pleasant, unfriendly-friendly; Cronbach’s Alpha 0,8) and

5. “trustworthiness” (corresponding items: deceitful-trustworthy, threatening-harmless, unreliable-reliable; Cronbach’s Alpha 0,7).

In the aggregated subscales of the Godspeed, there are no significant differences, although the differences on the sympathy scale approach statistical significance (p=0.055) (see Fig. 3). Participants in the condition without light rated BROMMI:TAK as more sympathetic than in the light condition.

Figure 3

Comparison of means of the adapted and reduced Godspeed scales.^[9]

6 Discussion

Regarding hypothesis H1 (relevance of a social address for the quality of an HRI), our data does not suffice to provide an adequate answer. Nonetheless, our data suggests that H1.1 (relevance of the LED light as status address for the quality of the HRI) can be confirmed, especially for first encounters. This data allows us to draw conclusions about H1 indirectly.

The categorisation of the video data with regard to gazes directed to the LEDs in the trunk revealed interesting findings depending on the amount of previously observed interactions. In first encounters, participants looked at the LED address in almost all the cases, whereas only about half of the participants looked when they already had observed prior interactions. Therefore we draw the conclusion (see also H1) that the address is more important in HRI when the situation is more contingent. Or in other words, if the participant does not have knowledge, he/she is more eager to request further information from a potential address on the robot. As the LED-Lights in the trunk did not contain much information, we could not expect an increase in communication or reciprocal exchange of information, but it nevertheless shows that there is a potential need on side of the participants for getting further information from the robot before and during the interaction. As we were testing a specific form of a non-anthropomorphic robot arm it is important to say that the findings presented here should only be generalized to the specific class of robotic arms. It should be clear, that there are several other possible forms of implementing a social address on a robot, which may have different effects on the user experience – we choose our concrete design considering the initial scenario for BROMMI:TAK, namely a working scenario where simple visual accounts would work better than – for example – auditive forms. Regarding the enormous variety of robots, one has to note that almost all HRI findings have to be interpreted respectively to the concrete form. If a socially acting robot is supposed to be able to interact with every human, whether he/she is knowledgeable or not, our data shows the need for such a social address. But it should be clear that further studies will be necessary until the findings respectively the concept of social addressability could be generalized onto other robotic systems.

H1.2 (more positive evaluation on the Godspeed scales in the light condition) has to be rejected as the statistical analysis shows no significant effects. There are several possible reasons as to why our data didn’t show significant differences between the two experimental conditions:

Firstly, the measurement instrument might not have been sensitive enough to trace the effects concerning the quality of the interaction that would be ascribed to the implemented light address (experimental manipulation). This argument is in line with the lack of adequate quantitative measures to evaluate an HRI. Effects of the light address might have had greater impacts on other subdimensions than those depicted by our modified Godspeed scales (anthropomorphism, animacy, intelligence, sympathy, trustworthiness). Beyond this, the Godspeed questionnaire might not be as appropriate for non-humanoid robots as for humanoids or androids.^[10]

And secondly, if we assume that the results of the Godspeed scales are reliable and valid, there are several possible reasons for why the implementation of the address could have been disadvantageous. The information displayed by the LED lights might have been too unspecific and possibly did not bear the intended communicative relevance. This can be supported qualitatively as we observed many interaction sequences in which the blinking of the LEDs took much too long before the apple was released by the gripper, which queries the information value of the LEDs. In the implementation of such an address, there is no room for ambiguities if the effect is supposed to manifest itself in an overall quantitative rating. The qualitative video data shows evidence that the implemented address elements have to be coherently synchronised in order to serve as unambiguous accounts for the interaction, otherwise the interaction becomes irritated quickly. Another possible explanation, which is in line with the approach of significance in the sympathy scale ratings between the two conditions is that the light colour of the LEDs is generally too cold. This could explain why participants in the without light condition rated BROMMI:TAK as slightly more sympathetic than participants in the light condition.

We can generally confirm hypothesis H2 (H2.1 and H2.2 – smoother handover-interaction, if the robot indicates where and when the handover will take place by following a direct pathway or indicating the progress in the interaction process) with reference to exemplary cases in our video material. We observed very smooth HRIs when BROMMI:TAK followed a direct pathway to address the potential interaction partner. Moreover, there are several instances in our video material in which the participants react to the blinking of the LEDs on the gripper and read them as an account for the following action of releasing the object. This only functions well if the object is released shortly after the blinking starts, otherwise it starts to get irritating or is interpreted falsely. We therefore conclude that the implementation of any address on a robot entity has to take place and be programmed to correspond strictly to the actions undertaken by the entity in order to be able to provide sufficient information about the robot or the interaction sequence. A non-coherently implemented address does more harm than good.

7 Conclusion

The basic idea was to implement social addressability via an LED interface on a robotic system. This implementation of a social address had only limited success, as the LEDs only had restricted informational and no reciprocal communicative capabilities. Nevertheless, they served as accountable instances in some interaction sequences, which promoted smooth and intuitive HRIs. The robot in the WoZ-setup was able to maintain an illusion of interactivity, reciprocity, and even sociality in some interaction sequences. Notwithstanding, our data shows that the interpretation of a materialised address as social bears significant challenges. A refined measurement tool or experimental setup is needed to account for the gradual differences that differentiate what we introduced as social address from mere mechanical behaviours or reactions.

Another challenge our study faced was the use of a non-humanoid robot to test the relevance of an addressable interface. We chose the concept of a social address to also include more abstract forms than faces or clearly anthropomorphically designed surrogates. Our implementation possibly missed the display of some sort of perceptive faculty (e.g. via a camera lens or other sensors), which might have maintained the illusion of an addressable interface more efficiently. Another plausible assumption is that – with regard to the smoothness of the interaction with non-humanoid robots – other parameters might play a more important role than social addressability (e.g. direct pathway, adequate speed of the interaction in general). Social addressability, however, could absorb other deficiencies in performance of the robotic system, if the address is able to display possible reasons for them.

Despite all the limitations of our study, we were able to achieve an untypically high number of cases compared to other HRI studies. Our sample contains a wider range of ages, levels of education, and social contexts (e.g. families, singles, couples) than studies that usually use university students. Because of the public character of the event, we were able to analyse a situation containing a variety of interaction forms, which is much closer to everyday occurrences in which robots will be introduced in the near future. The social component of other persons present is of particular importance, especially concerning the learning effects.

Our theoretical perspective proved to be very fruitful in gathering interesting insights in the large amount of (primarily audio-visual) data. We were still able to preserve an analytic openness and impartiality with our qualitatively oriented approach. Videography as a method is particularly suitable for this type of experiment as it takes the in situ construction of the situation seriously. It poses the question: How do the persons involved create the HRI situation sequentially? The assumption is that what is observable is anything but coincidental. Beyond that, we also see the necessity for standardised quantitative measurement tools for HRI in order to measure and compare the quality of an HRI. With respect to our findings, we see the need for further research under stricter laboratory conditions in order to validate the outcomes, especially regarding the relevance of a more elaborated social address. This relevance was basically already approved qualitatively, especially for first encounter interactions in which the human interaction partner does not have specific knowledge about robotic systems.

Referencing sociological theory in order to reformulate the phenomenon and acquiring an abstract understanding of the situation helped us to formulate hypotheses and thereby investigate the categorised video data. The notion of a social address was introduced as an embodied form so it can be recognised as a distinct form (or: body/entity), which can be ascribed the ability to recognise a social environment and being able to give accounts of what will happen next in the interaction. By giving accounts, an entity can be ascribed the ability to differentiate between a self and an environment and therefore has to be able to process some sort of self-reference. Being able to represent these abilities is the crucial quality of a social address we would like to emphasise. In this regard, BROMMI:TAK might be seen as a robot that behaves with self-referentiality, but with a limited potential for complex reciprocal communication. An ethnomethodologically informed perspective indicates the strong need for accountable actions or communication in an interaction with an unfamiliar artificial interaction partner. We conclude that if the interaction partner has learned to differentiate between the robots accountable actions (i.e. LEDs, gazes, or speech) from the simply necessary ones (i.e. technically induced movements, noise), the use of additional behavioural forms of communication (or the use of an address) makes the interaction significantly smoother and thus faster. As a precondition for this, the temporal implementation and the use of accountable behaviour has to be strictly coherent in order to avoid irritations.