Design and Evaluation of a Natural User Interface for Piloting an Unmanned Aerial Vehicle

1 Introduction

In recent years, unmanned aerial vehicles (UAV) have become increasingly prominent in a wide range of applications. They are used for inspections, delivery or search and rescue operations. Benefits from UAV usage are the possibilities for monitoring activities which are extensive and dangerous for human intervention. At present, the UAV flight control is mainly based on joysticks, buttons and switches. Such devices are not intuitively easy to manipulate, and long training times are needed for precision flying [6], [14]. A conventional remote control (R/C), which is a typical UAV control system, involves the use of several joysticks simultaneously and additional switches for setting the flight parameters and performing other tasks like payload task interactions as well as displays for monitoring the flight and payload parameters (see Fig. 1).

Figure 1

A conventional remote control (R/C) for a professional unmanned aerial vehicle (UAV).

Controlling a UAV is challenging and stressful. The operator has constantly to observe the UAV and its surrounding to avoid collisions. A study has proved that among others the demanding workload is one major issue causing air accidents [36]. To reduce the demanding workload scientific activities focus two aspects: providing assistance for autonomous flight and developing user-friendly interfaces, whose functions are easy to operate. This paper outlines such a user interface.

According to Norman [23] natural user interfaces (NUI) are very popular. Those interfaces are using nonconventional input modalities like gesture, speech or gaze direction. Despite Norman’s [23] and Nielsen’s [21] warnings about the usability of such interfaces, scientists promise natural and intuitive interaction techniques with „the potential to make human-robot interaction more natural“ [26, p. 34]. In addition, the development of low-cost devices such as smart wearables, smart watches or smart glasses, keep pushing researches to envision new interaction possibilities which will replace conventional input devices [15, p. 572]. Following Norman [23] and Nielsen et al. [21] it is important to take account of the application. According to this conclusion, the question arises: Is it possible to replace the conventional input device for UAV with a NUI? To answer this question, we present a NUI composed of an optical see-through head-mounted display (HMD) and a motion control device and compaired this with the conventional input device.

In the following sections, we first present current research results about NUIs (Section 2). Afterwards we discuss current design recommendations (section 3). In section 4, we describe the concept and implementation of a NUI. Then, a laboratory experiment is described (section 5) and the results will be outlined (section 6).

2 Previous Work

The main propose of NUIs is to overcome the constraints of conventional input devices. As Norman [22, p. 210] has noted, the problem with interfaces is, that they are interfaces and that the users must focus attention on them. The underlying cause is the effort to transmit an intention to concrete actions regarding the interface or rather the input device. Conventional input devices, like a R/C, require an enormous effort.

NUI refers to an interface that can easily and efficiently transmit an idea from user’s mind into action with little additional effort [16, p. 10]. NUIs are using natural human communications capabilities. Therefore, they should lend themselves to easily learned, intuitive interaction techniques. Among NUIs, gesture-based interfaces have always played a fundamental role in human-machine interaction. Gesture input is a natural communication, because humans are using gestures for communications in their everyday life.

Therefore, researchers have shown increased interest in developing gesture interfaces for controlling UAVs. These developments include definitions of intuitive gestures [26], [2], [15], methods for gesture recognition [7], [18], [32] or the design of feedback [20]. The underlying theory is that a natural and intuitive interface leads to an effective and reliable interface.

These assumptions have been criticized as unproven and too generalized by Nielsen et al. [21] and Norman [23]. First, gestures are not natural per se and second „a gesture interface is not universally the best interface for any application“ [21, p. 409]. The first point focuses the lack of well-known standards for the design of gestural interfaces, which suggests that something like a universal intuitive interaction does not exist. For example, many studies have shown that the cultural background has a vast influence of the intuitive use of gestures [34]. The second point addresses the lack of feedback and accuracy. Also, evaluations of gesture interface for UGVs (unmanned ground vehicles) and UAVs have shown these problems [2], [15]. In previous work we analyzed a gesture set for the teleoperation of a UAV [13]. Based on a laboratory experiment with 14 participants we demonstrated that the gesture-based teleoperation was neither better nor worse than the conventional R/C. But among other things, the participants criticized the missing feedback.

About the second point, the conjunction of a HMD and an augmented reality (AR) interface can compensate the lack of feedback by displaying information in the user’s field of view. So far little attention has been paid to the relationship of NUIs, which are using HMDs and gestural input, and the usability related to the teleoperation of UAVs. Hence, Norman’s [23] question, whether a NUI is the best interface, can, in case of teleoperation an UAV, not be answered. To close this gap in the current research, a NUI composed of an AR interface and a gesture and speech interface was developed and evaluated.

3 Design Recommendations

Peshkova et al. [26] and Cauchard et al. [2] motivate to use multimodal, intuitive input alphabets like gesture and speech commands. Following Cramar et al. [3], Norman [23] and Raskin [31] it is probable, that a universal intuitive gesture or speech command alphabet for the teleoperation of UAVs does not exist. However, gesture or speech alphabets, which are corresponding to mental models of target users, are felt intuitive [26, p. 40].

In addition, there are other requirements focusing ergonomic [21], [19], technical [35], [33] or usability (like feedback or execution times) aspects. For example, Nielsen et al. [21, p. 414] disclose six principles in ergonomics, including the avoidance of non-ergonomic positions. According to McMillian [19] gestures should be concise and quick to minimize fatigue. To ensure learning and performance advantages gestures should be designed in such a way, that natural co-ordination of the body can be employed to co-ordinate multiple degrees of freedom to the external device [19].

Certainly, these diverse requirements are not necessarily compatible with each other. For example, the requirement on the user not to wear markers, gloves or not to fix the background conflicts with the requirement of accuracy (detection, tracking and recognition) [35]. In fact, balancing these conflicting factors according to ensure a successful teleoperation of a UAV may be the key to a usable NUI [37, p. 162].

There can be no doubt that feedback is one important aspect to ensure a usable interface [5], [23], [21]. To avoid limitations related to a constant line-of-sight between operator and UAV, visual information should be displayed in the field of view but not overlap the UAV. One possibility to fulfil these requirements is the usage of a HMD with an AR interface. Concerning displaying information (absolute or relative position, colors) ISO 9241-13 suggests, that status information (like battery status or signal strength) should be displayed in a consistent way, always placed at the same position. Furthermore, information should be displayed noticeable for users, but users should not be disturbed in any case and be distracted as less as possible.

As Livingston [17, p. 8] has noted, there are only a few studies on the human factors associated with AR systems. Often studies focused on the use of HMD in aerospace applications. In a research report from the NASA researches identified “attention” or “cognitive capture” as one major issue causing air accidents [28]. This term describes the issue that it is difficult for pilots to switch between information displayed in a HMD and information in the “outside world”. Furthermore, efficient text visualization in AR displays is critical because it is sensitive to display technology, ambient illumination and various backgrounds [4].

Figure 2

Set of gesture commands. Red lines represent thresholds.

Based on previous work [4], [8] we experimented the effects on the readability during a UAV-teleoperation of three different text styles (plain text and bill boarded text) [12]. To evaluate the different text styles 30 participants were asked to pilot a UAV through a course, while they wore a HMD. At randomized intervals, textual information were displayed in the HMD which the participants had to react. Our findings can be summarized as follows: Textual information should use font colors black or red on a white billboard color.

However, it cannot be assumed that the user is always focusing fade-in textual information in the HMD. It is more likely to be assumed that the focus is on the UAV and its surroundings. To achieve an effective and efficient feedback ISO 9241-13, Norman [23], Zhai et al. [37] propose the usage of multimodal feedback.

4 Concept and Implementation

NUIs and AR visualization in HMDs date back over a long time [24], but so far implementations have been bulky and not very user-friendly. The technological development for HMDs and wearables has already achieved considerable progress, so it is possible to design and implement a NUI in compliance with the requirements formulated in section 3. Regarding design recommendations, we designed and implemented a NUI.

4.3 Feedback

As pointed out in section 3 we designed three different areas for the visual feedback. The central area of the field of view is kept free to allow a direct line of sight to the UAV without occlusions. Below this area we designed two areas, one to display alerts, like low battery, and another one to display a constant visualization of the UAV attitude. The visualization of status information, like battery or received signal strength, is displayed above the central area. Warning messages and the constant information about UAV attitude are visualized in font colors red or black on a white billboard color, whereas status information are displayed in font color white without billboard (see Fig. 3). Further we implemented audible feedback as soon as the UAV is changing its attitude. For this purpose, we used the google voice response to create short verbal prompts. In addition, we have opted for tactile feedback, provided when the thresholds (red lines in Fig. 2) are exceeded.

Figure 3

Concept of the AR interface (left) and its realization (right).

4.4 Hardware Components

Figure 4

Architecture of the system: Elements of the ROS infrastructure (yellow), main control (green), mixed reality application for the HoloLens (blue), hardware components (grey). Dotted arrows are representative of transmission protocol.

For the recognition of the forearm pose we used the Myo wristlet by Thalmic Labs. It is a low-cost 8-channel, consumer-grade surface electromyography (sEMG) device. The Myo wristlet is non-invasive, as it enables users to simply put it on without any preparation. This ensures that the requirement “come as you are” has almost been fulfilled [35, p. 63]. In addition, the wristlet includes a 9-axis inertial measurement unit (IMU). The IMU can be used to determine the pitch, roll and yaw of the user’s forearm. Moreover, it provides tactile feedback by vibration.

Several companies are researching and developing hardware in form of glasses to make AR-applications possible. One resulting device is the Microsoft HoloLens. It consists of depth camera, speakers, holographic lenses and an IMU, composed of accelerometer, gyroscope and magnetometer.

We used an unmodified multi-rotor Parrot Bebop UAV. The device weighs approximately 400 g with a maximum flight time of 11 min and can be controlled via a mobile device, or via an imitated R/C named Skycontroller.

4.5 Architecture of the System

We used Robot Operation System (ROS), a hybrid P2P distributed architecture, to pass information [30] and assigned the ROS Master role to a laptop (see Fig. 4). The UAV and the Myo are communicating with the ROS Master through a wireless network. We implemented three nodes, which are preforming computation. The first node is used to get the current pose of the Myo and to instruct it to vibrate (Myo Node in Fig. 4). The second node is used to control the UAV (UAV Node in Fig. 4). The third node is collecting and processing information from the Myo and UAV Node (Control Node in Fig. 4) and passing the information to the main control, which is not part of the ROS architecture as well as the Microsoft HoloLens. Instead, we implemented the main control (Main Control in Fig. 4), which is used to control the whole process, as a stand-alone program and a mixed reality application for the HoloLens, which communicates with the other elements of the architecture via User Datagram Protocol (UDP). ROS Master, UAV Node, Control Node, Myo Node and Main Control are running on the same laptop. The Main Control contains the control algorithms to determinate the state variables, which indicates the desired status of the flight attitude, the visualizations in the HoloLens and the vibrate behavior of the Myo. According to the state variables the Main Control is passing appropriate orders to the Control Node and the mixed reality application for the HoloLens (HoloLens App in Fig. 4). The HoloLens application was developed with Unity Personal and C# in combination with the MixedRealityToolkit-Unity (for speech recognition) and runs on the HoloLens, whereas the UAV Node, Control Node and Main Control were developed in C / C++ and the Myo Node in Python. For the development of the UAV Node we used the Software Development Kit (SDK) provided by Parrot and for the Myo Node the Myo SDK.

Figure 5

Visualization of the four courses.

5 Method

In order to evaluate the usability of the NUI, a laboratory experiment with 30 participants was conducted. Within this experiment the Parrot Bebop UAV had to be piloted with two modalities of control (Skycontroller and NUI). To avoid learning and “carryover” effects we decided to split the participants into two groups. 15 participants (6 female) with a mean age of 30 ± 3.25 years were using the NUI. 15 participants (7 female) with a mean age of 31 ± 4.73 years were using the Skycontroller. Within the NUI-group 8 participants had used a gesture control before (in the context of games) and within the Skycontroller-group 4 participants had used an R/C to teleoperate a UAV.

In the beginning of the experiment, participants were informed about procedure of the experiment and filled out a demographic questionnaire. To evaluate the interfaces the participants were asked to pilot the Parrot Bebop UAV through four different courses (see Fig. 5 and Fig. 6).

Figure 6

Participant equipped with HoloLens and Myo piloting the UAV through course 3.

Within these courses different flight maneuvers had to be executed. The aims were to take off, bypass obstacles and to land within a mark (landing area). Course 1 required to bypass an obstacle by rising. Course 2 required to fulfil left and right roll maneuvers. Course 3 and 4 included a right or rather left rotation.

Course 1 and course 2 had to be completed three times, whereas course 3 and 4 two times. The courses were considered as accomplished successfully, if the investigator did not have to intervene to avoid UAV collisions, the UAV did not crash and the UAV was landed within the marks.

As dependent variables, effectivity was measured by successful executions, efficiency measured by the required time for passing (from the take off to landed UAV), and suitability for learning determined by the questionnaire ISONORM 9241/10 [29]. The latter questionnaire operationalizes the ergonomic principle “suitability for learning”. For this principle, five bi-polar statements are to be rated on a 7-point scale ranging from “- - -” (1) to “+++” (7). Following this metric, we created a further questionnaire, which focuses the satisfaction related to the modality of control. The questionnaire required to rate five bi-polar statements with the same rating scale as the ISONORM 9241/10. After completing all four courses a structured interview with questions about the interface was conducted. In the end, the participants had the opportunity to mention non-addressed aspects and general comments and suggestions.

6 Results

6.1 Effectivity

Table 2

Effectivity comparing both modalities of control (Skycontroller vs. NUI).

Successful completions with	Course 1	Course 2	Course 3	Course 4
Skycontroller	41 of 45	40 of 45	25 of 30	26 of 30
NUI	43 of 45	43 of 45	27 of 30	27 of 30

Table 3

Used speech and gesture commands (NUI, only successful completed tasks regarded).

Command	N	Course 1	Course 2	Course 3	Course 4
Gesture	2671	695	643	655	678
Speech	206	33	59	56	58

140 of 150 flights were successfully completed using the NUI, whereas 132 of 150 flights were successfully executed using the Skycontroller. A chi-square test of independence was performed to examine the relation between type of control (NUI or Skycontroller) and successful completed tasks. The relation between these variables was not significant, χ2(1)=2.52, p=.11. Tab. 2 shows the results in detail.

Figure 7

Time to complete the courses.

Regarding the NUI, 2671 gesture and 206 speech commands were used (see Tab. 3). Most speech commands were used for “take off”, “land”, “rotate left” and “rotate right”.

7 Discussion

In our study, which we partly presented on a conference [10], a NUI for the teleoperation of a UAV has been developed and evaluated. This NUI has been compared to a conventional input device. Regarding the effectivity it is not possible to prefer one modality of control. Our findings support the view that a conventional input device is more efficient than the NUI. This result must be interpreted with caution. Several participants noted that there were too many possibilities (speech and gestures commands) to pilot the UAV. Thus, participants considered at length whether which possibility to choose. Therefore these results must be treated as tentative as more research is conducted to identify the relation between multimodal, intuitive input alphabets and efficiency. Partially we could observe significant learning effects with the NUI. This should be considered in further experiments.

The satisfaction was greater for the NUI than for the Skycontroller and participants confirmed that the NUI is intuitive. Our findings strongly support the fact that an intuitive interface is not the most efficient interface. The answer to the question whether a NUI can replace the conventional remote control depends on the relevance of efficiency. Future research should concentrate on improving efficiency.

This overall impression could also be confirmed at a demonstration at a conference where layperson people were able to pilot a UAV (see Fig. 8) [11].

Figure 8

Demonstration of the NUI at a conference. Laypersons were able to pilot the UAV.