Training of patient handover in virtual reality

Sebastian Streuber; Patrick Saalfeld; Katja Podulski; Florentine Hüttl; Tobias Huber; Holger Buggenhagen; Christian Boedecker; Bernhard Preim; Christian Hansen

doi:10.1515/cdbme-2020-0040

Article Open Access

Training of patient handover in virtual reality

Sebastian Streuber , Patrick Saalfeld , Katja Podulski , Florentine Hüttl , Tobias Huber , Holger Buggenhagen , Christian Boedecker , Bernhard Preim and Christian Hansen

Published/Copyright: September 17, 2020

Published by

Become an author with De Gruyter Brill

Author Information Explore this Subject

From the journal Current Directions in Biomedical Engineering Volume 6 Issue 1

Abstract

Patient handover is an important part for information transfer between medical professionals in a clinical setting. Yet, in current medical education, these conversations are only trained sparsely, since they are costly to perform as they take place in offsite courses and are led by experts over several days. Virtual reality (VR)-based training courses could increase the availability of training, by eliminating travel costs and reducing the time-commitment of the teaching experts. This work presents a VR prototype of a multi-user training and examination application for patient handover. To ensure a similar interaction quality to its current real world counterpart, this work used omni-directional video recordings to create a realistic setting and compared different projection methods. A pilot study highlighted distinct use-cases of sphere and mesh projections to visualize the recordings. The results suggest enhanced spatial presence relating to the usage of omni-directional videos in VR-applications.

Keywords: human–computer-interaction; medical training; patient handover; virtual reality

Introduction

In daily clinical routine, it is necessary to effectively communicate information about patients between a variety of staff members. This includes nurses informing the resident physician of the patient’s status or, alternatively, an interdisciplinary patient handover. Handover practices are of vital importance to ensure patient safety [1]. Standardized handover methods were created for this purpose, which aim to reliably convey relevant information, while lowering the time needed for a handover [2], [3]. These methods are not yet standardized across disciplines or institutions. Commonly used methods are SBAR [3] and its expansion ISBAR (Identity, Situation, Background, Assessment & Action, Response & Rationale) [4]. SBAR is a communication model, aiming to direct the conversation to be concrete and without ambiguity. The conversation should focus on the current Situation, the Background of the patient, the concrete Assessment by the physician and lastly, the recommended Response (SBAR). To ensure that the relevant communication models are understood, handover scenarios became part of medical curricula [5]. One arising problem though, is the cost for an effective handover training and later examination. For example, examination is employed using the OSCE method (Objective structured clinical examination) [6] in a three day course using 24 examiners per 48 students. Due to the costly nature of this course, training opportunities for the students are sparse. Following the approach of Huber et al. [7], conducting the examination in virtual reality (VR) is a promising alternative. This reduces costs and time investments for the examination experts as well a providing students multiple training opportunities. Yet, for a virtual examination to be viable, all gradable elements of a handover should be displayable and any adverse effects relating to VR [8] should be minimized. This paper proposes the use of a handover VR multi-user application for examination and training. A prototype was developed using omni-directional video footage that consists out of two handover scenes (handover by phone and in person). It aims to highlight opportunities arising from VR multi-user applications as well as comparing different suitable visualization-methods.

Methods

To generate a realistic immersive environment, the Pimax 5K Plus in conjunction with the HTC Vive controllers and trackers were used. The Pimax display offers a large field of view with 200° and a resolution of 2,560 × 1,440 pixels per eye. The HTC Vive trackers offer a 4.5 × 4.5 m tracking area allowing for spatial immersion in VR applications. The prototype was developed with the Unity game engine. To record the omni-directional footage, the Z-Cam V1 was used, providing 10 lenses of 3,376 × 2,768 resolution at 30 fps. After stitching, the resulting 360° video will have a 6,720 × 3,360 resolution in 30 fps. The spatial mapping, a 3D reconstruction of an object or room, was made using the Microsoft Hololens since it already natively provides the mapping feature.

Handover scenario

In close collaboration with medical experts, a handover scenario was developed. The scenario comprises of a male patient (63 years old), who is scheduled for a knee replacement surgery the next day, being found unresponsive by a student doctor. After a physical examination, a rectal bleeding was discovered. The task of the student is to do a short telephone handover to the resident physician, and afterwards a long-form handover in person.

The SBAR method should be used for the handover. This was implemented in two VR scenes. The first being a doctors’ office, where all relevant information is displayed on a virtual computer-screen to the student, while the resident is in a different office waiting for a phone call. Once a call is initiated, a voice transmission between both users is enabled by an application-observer.

After the call is completed, both resident and student will teleport into the second scene, a virtual patient room, where both are visible in form of their avatars. Here, the task is to perform a long-form handover using the SBAR method. Both performances will be graded by the observer. As visual representation for the scenes in the office and patient room, two approaches were tested. The first being a projection of the omni-directional video footage onto the inside of a sphere, which is the current standard for 360° video display. This sphere will move with the VR user in a way that the user always remains in its center. As a result, this method precludes any apparent spatial movement of the user, adversely affecting spatial immersion and possible resulting in nausea. To mitigate this, a mesh projection was tested. Here, the omni-direction video was projected onto a mesh recorded via a spatial mapping with the Hololens. This would in principle allow users to move in the scene providing a better spatial immersion.

Humanoid avatar

A high degree of realistic social interaction is necessary in order to enable the possibility for examination. Humanoid avatars could help to display mimic-gesticulatory performances of the students to each other and the examiner. For this reason, the users were represented by complete humanoid avatars with a one-to-one movement-mapping. Avatar-personalization, matching an avatar to the user, is often done in the literature by scanning the user with multiple cameras and creating a 3D model from the recordings [9], [10]. Even though high-quality results can be acquired, these techniques either take many time-intensive captures or contain artifacts arising from obstructed faces while scanning. As mitigation and to keep the setup affordable to use in an educational setting, an avatar-individualization was implemented using only the sensor-data of the deployed VR system. Following Bachkanov et al. [11] the Autodesk Character Generator was used to create general avatars. This tool allows the creation of humanoid models, exportable via FBX to Unity. The imported model consists of a list of separate meshes for the body, eyes and teeth, in addition to a hierarchical bone structure, with which it is possible to transmute the pose of the avatar on run-time to match the users pose. To display the user-movement, these bone translations and rotations need to be approximated. For this purpose, the Final IK library was used, utilizing inverse kinematics to calculate the necessary translations and rotation in order to imitate the users movement from the tracking data of the VR system.

Avatar calibration

To increase the pose approximation accuracy with inverse kinematics, the proportions of the avatar should match the size of the user. Inverse kinematics aims to match the goal position set by the VR system (the location of the head and hands) to the corresponding avatar body parts without stretching the avatar or angling the joints to an unnatural degree. If the proportions of the user and avatar do not match, this might not be possible, forcing the algorithm to make undesired compromises. This may include the arms of the user avatar clipping through its body or the hand positions not matching the expectation of the user. To customize the avatar, a calibration scene was implemented through which the arm and leg length as well as the body size of the user was approximated. This was accomplished by asking the user to take four dedicated poses shown in Figure 1. The difference in height of the controllers between (a) and (b) divided by two is set to represent the arm length. The forearm length can be derived from the pose shown in (c) and finally, the pose (d) allows to estimate the leg length. The mean head height over all calibration positions is used to set the body size of the avatar. This calibration method represents an useful compromise between a long and taxing calibration for the user and an ill-fitting avatar, being unable to replicate user-movement.

Figure 1:

Calibration positions used in prototype to approximate arm-, leg- and body-size.

Voice transmission

Since voice communication is the central element in a patient handover, it is crucial to provide the ability to hear one’s counterpart in the application. The free audio transmission tool Photon Voice (Exit Games GmbH, Hamburg, Germany) was used for its low latency, high-quality audio and easy integration into a Unity application. An important part of the handover skills is the ability and courage to ask the handover-partner to repeat themselves, if some information was not clear or heard. To purposefully create such a situation, it is possible for an external observer to obstruct the voice transmission of the user by inserting interfering noises. These noises consist of a ringing telephone, a loud crowd, the siren of an ambulance and the beeping of a vital signs monitor. The volume hereby is purposefully adjusted to obstruct the communication and force inquiries about inaudible information.

Network management

In addition to the voice transmission, Photon also offers a network management solution integrated to Unity. Hereby Photon has a different approach to Unity’s implemented multiplayer functionality (HLAPI). While in the HLAPI implementation, a user is required to act as a server for the application, Photon makes a clear separation between server and user. If a player desires to connect to a multiplayer game, they must either connect to a server provided by Photon itself, or to a server provided by a third party. To create and run one’s own server, Photon provides a Photon Server SDK. This a useful alternative in case of privacy concerns since the audio will be contained to on premise servers, or if the application is deployed without a stable Internet connection, but where a stable local network is available.

Results and discussion

The prototype was presented to a group of four medical professionals in an informal pilot study. The handover tool was presented as a multi-person VR application with two users in VR and one observer on a display, evaluating and grading the performance of the handover, while also being able to introduce noise obstructions. The group of experts were asked to perform a handover in both mesh- and sphere-projection scenes, while the investigator took the role of observer introducing noises and leading the users through the scenes. The presented findings were obtained through observation of the participants and from a following discussion. Though in principle being useful for examination purposes, the visual mesh projection method was criticized. Unfortunately, the recorded geometry contained several artifacts and gaps, which was perceived as irritating (see Figure 2).

Figure 2:

Screenshot of the prototype with mesh and sphere projection, presenting the artifacts of the mesh projection, when displaying the computer monitor and bottle.

The tested mesh projection visualization improved maneuverability, though came at the cost of visual artifacts arising from imperfections in the recorded mesh. This trade-off was not well received by the participants. The handover itself was possible in both representations without complications and was well received. The avatar calibration lead to a more realistic inverse kinematic approximation, matching the user- and avatar-movement more closely. This improved the spatial presence and could lead to better performances in the handover task.

In future work, several improvements are possible. The mesh-recording can be improved, for example by utilizing a dual line-scan camera [12] for fast and accurate measurements. The avatar customization can be improved by employing full body tracking and thereby, exactly approximating the users size, or by creating a custom model per user. In addition, a formal study should be conducted with medical students, evaluating the advantages of a VR training application in contrast to conventional training.

Corresponding author: Sebastian Streuber, Faculty of Computer Science & Research Campus STIMULATE, Otto-von-Guericke University Magdeburg, Magdeburg, Germany, E-mail: sebastian.streuber@st.ovgu.de

Funding source: Federal Ministry of Education and Research of Germany

Award Identifier / Grant number: 16SV8054

Research funding: This work was funded by the Federal Ministry of Education and Research of Germany under grant number 16SV8054.
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Competing interests: Authors state no conflict of interest.

References

1. Devlin, MK, Natalie, KK, Kiss, A, Richardson, L, Wong, BM. Morning handover of on-call issues: opportunities for improvement. JAMA Internal Med 2014;174:1479–85. https://doi.org/10.1001/jamainternmed.2014.3033.Search in Google Scholar

2. Collins, G. Using simulation to develop handover skills. Nursing Times 2014;110:12–4.Search in Google Scholar

3. Farhan, M, Brown, R, Vincent, C, Woloshynowych, M. The ABC of handover: impact on shift handover in the emergency department. Emerg Med J 2012;29:947–53. https://doi.org/10.1136/emermed-2011-200201.Search in Google Scholar

4. Thompson, JE, Collett, LW, Langbart, MJ, Natalie, JP, Stephanie, MB, Yuminaga, Y, Ossolinski, G, Susanto, C, McCormack, A. Using the ISBAR handover tool in junior medical officer handover: a study in an Australian tertiary hospital. Postgraduate Med J 2011;87:340–4. https://doi.org/10.1136/pgmj.2010.105569.Search in Google Scholar

5. Jana, J. Ärztliche kommunikation: praxisbuch zum masterplan medizinstudium 2020. Klett-Cotta 2018;497.Search in Google Scholar

6. Nikendei, C, Jana, J. OSCE-praktische Tipps zur Implementierung einer klinisch-praktischen Prüfung. GMS Z Med Ausbild 2006;23:2006–23.Search in Google Scholar

7. Huber, T, Tom, W, Paschold, M, Lang, H, Werner, K, Hansen, C. Highly immersive virtual reality laparoscopy simulation: development and future aspects. Int J Comput Assisted Radiol Surg, 2018;13:281–90. https://doi.org/10.1007/s11548-017-1686-2.Search in Google Scholar

8. Anthony, S, Pan, Y, Zisch, F, Steptoe, W. The impact of a self-avatar on cognitive load in immersive virtual reality. In: 2016 IEEE virtual reality (VR). IEEE; 2016, pp. 67–76.Search in Google Scholar

9. Chen, Y, Dang, G, Cheng, Z-Q, Xu, K. Fast capture of personalized avatar using two Kinects. J Manuf Syst 2014;33:233–40. https://doi.org/10.1016/j.jmsy.2013.11.005.Search in Google Scholar

10. Lin, S, Chen, Y, Lai, Y-K, Martin, RR, Cheng, Z-Q. Fast capture of textured full-body avatar with RGB-D cameras. Vis Comput 2016;32:68–691. https://doi.org/10.1007/s00371-016-1245-9.Search in Google Scholar

11. Bashkanov, O, Saalfeld, P, Gunasekaran, H, Mathews, J, Preim, B, Huber, T, Hüttl, F, Werner, K, Hansen, C. VR multi-user conference room for surgery planning. In: Proceedings of the annual meeting of the German Society of Computer- and Robot-Assisted Surgery (CURAC); 2019, vol 6, pp. 264–8.Search in Google Scholar

12. Sun, B, Zhu, J, Yang, L, Yang, S, Guo, Y. Sensor for in-motion continuous 3D shape measurement based on dual line-scan cameras. Sensors 2016;16:1949. https://doi.org/10.3390/s16111949.Search in Google Scholar

Published Online: 2020-09-17

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/cdbme-2020-0040

Keywords for this article

human–computer-interaction; medical training; patient handover; virtual reality

Creative Commons

BY 4.0