Volume 21, Issue 1 - Special issue "Culture and Computer Science"

Writing is an essential cultural technique, and the resulting artefacts are an important part of cultural heritage. The Cluster of Excellence ‘Understanding Written Artefacts’ is an interdisciplinary and cross-cultural long-term project dedicated to studying so-called ‘written artefacts (WA)’. Our work introduces immersive technologies such as virtual, augmented, and mixed reality to the research cluster for the first time. In this paper, we outline the scope of our research project and present our current implementations of immersive applications based on two scenarios involving inscribed spaces. So far, immersive technologies have not been used in academia to create research focused applications for exploring, analysing, and understanding WA within their inscribed space, including providing access to appropriate spatial and temporal contexts. Thus, we collaborate closely with researchers from the humanities to create interactive and immersive applications for the novel field of WA research. The results of our preliminary user study show high ratings in the sense of presence in the virtual environments and indicate that immersive spatial context could add new perspectives for understanding WA. We hope to provide valuable insights on the design of immersive applications to support future research in novel fields.

Architectures are always subject to transformation in time. When a historical building is seen by a tourist or by an occasional visitor, it appears like a complete and final artefact, showing its main characteristics like a clear example of a style or of an artistic phase. This interpretation may turn out to be mostly a simplification, while the building is often the result of a large set of interventions in time. It is the case of a large number of main and secondary architectures, changed according to the mutation of needs and tastes. In Fabriano, in central-eastern Italy, the St. Venanzio Church (later Cathedral) was subject to a significant transformation of the apse interiors during the XVII century, with the demolition of the original chapels’ asset and their replacement by a larger unitarian space behind the main altar. This intervention has afflicted all the frescos that were the decoration of those chapels, they were partially deleted by the new masonry works, covered by paint, or walled in the new shape of the church. Since its early discovery, this patrimony has received various restorations. The research for a virtual reconstruction was based on a detailed digital survey of the building.

We discuss the design requirements of a software platform for constructing immersive environments through handmade spherical perspective drawings in a performative setting, with concurrent interactive live feed of the spherical drawing’s VR visualization. We investigate current best practices and available software in order to extract functionalities, requirements, improvements, possible integrations and future developments. We map the base requirements of the software from three sources: the state of the art of drawing techniques for spherical perspectives (equirectangular, azimuthal equidistant and cubical), the available software for their practice and the experimentation with novel hybrid artefacts. For the latter, we use a node-based program that allows us to prototype the workflow before entering a pure coding stage. The desired software platform should integrate well within digital art practices, stimulate and facilitate the practice of anamorphic handmade spherical drawings, and expand spherical perspectives’ applications through the emerging media of Hybrid Immersive Models (HIMs).

The prevalence of immersive head-mounted display (HMD) social virtual reality (VR) applications introduced asymmetric interaction among users within the virtual environment (VE). Therefore, researchers opted for (1) exploring the asymmetric social VR interaction dynamics in only co-located setups, (2) assigning interdependent roles to both HMD and non-HMD users, and (3) representing non-HMD users as abstract avatars in the VE. Therefore, we investigate the feasibility of supporting Self-Embodiment in an asymmetric VR interaction mode in a remote setup. To this end, we designed an asymmetric social VR game, QuarantivityVR , to (1) support sense of self-embodiment for non-HMD users in a remote setting by representing them as realistic full-body avatars within the VE, (2) augment visual-motor synchrony for the non-HMD users to increase their sense of agency and presence by detecting their motion through Kinect sensor and laptop’s webcam. During the game, each player performs three activities in succession, namely movie-guessing, spelling-bee, and answering mathematical questions. We believe that our work will act as a step towards the inclusion of a wide spectrum of users that can not afford full immersion and will aid researchers in creating enjoyable interactions for both users in the physical and virtual spaces.

Volumetric Video is a novel technology that enables the creation of dynamic 3D models of persons, which can then be integrated in any 3D environment. In contrast to classical character animation, volumetric video is authentic and much more realistic and therefore ideal for the transfer of emotions, facial expressions and gestures, which is highly relevant in the context of preservation of contemporary witnesses and survivors of the Holocaust. Fraunhofer Heinrich-Hertz-Institute (HHI) is working on two projects in this cultural heritage context. In a recent project between UFA and Fraunhofer HHI, a VR documentary about the last German survivor of the Holocaust Ernst Grube has been produced. A second project started in collaboration with the University Munich, faculty of languages and literature and Geschwister-Scholl-institute for political science, creating a concept for a VR experience together with Dr. Eva Umlauf, the youngest Jewish survivor in the concentration camp in Auschwitz. This paper presents key aspects of volumetric video and details about both projects including a discussion about the user perspective in such a VR experience.

Image Garden is an AI-based toolbox for curating collections and designing smart exhibitions that was developed by ART+COM Studios [ART+COM Studios. 2021. ART+COM. https://artcom.de/en/] in the research project QURATOR [QURATOR Bündnis. 2021. Qurator. https://qurator.ai]. This paper describes the application scenarios underlying the development of Image Garden. It also presents the technologies integrated in its processing pipeline, and sketches Show Cases that were implemented to demonstrate and evaluate functionality and usability.

This paper presents the progress implemented during the AURA project, funded by the Creative-Europe program with project partners from Germany, Italy, and Ukraine. The project aims to create auralized applications for three music venues in each of the project countries, namely the Konzerthaus Berlin, the Teatro del Maggio in Florence, and the Opera House Lviv. Each will be digitally recreated and auralized before they are then used to conduct case studies. This paper gives insights into current digitalization and auralization techniques. The results of a digital survey will be laid out and the conception and implementation of a first auralized prototype using a hand-modeled 3D object from the Great Hall of the Konzerthaus Berlin will be demonstrated. Furthermore, the usage of auralization for touristic purposes will be investigated using artificial intelligence for an audience preference analysis. A conclusion will be drawn and a short outlook into the ongoing course of the AURA project will be given.

This paper presents a first prototype of a virtual Theremin instrument for accompanying film scenes with sound. The virtual Theremin is implemented as a hybrid application for the web. Sound control is achieved by capturing user gestures with a webcam and mapping the gestures to the corresponding virtual Theremin parameters pitch and volume. Different sound types can be selected. The application’s underlying research is part of the multi-modal digital heritage project KOLLISIONEN which targets to open up the private archive of the Russian film maker Sergej Eisenstein to a broader public in digital form. Eisenstein, a film theorist and pioneer of film montage, was particularly intrigued by the Theremin as an instrument for film sound design. The virtual Theremin presented here is therefore linked to a film scene from the 1929 Soviet drama “The General Line” by Sergej Eisenstein which was never set to music originally. In its first implementation state, the application connects music interaction design with digital heritage in a modular, flexible and playful way and uses contemporary web technologies to enable easy operation and the greatest possible accessibility.

In this paper we present different approaches for interdisciplinary collaborations between art and culture professionals resp. cultural institutions and the interdisciplinary XR experts of our research group. We discuss lessons learned and best practices devised from multiple years of experience in interdisciplinary collaborations. Different project settings need different solutions: With our contribution, we hope to show exemplary ways for successful interdisciplinary collaborations at the intersection of culture and computer science – and to show the potential of such projects for the development of new ways and tools of collaboration.

Interaction becomes increasingly digital, including interactions with public authorities, requiring websites to be accessible for all. The strong focus on written words in digital interactions allows for assistive technology to improve access for many users. However, it might impede usability for users with reading and writing difficulties. The present paper examines whether guidelines such as the Web Content Accessibility Guidelines (WCAG) sufficiently cover users with dyslexia and how usability can be improved for this user group. This paper expands a previously published version at the Mensch und Computer 2021 conference [1]. Using literature research and interviews with users with dyslexia and focusing on an application of the WCAG on the country level (a German law regulating accessibility for e-government websites), we confirmed and identified gaps in the WCAG for this group. We focus on within-site search, as this function is frequently used to find relevant information, esp. on infrequently visited sites such as e-government websites. Modifications to improve search were developed based on literature and the results of the interviews. They were empirically evaluated in an online study with 31 users with dyslexia and 71 without. Results indicate that an auto-complete function, a search that compensates for spelling errors, an indicator that the search was corrected, search term summary information, and avoidance of capital letters were useful for both groups, while wider line spacing should only be used in end-user customization.

Frequent repetition of vocabulary is essential for effective language learning. To increase exposure to learning content, this work explores the integration of vocabulary tasks into the smartphone authentication process. We present the design and initial user experience evaluation of twelve prototypes, which explored three learning tasks and four common authentication types. In a three-week within-subject field study, we compared the most promising concept as mobile language learning (MLL) applications to two baselines: We designed a novel (1) UnlockApp that presents a vocabulary task with each authentication event, nudging users towards short frequent learning session. We compare it with a (2) NotificationApp that displays vocabulary tasks in a push notification in the status bar, which is always visible but learning needs to be user-initiated, and a (3) StandardApp that requires users to start in-app learning actively. Our study is the first to directly compare these embedding concepts for MLL, showing that integrating vocabulary learning into everyday smartphone interactions via UnlockApp and NotificationApp increases the number of answers. However, users show individual subjective preferences. Based on our results, we discuss the trade-off between higher content exposure and disturbance, and the related challenges and opportunities of embedding learning seamlessly into everyday mobile interactions.

While traditional videoconferencing causes privacy issues, virtual meetings are not yet widely used. Their communication quality still lacks usability and important non-verbal communication cues, such as body language, are underrepresented. We aim at exploring virtual avatars’ body language and how it can be used to indicate meeting attendees’ communication status. By comparing users’ perceptions of avatar behavior, we found that avatar body language across gender can be an indication of communication willingness. We derive resulting body language design recommendations and recommend using attentively behaving avatars as default body language and to indicate being busy through actions of the avatar, such as drinking, typing, or talking on a phone. These actions indicate that users are temporarily busy with another task, but still are attending the meeting. When users are unavailable, their avatars should not be displayed at all and in cases of longer meeting interruptions, the avatar of a user should leave the virtual meeting room.

Hololingo! is a social virtual reality app for real-time immersive distance learning of German as a Foreign Language (GFL). The acquisition of discoursive oral language skills for applied authentic contexts is challenging for group-based classroom settings and is often outsourced to autonomous analogue tandem learning. We operationalise a Digital Game-Based Language Learning (DGBLL) approach for distance learning, which relieves tandems from overstraining autonomy and self-guidance. Hololingo! supports tandems in their learning activities by providing entertaining communicative, collaborative, and didactically designed team tasks. These are embedded in a narrative adventure, target linguistic phenomena, and support the holistic acquisition of oral discourse competencies and fluency. The combination of the tandem principle, immersive Social VR gamification, barrier-free access to a global pool of expert/native speakers, and a curricular connection through task selection facilitates language learning and provides transcultural contacts. App development follows a Design-Based-Research cycle of conception, implementation, and evaluation. The results of a qualitative discourse analysis of the learning interaction are discussed with regard to tandem role behaviour, the relation between task design and communication as well as affective and creative behaviour.