Immersive Inscribed Spaces – Bringing Virtuality to Written Artefacts for Humanities

4.1 Immersive City Scripts: Inscriptions and the Construction of Social Spaces in Miletus

Figure 1

Archaeologist doing field research and recording an inscription in the ancient theatre of Miletus. ©A. L. Osthof.

This large-scale outdoor scenario focuses on inscriptions in the ancient theatre of Miletus in Asia Minor (Turkey) (see Figure 3) from the Greco-Roman period, which range from hastily engraved graffiti to carefully carved letters. Their perception is influenced by the particular text and their formal design, material, and location, which reveal the spatial constructions of the public space. For example, so-called topos-inscriptions were created to permanently represent different groups of people in the cityscape and structure public spaces. This project involves researchers from classical archaeology, epigraphy, ancient history, and HCI. Over 300 inscriptions and other incised signs of usage, such as game-boards, will be comprehensively recorded and analysed in their entirety for the first time. While most of the WA are located on top of the seating stairs, others are found in the profile of the stairs and on walls or columns inside the remains of the ancient theatre. Field research for collecting and recording the WA data is being done at the theatre of Miletus, as illustrated in Figure 1. To give some simplified examples, the WA are carved into the stone and can be an inscription written in ancient Greek, a game-board consisting of squares or circles, or a little drawing among others. They can be large, for example, spanning across a whole seating stair or tiny and shallow or hidden inside the profile of stairs which makes them hard to detect. The recorded findings include the location, carrier, size and a detailed description of the WA among other information. In addition, photos of the WA are taken from different angles and distances for documentation. All findings will be digitized and integrated into a digital version of the theatre, where the real-world positions of the WA are mapped to the digital 3D model. We created the model from 3D scanning and photogrammetry data captured at the theatre. The theatre of Miletus used to have a seating capacity of about 15,000 people on three tiers^[3] but today, only the first tier and parts of the second tier remain. Therefore, former structures such as the stage and third tier of the theatre were digitally reconstructed. During the project, we are implementing immersive applications, for example for virtual reality headsets, and integrate the digital data. The VR application allows the researchers to walk around and explore the virtual theatre and view the reconstructions. Furthermore, researchers can interact and view the collected information on each WA. We are testing various functions for visualizing, analysing, and categorizing the recorded data on the WA inside the virtual theatre. With the applications, we create an immersive inscribed space in which researchers can gain new perspectives, study the WA with spatial and temporal relation, and visualize ancient social practices over time.

4.2 The Interior of the Church in Lucklum: A Compendium of Early Modern European Emblematics

Figure 2

Left: XRF-scanning of an area of an emblematic painting; right: intensity image of one scanned area. ©S. Bosch.

The church in Lucklum (located in Lower Saxony, Germany) houses an outstanding collection of 209 Latin inscriptions and 156 emblematic paintings from the early eighteenth century (see Figure 4), collected and published by Steiger et al. [27]. This indoor scenario covers a multi-story enclosed space and examines particularly relevant constellations of the WA and how they structure the church’s interior as a space of meditation and prayer. The inscriptions and emblematic paintings are applied to wooden panels which are located all over the interior of the church — with structures near the ground, at the church’s pulpit, on the gallery, on the walls, and on the ceiling. The emblematic paintings come in different sizes and shapes. Most of the Latin inscriptions are located on top and below the paintings. This project involves researchers from the fields of theology, literature, art history, materials science, and HCI. Part of the project is dedicated to the non-invasive detection and reconstruction of potential earlier, hidden layers in the inscriptions and emblematic paintings in the interior of the church, which were painted over in the early eighteenth century. For this process, X-ray fluorescence analysis (XRF-scanning) is used which is a method for determining the elemental composition of inorganic materials, in this case inside the used paint. The XRF-scanning process and an example of the output image for an emblematic painting can be seen in Figure 2. The XRF-scanning outputs colour mapped square images of each scanned area, which shows the intensity of the measured X-ray fluorescence. The detailed results of this material analysis and the architectural-historical interpretations will be published in corresponding further publications within the scope of the UWA project. We digitized the interior of the church based on laser scanning data and high-resolution photographs of the emblematic paintings. The 3D model, inscriptions, emblematic paintings, and uncovered layers will be integrated into various immersive applications within the course of the project. In the VR application, researchers can freely explore the interior of the church and view the emblematic paintings in real size. In addition, information on each emblematic painting and their corresponding inscriptions is extracted from the publication of Steiger et al. [27] and integrated into the applications. The information includes the titles of the paintings, sources, and descriptions among others. Reference pictures for the motifs depicted in the painting will be included as well. The applications allow researchers to explore and study the WA with spatial context in an immersive environment, such as inside the digitized interior of the church in VR. Moreover, the visualization of earlier hidden layers provide additional temporal context for analysing the WA and their constellations.

Figure 3

(a) photo of the theatre of Miletus ©A. L. Osthof and C. Berns, (b) 3D model of the theatre with reconstructions, (c) reconstructed stage, and (d) side view of theatre with reconstructed 3rd tier and seating rows.

5.1 Virtual Reality – HMD and CAVE

For the virtual reality setting, we developed two VR applications based on each scenario for use with head-mounted displays (HMDs). We chose to use the standalone Meta Quest 2 HMD in the current project phase (and due to the COVID-19 restrictions) as this allowed us to send out the devices to the WA researchers from the humanities. Within the VR applications, users can explore the true-to-scale 3D models of the inscribed spaces. An example of the interaction in the VR environment is shown in Figure 7. Inside the virtual inscribed space, reconstructions of former structures of the ancient theatre in Miletus (see Figure 3) and hidden layers in the emblematic paintings inside the church in Lucklum (see Figure 5) are visualized. Moreover, the real-world locations of the WA are marked with respective 3D pins or markers in the virtual environment and users can open spatial information panels containing the digitized WA data at the respective WA locations as displayed in Figure 6.

Figure 7

VR application of the church’s interior showing the teleport and pointer highlight function.

Figure 8

CAVE in monoscopic view mode without positional tracking of the user, allowing multiple users to join and leave at any time. CAVE shows the reconstructed stage of the theatre.

We also created a prototype for displaying the theatre of Miletus including the reconstructions and information panels inside the Cave Automatic Virtual Environment (CAVE) in our HCI laboratory, which is pictured in Figure 8. Our 4-sided CAVE setup uses four full-HD projectors with three walls and the floor as projection surfaces and has a total size of 3.15 × 4.2 × 2.36 metres (D × W × H). A detailed description of the technical setup including an illustration of our CAVE is given by Schmidt and Steinicke [25]. The user’s head position and rotation are tracked with retroflective markers and an optical camera system by OptiTrack. The markers are attached to active shutter glasses and defined as a so-called rigid body within the tracking system. The shutter glasses allow the CAVE projection to be displayed in two modes — a stereoscopic view mode which shows one image for each eye and a monoscopic view mode as pictured in Figure 8.

The current CAVE setup allows interactive exploration of the inscribed space via marker-based head pose tracking and simple controller and keyboard input. In the future, spatially tracked controllers (6DoF) and the interior of the church will be added to the application.

5.2 Augmented and Mixed Reality

As a major part of the work for the WA researchers takes place on-site, we are investigating the use of immersive technology to support and improve their current workflow. In our project, we are exploring the use of mixed reality HMDs such as the Microsoft HoloLens 2 or the Nreal Light glasses for displaying spatial mixed-reality content inside the real world. In addition, we are also prototyping and testing mobile augmented reality solutions for smartphones to make the applications more easily accessible to a larger group of users.

Figure 9

Left: testing overlay function of smartphone AR application with a digital display; right: Nreal light showing 3D pin and information panel on theatre stairs, captured in Miletus.

We built two prototypes to display hidden layers and spatial WA information panels at their respective real life locations. For our first prototype, we created a mobile AR application for the emblematic paintings in Lucklum, which uses Unity’s AR Foundation framework with image tracking to display the scanned hidden layer on top of the respective painting. To create the image marker, a picture of the unique emblematic painting is imported into a reference image library. The hidden layer is shown as an overlay as soon as the image marker is recognized. The opacity of this overlay can be manually controlled by the user through a UI slider on the display, as shown in the screen capture in Figure 9. The painting’s name and numbering are also displayed on the smartphone screen. Due to the COVID-19 restrictions, we were unable to travel and test our application on-site in the church. Therefore, we use a vertical digital display screen to show emblematic paintings for development and testing purposes, as pictured in Figure 9. Our second prototype uses the Nreal Light glasses to display holograms of the 3D spatial markers and inscription information panels inside the theatre of Miletus, as seen in Figure 9. Currently, users can place a small selection of 3D Pins and interact with them to toggle the information panels. In the future, we will work on additional AR/MR concepts and plan to develop and test applications for the Microsoft HoloLens 2. One interesting future use case for the large-scale scenario in the theatre of Miletus would be to help users find the inscriptions at their real-world location. Since the placement of the 3D WA pins in the true-to-scale 3D model of the theatre in VR application correspond to the real-world positions of the inscriptions, GPS-coordinates can be calculated based on their location in the virtual environment. The coordinates could be used to provide users on-site with approximate positions of the inscriptions, as well as display additional information on the large amount of WA in the theatre.

Figure 10

Left: researchers discussing the content of the WA information panel on the multi-touch table; right: adjusting opacity of the emblematic painting overlay on the multi-touch table.

5.3 Multi-Touch Interfaces

We also use the digitized WA data with multi-touch interfaces. Our current prototype supports touch interfaces with native touch input, as well as the TUIO-protocol^[4] for custom multi-touch displays. This allows us to run the application on our stereoscopic multi-touch table as well as on commercially available touch screens and tablet devices. Our current implementation uses our custom-built multi-touch table in monoscopic view mode (see Figure 10) with TUIO input. The touch table consists of a 55-inch ultra-high-definition TV display equipped with the infrared multi-touch frame G4S by PQ-Labs. This setup is attached to an adjustable mounting stand for changing height and inclination, which allows the display to be used in both vertical and horizontal modes. The stand is also movable and carries the PC the display is connected to. Our current implementation offers direct manipulation through multi-touch touch gestures such as tapping, scaling, rotation, and translation of WA information panels and emblematic paintings. Moreover, it provides UI controls for displaying and changing the opacity of the hidden layers on the emblematic paintings. The use of a large multi-touch display for both scenarios could offer additional benefits such as collaborative work where users can freely discuss, sort, and arrange information on the WA. Moreover, the multi-touch prototype can be used during the user-centred design and development of the immersive applications for discussing specific aspects such as layout and design for the WA information panels as pictured in Figure 10.

6.1 Study Design

For the preliminary study, we tested the applications with seven participants (2 female, 5 male), aged from 28 to 54 (M=39.57, SD=11.47), including five members of the UWA Cluster, with expertise ranging from classical archaeology, ancient history, epigraphy, literary study, and HCI. The other two participants where PhD students recruited from outside the Cluster project — one from the field of HCI and one from ancient history. Five of the participants were inexperienced VR users with no prior VR experience outside using our applications. Out of the seven participants, two were VR expert users from the field of HCI, who had no expertise with analysing written artefacts. All participants signed an informed consent form before the study.

We conducted the study remotely via Zoom and provided the participants with Oculus Quest 2 HMDs. We gave them instructions on the use of the HMD and the Oculus Touch Controllers, as well as the basic interaction and navigation functions in the VR applications. As qualitative evaluation method, we used free exploration and thinking-aloud. Each participant was given 15 minutes to freely explore and interact with each VR environment. We tested both applications at once in order to compare the requirements of both the scenarios and possible differences related to the size and type of the immersive environment. Afterwards, we asked the participants to fill out two questionnaires, the ‘igroup presence questionnaire (iPQ)’ [23], [26] and the ‘System Usability Scale (SUS)’ [8].

6.2 Results

For the evaluation of the System Usability Scale (SUS) questionnaire, we calculated the overall mean (M) and standard deviation (SD) as well as the SUS subscales for Usability and Learnability according to Lewis and Sauron [18]. Following the rating and visualization proposed by Bangor et al. [1], the SUS adjective ratings and acceptability ranges for the overall and subscale scores are shown in Figure 11. Our applications scored within the acceptable SUS range on all three scales. Overall, the applications were rated ‘excellent’ on the adjective rating scale, with M=85.36 and SD=8.59. The Usability subscale (M=84.82, SD=10.43) falls slightly behind the overall rating, while the results revealed that the Learnability (M=87.50, SD=12.50) is excellent according to the adjective rating.

Figure 11

SUS mean and SD with overall and subscale scores.

The igroup presence questionnaire (iPQ) contains a General Item for measuring the general ‘sense of being there’ and three subscales: Spatial Presence, Involvement, and Experienced Realism. The iPQ uses a seven-point scale, with scores from 0 to 6, where scores below 3 indicate disagreement and scores above indicate agreement with the 14 iPQ items. The results of the iPQ and each subscale are shown in Table 1. The General Item (M=5.14, SD=0.90) and Spatial Presence (M=4.86, SD=0.62) were rated positively and well above the median value of 3 by the participants, indicating that the sense of being physically present in the virtual environment is perceived to be quite high. The Involvement (M=3.46, SD=1.21) was rated slightly above and the Experienced Realism (M=2.50, SD=0.80) slightly below the median value. The reduced involvement experienced in VR could be due to the limited amount of placed inscriptions and implemented interactions in the current development stage of the VR applications. We expected a lower rating for Experienced Realism, since the currently used 3D models are in a prototype stage and require additional 3D scanning data for further processing and improvement of the 3D meshes and textures. Since we only collected data from a limited sample of seven participants for this preliminary study, no further statistical data analysis was done.

In addition to the results from our questionnaires, we were able to get qualitative feedback and first insights on the applications from the thinking-aloud and our observations. All participants reported having a positive user experience (UX) in VR. They stated that they had the feeling of being right there at the locations and that standing in the virtual inscribed spaces, especially the large theatre of Miletus, felt very impressive. Everyone, including the inexperienced VR users, was able to securely navigate and interact with the virtual environments during the study. The inexperienced participants needed explanations for the teleport function, but were able to use it successfully after trying it out once. Two participants stated that they required a few trials to get used to being teleported. Another important observation was the intuitive selection of the inscription pins/markers and interaction with the information panels by all participants, although we did not instruct them on the implemented selection pattern beforehand. All inexperienced participants reported that the interactions in VR were easy to understand and execute.

We can also report some findings regarding actual benefits and advantages for using these immersive applications in the context of WA research. The researchers working on the ancient theatre of Miletus stated that being able to see and walk through the virtual theatre with the reconstructions allowed them to get a good sense of the distances, scale, and height of the structures. Moreover, spatial understanding, the architectural atmosphere, as well as paths through the theatre can be investigated by moving through the virtual theatre with its reconstructions. The researchers explained that this allowed them to gain new perspectives and a better spatial understanding for the analysis of the WA, which is not possible in the same way without the reconstructions on site at the real theatre. During the study, this advantage of the virtual environment already resulted in new questions for the WA researchers about the placement and importance of certain inscriptions — especially regarding an inscription located on a remaining statue base next to the stage that was not in question before. In Lucklum, spatial understanding was also of great importance to the researchers. In addition, the visualization of the hidden layers added a temporal aspect to the emblematic paintings within the inscribed space. The researchers stated that the findings from Lucklum and the uncovered hidden layers based on the XRF-scans have already led to the earlier church interior being dated correctly in comparison to previous research. They noted that the virtual environment gives the unique opportunity to easily visualize, compare and directly contrast the ‘layers’ inside the inscribed space (the interior of the church). They stated that this is especially helpful when working remotely, which is an aspect ever more important in times of restricted travel and access to actual sites. During the user study, new requirements and feature requests for investigating potential new research questions came up: Tools for visualizing distribution, clustering, and connections between different WA as well as teleport markers to special points of interest inside the virtual spaces.

Reported and observed issues regarding the interactions were the limited selection range and difficulties with precise selection and interactions of remote objects such as pins, markers, and information panels. Another issue was the overlapping of the information panels in certain areas.

Generally speaking, the qualitative feedback and observations match the results from the questionnaires, which achieved high ratings in Learnability in the SUS and for the sense of being physically present in the iPQ. The remote user study provided us with first insights on the existing challenges of interaction design for immersive research applications. We identified two important issues to address in future iterations: precise selection of distant objects and structuring of content in information-rich virtual environments. It has to be noted that the preliminary user study lacks detailed results and has limited validity due to the small sample size. In the future, further comprehensive user studies are needed to obtain more in-depth results. Nevertheless, we gained valuable initial insights on potential benefits of immersive research applications and challenges regarding the design of appropriate interactions which affect the usability and UX.