iSAM – towards a cost-efficient and unobtrusive experimental setup for situational awareness measurement in administrative crisis management exercises

2.1 Situational awareness in administrative crisis management

Situational awareness describes the internal representation of an individuum about the situation it is situated in Ref. ³ Being actually situational aware covers the mental state of having a factual correct representation of the environment and has been researched since at least Clausewitz describing the absence of situational awareness as fog of war. ⁴ The concept is researched as part of decision-making processes and is in both academia and practice commonly accepted as predicator for good decisions. For instance, Endsley has detected a correlation of r = 0.46 between situational awareness and task performance in a meta study over several domains, where situational awareness was measured through several direct measuring approaches. ⁵

Accordingly, situational awareness also serves as proxy for measuring the effectiveness of software in crisis situations. ^{5 , 6} Due to its importance, it is highly applied in the domain of human-computer interaction: while modern research originated in the domain of air fighting ⁷ its application has widened to a broad variety of domains, such as health and medical management, ^{8 , 9 , 10 , 11} (cultural heritage) building management, ^{12 , 13} cyber security, ^{14 , 15 , 16 , 17 , 18} crisis management ^{19 , 20 , 21 , 22 , 23 , 24} or industrial applications. ^{10 , 25}

Endsley defines situational awareness as the level of understanding needed to perceive pertinent elements in the environment within a defined timeframe and spatial context, interpret their significance, and anticipate their future status. ¹⁵ Hereby, Endsley splits situational awareness into three levels: perception, understanding and prognosis. ²⁶

1. Perception: This is the foundational level where individuals gather information about their environment through observation and sensory inputs. It involves recognizing and identifying relevant data.

2. Comprehension: At this level, individuals take the gathered information and begin to understand the meaning and significance of what they perceived. It involves processing and organizing the data to create a coherent mental representation of the situation.

3. Projection: This is the highest level, where individuals use their understanding of the situation to make predictions about what might happen in the future. It involves the ability to anticipate potential developments and outcomes based on the current situation.

Endley’s approach frames situational awareness in stages, making it highly applicable to assess human-machine-interaction, as it allows analyzing the achieved level of information instead of focusing on the perception process that constitutes situational awareness. Figure 1 shows situational awareness as basis for decision making in the domain of crisis management and gives examples for required information in the given levels of awareness.

Figure 1:

Crisis situation awareness in the crisis management decision making process, own representation based on Ref. ²⁷

The importance of situational awareness intuitively transfers to the domain of crisis management and has been emphasized by crisis management practitioners in the aftermath of disasters. In research it is argued that situational awareness plays a crucial role in the management of crisis situations, as it supports managing the time-critical and safety-relevant processes significantly contributing to the successful management during all phases of crisis situations. ^{19 , 28 , 29 , 30 , 31} The practitioners view undermines this aspect: according to the responsible emergency management team, only through immediate on-site information about the situation in Trier during the 2021 central-european floods, an appropriate response could be organized, leading to a reduction in the impact of the flood. In contrast, the investigative committee regarding the catastrophe during the same flood in the Ahrtal area, deplored the absence of a coherent, reliable situational awareness repeatedly; through this, it was argued, the extent of the disaster was not known to the emergency operations center, and the responses were deemed inadequate. ^{32 , 33 , 34} It has been pointed out that a situational aware crisis manager can make informed decisions improving his crisis management performance.

As the significance of situational awareness as predicator for task fulfillment in crisis management is consensual, researchers often try to measure the level of situational awareness gained by users of a system. ³⁵ We assume this to be particularly true for crisis management software.

2.2 The structures of crisis management in Germany

To understand the requirements of administrative crisis managers to crisis situational displays, we explain the structures of crisis management in Germany, which is in general organized on a federal level. In the following, we explain its organizational structure at hand of the federal state of Baden-Württemberg, shown in Figure 2. In Baden-Württemberg the Ministry of the Interior (Innenministerium) serves as the Highest Disaster Protection Authority. The administrative districts (Regierungsbezirke) can be distinguished in the figure by the different coloring. Stuttgart is colored orange, Tübingen in red, Freiburg in yellow, and Karlsruhe in green. These administrative districts function as the Higher Disaster Protection Authorities, while the counties (Landkreise) and independent cities (Kreisfreie Städte), along with their respective specialist departments, constitute the Lower Disaster Protection Authorities. ^{36 , 37} The borders of the counties and independent cities are indicated in the map as well.

Figure 2:

Baden-Württemberg, its counties, independent cities and administrative districts. ³⁸

The hierarchical structure is reflected in Figure 3: 44 counties and independent cities report to 4 administrative districts reporting to the Ministry of the Interior. The responsible jurisdiction for managing a situation is determined based on the capacity to manage it effectively. In general, the responsibility for handling an emergency situation primarily lies with the lower disaster protection agency within whose jurisdiction the situation occurs. For disaster protection tasks that extend beyond the jurisdiction of a lower disaster protection agency, the higher disaster protection agencies are responsible. They are also accountable in special events in disaster protection, such as in the vicinity of nuclear facilities (nuclear situations). The highest disaster protection authority becomes responsible when disaster protection tasks extend beyond the boundaries of an administrative district or if specific regulations demand it to (for example in the case of a terrorist attack). Higher level authorities can assume control of an operation in the lower authority or delegate it to another disaster protection authority. ³⁶ When managing a crisis, jurisdictions across all hierarchical levels can be involved in the management of a crisis; yet, each entity might have different access to the given information, where usually the highest disaster protection authority has the highest access level and the lower disaster protection agency has the lowest access level.

Figure 3:

Organization of the public crisis management in Baden-Württemberg.

2.3 Putting dashboards into a crisis management perspective

A common tool for the creation of situational awareness are dashboards, ³⁹ also more precisely described as computerized displays reflecting the current state of the world. ⁶ Dashboards originate from the field of business intelligence, offer a visual representation of information and are commonly used to provide situational awareness for crisis managers. ⁴⁰ A definition for dashboards was established by Stephen Few in 2006. ⁴¹ He defines a dashboard as a visual display of the most important information needed to achieve one or more objectives, consolidated and arranged on a single screen so the information can be monitored at a glance. ⁴¹ Nowadays, interactive dashboards, which employ multiple pages or views are commonly used. These dashboards also support further data filtering through user inputs. ⁴² Therefore, a dashboard is no longer limited to interaction “at a glance”. Few defines three categories for dashboards:

Strategic dashboards display information at a high level of abstraction and provide predictions about future trends. They are used for formulating and monitoring long-term goals. Therefore, real-time data is not displayed.

Analytical dashboards focus on comparing and visualizing historical data. In this category, interactive components, such as filtering, are relevant. These are used to understand problem sources and enable users to forecast information into the future.

Operative dashboards in contrast give insights about more dynamic situations. They sum up activities whose constant changes are relevant and must be displayed as soon as possible. Operative dashboards should enable managers to make time-critical decisions, which is why changes must be immediately recognizable, for example through color indications.

In crisis management, dashboards, more commonly referred to as situational displays, can emerge from all three previous categories, depending on their specific use cases. While the tactical division of crisis management, specifically the administrative crisis management, requires information on a high level to fulfill its tasks, first responders and their staff utilize operational dashboards. ⁴⁰ Thus, both strategic and operative dashboards can be found in crisis management, each one with their own area of application and contributions. Best practices and standardized instructions in crisis management recommend the use of dashboards to handle high information throughput and enhance situational awareness during crises. ⁴³ This is in line with the political desire for developing user-friendly and effective systems to provide clear and concise information for crisis management across all hierarchical levels. ^{2 , 44 , 45}

2.4 Methods determining the effectiveness of crisis situation displays

To develop effective displays, that allow users to gain an understanding of the situation, one must first understand how people process the vast amount of data available in a crisis situation. ⁴⁶ This leads the interested reader to the question, about how developers and researchers can measure the level of situational awareness generated through situational displays. As situational awareness is critical to performance, many efforts have been undertaken to improve system capabilities to provide users good situational awareness. ⁴⁷ To reach this goal, Durso states the necessity of integrating situational awareness as target variable into the engineering process to actually control the level of situational awareness generated by a product under development. ³⁵ This is supported by the Sendai Framework, a global agreement adopted in 2015 aimed at reducing disaster risk and enhancing resilience. To achieve this, it outlines demands such as considering anthropogenic factors and creating tools that are both user-friendly or clear and concise. ⁴⁸ Commonly, these dimensions applied to quantify the experience a user has when using a software product and are considered very important in modern software development. ⁴⁹ Nevertheless, their validity for the domain of crisis management tools is topic of an ongoing discussion. ^{50 , 51} For the sake of brevity, we do not want to take part in this discussion and focus on the target variable situational awareness. In the following, we therefore introduce different methodologies from the literature, that can be applied to measure situational awareness allowing developers to create situational displays in a goal-directed manner.

Many attempts have been established to measure situational awareness. Figure 4 shows how Endsley distinguishes process measures, performance measures and direct situational awareness measures, ⁵² where situational awareness is usually measured on the individual level. Approaches exist to calculate the situational awareness of a given team based on the situational awareness of its team members. ⁶ Studies on situational awareness in distributed teams exist, yet have not left the theoretical level and have not delivered a standardized approach. ⁵³

Figure 4:

The different situational awareness measurement approaches, own representation based on Ref. ⁵²

Process measures measure the cognitive process to form and maintain situational awareness. ³ A common tool is eye tracking, that tracks the participants fixations on the display to indicate, which components were observed. Eye-Tracking gives insights about the elements that draw attention of the user. They show visual paths and hotspots that are most viewed. This can be used to track down elements that are visually most used, but also to sort out elements irritating the user. Thus, it helps in analyzing the physical process of utilizing a software. de Winter argues it to be more effective in situation awareness measurements in certain states of a task environment and provides the advantage of less interruptive testing procedures compared to other methods. ⁵⁴ Yet, measures like these research the source of situational awareness, rather than the actual understanding of the situation.

Direct situational awareness measures capture the operator’s perceptions directly, rather than inferring them from behaviors that may be influenced by various other factors besides situational awareness. ⁵⁵ Additionally, it does not require subjects or observers to make judgments about situational knowledge based on incomplete information. By collecting samples of situational awareness data in this manner, situational perceptions can be gathered immediately while they are fresh in the operators’ minds. ⁵⁶ Commonly used direct measures for situational awareness are Situational Awareness Global Assessment Technique (SAGAT), Situation Present Assessment Technique (SPAM) and Situation Awareness Rating Technique (SART).

Performance measures measure situational awareness indirectly by observing performance indicators, such as response times or number of errors during a software usage. They can often be automatically collected without any ado of the tested subject. Performance measuring studies collect for example information about hazard detection or crash avoidance when assessing a new feature of a car. ⁵⁷ Performance measures do not measure situational awareness and often deliver insufficient sensitivity and diagnosticity. ⁶

3.1 Research objective

Although both researchers and participants of the mentioned exercise are motivated to analyze situational awareness in a real world scenario, we faced challenges not only inherent to crisis management exercises, but simply practical issues: the federated nature of large-scale crisis management meant that not every crisis management staff room could be overseen by a researcher; in fact, we assumed no researcher was allowed in the situation room during the crisis management exercise at all. Furthermore, our approach needed to address the decentralized nature of administrative crisis management and facilitate querying different individuals simultaneously across various geographical locations. Given that our study was not the primary focus of the exercise, we aimed to develop an unobtrusive process with minimal user demand, both in the experimental setup phase and exercise. The rarity of these exercises urged us to proceed cautiously and design a meticulous experimental setup.

Therefore, the primary objective of our study is to develop an experimental setup for measuring situational awareness in a manner that is minimally invasive to users, acknowledges the decentralized and geographically dispersed nature of administrative crisis management teams, and operates without the presence of researchers in the staff room. The method must be capable of providing insights into the software features and information necessary for administrative crisis managers to make informed decisions and mitigate the impact of a given crises. We formulate these challenges into the research question: How can an experimental setup be designed to measure situational awareness in large-scale, decentralized crisis management exercises in a minimally invasive manner, while providing valuable insights into the software features and information necessary for administrative crisis managers to make informed decisions without the physical presence of researchers in the staff rooms?

By answering this question, we try to contribute an experiment design and a software implementation, that allow the human factor community to conduct situational awareness measurements in the domain of administrative crisis management taking its regionally distributed nature into respect. Effectively, we aim to enable crisis situation displays to inform crisis managers in a way they can take good decisions and reduce the potential impact of a crisis.

3.2 Formative research for the experimental setup

To gather formative information about the design space for our experiment, we have conducted two online workshops. The workshops took 6 h each, including breaks, and were attended by n = 12 participants (six experts in each meeting). We cannot discuss further traits of the participants, as we did not receive clearance for further information disclosure. The workshops were supplemented by a series of unstructured interviews with members from administrative crisis management in Baden-Württemberg. Participants and interviewees were in active duty for disaster management in municipalities, administrative districts or on federal state level. The Ministry of the Interior requested participants from the administrative crisis management to participate in the workshop; participants, who indicated availability were invited and subsequently took part in the workshops.

During the workshop, we primarily collected requirements for a new crisis situation dashboard for the authorities in Baden-Württemberg and discussed integrating a SAGAT study into a planned crisis management exercise in which a crisis situation display could be assessed. The workshops were recorded as videos. In a post-processing phase, the workshop recordings were transcribed according to the transcription methodology of Dresing and Pehl. ⁶³

During the workshops and interviews, we have discussed the restrictions, under which we would be allowed to participate in the exercise and analyze the subjects’ situational awareness. The formats were unstructured in nature as we needed to understand the given dos and don’ts so we could attach our study to the planned crisis management exercise. During the exercise, different hierarchical levels will participate and collaborate using the information showed in their in-production crisis situation display. The subjects participating in the exercise are professionals in their field and will exercise from their situation room, where they will try to create an understanding of the exercised crisis at hand, communicate and give out orders to overcome the given challenges. The exercise will take place on a high-level strategic basis and will focus on enhancing cooperation among various governmental departments and offices located in different regional entities. Ministries and authorities from both federal and state levels will collaborate with other civil protection actors to simulate an extraordinary crisis. To allow participants to concentrate fully on interinstitutional coordination, the responses decided upon are not implemented but are simulated and evaluated by the exercise controllers. Thus, the main goal of the exercise will be the strategical aspects of administrative crisis management: these are practiced periodically and help sensitizing crisis management teams from the administrative level about various crisis scenarios. ^{64 , 65} This is the context, in which we are allowed to research the level of situational awareness achieved through the utilized dashboards. During this, we must ensure that the exercise is not disturbed “too much”; thus, the experiment must ensure an unobtrusive way of measuring the situational awareness of the participating subjects. Additionally, triggering the querying of the users must be done manually. The possibility, that this trigger also required approval from a manager of the exercise, was mentioned in the workshops. Furthermore, the distributed nature of administrative crisis management imposes requirements regarding the test setup. The experimental setup must therefore be easily deployed over the different workstations and due to practical questions of funding, must work unsupervised, as we most likely will not have enough resources to afford a researcher overseeing the exercise in every control room.

From the transcription, requirements were elicited using Mayring’s inductive categorization approach. ⁶⁶ In a front-to-end approach, the transcripts were analyzed for identified functional and non-functional requirements. Requirements that had not been previously established were documented. For those that had already been defined, a reference to the previously documented requirement was made. The deducted requirements for the experiment are listed in Table 2. The list comprises behavioral, as well as practical aspects.

We supplement the given requirements with Endsley’s suggestions for a test of situational awareness. ²⁷ By enhancing the given practical requirements with theoretical considerations, we want to ensure our given requirements list is grounded in well-established principles, increasing the robustness and scientific reliability we want to deduce from them (Table 3).

3.2.2 The iSAM-architecture

The given circumstances, as well as the listed requirements, design goals and research object led to the decision to develop a new framework, that would allow the measurement of a user’s situational awareness when using a web-based crisis dashboard. Thus, have developed a framework for integrated Situational Awareness Measurement (iSAM). It consists of web and server components that offer SAGAT-tests that can be integrated into web-based situational displays. In this section, we describe the implementation of iSAM.

At the core of the architecture stands the crisis situation display, which will be analyzed regarding the situation awareness it offers its users. iSAM offers components based on the web technologies JavaScript, HTML and CSS that be integrated into this dashboard; the frontend component connects through WebSockets ⁶⁸ to the iSAM-backend, from where the issuing of SAGAT-questionnaires is triggered.

The architecture of iSAM is shown below in Figure 5. The object of investigation – here the crisis situation display – is supplemented with the iSAM web component and shows (changing) data. iSAM is indifferent about how the data within the dashboard changes.

Figure 5:

The architecture of iSAM.

3.2.2.1 The iSAM-frontend

The frontend of iSAM holds user interface implementations and opens a connection channel to the backend. To enable appropriate question and response modalities, input types such as free text, radio or checkboxes are available. Other, more complex answer types are possible as well, such as maps on which the user needs to locate the region of a crisis, as shown in Figure 6. It shows a web-based dashboard with a region indicated in yellow. After triggering a questioning session from the backend, the screen blurs, as shown in the middle picture and finally shows a questionnaire in the last image, in which the user should indicate the regional entity, in which the crisis takes place.

Figure 6:

From dashboard to iSAM-questionnaire.

All HTML, JavaScript and CSS elements are encapsulated and require no interaction with the research object. The frontend component is realized with a Bootstrap-Modal, which covers all other visible parts of the research object; the questionnaire overlays the dashboard preventing visual aids (REQ12). Bootstrap is a free and open-source HTML, CSS and JavaScript framework allowing the development of responsive web-applications. ⁶⁹ The WebSockets-based connection to the backend is used to deliver the questions from the backend to the front-end iSAM JavaScript component, as well as to end the survey and collect the given responses.

3.2.2.2 The iSAM-backend

The iSAM-backend is developed in Java 11. It is used to model questions and response types and is responsible for issuing questions and collecting user responses. It also opens a WebSocket-connection to which the frontend-component can connect to.

The management of questions and their types in the backend is handled in the backend at hand of two attributes: the central attribute is the data structure where the specified questions are stored. A question is modelled by an according Query class, which includes the question itself and its associated answer type, such as a selected region or an answer from a list, modelled through the class Answer. The answer type defines which UI-component will be prompted in the frontend. Additional answer types can be added through object-oriented programming, inheriting from the Answer class. Questions and answers can be serialized by the iSAM-backend from the given data structure to JSON and vice versa. As SAGAT demands querying the situational understanding based on several levels, the data structure distinguishes the questions regarding their targeted situation awareness level.

As required by SAGAT, the handler can issue questions in a randomized time interval but also allows handing out the questions on request. When a questionnaire is triggered, a request is issued from the backend to the frontend. Hereby, the client is informed to retrieve a new set of questions. The following message object contains the question itself, as well as the type of visualization needed to respond to the questions. On user demand, or after a defined time, the (backend) application requests the JavaScript component via the WebSocket to collect the responses and return them to the backend where they are stored for further processing. This process is depicted in the Business Process Modell and Notation (BPMN) flowchart in Figure 7.

Figure 7:

BPMN chart depicting the communication for question retrieval and view rendering.