Breaking down barriers to warning technology adoption: usability and usefulness of a messenger app warning bot

2.1 Mobile warnings and warning app requirements

Warning apps provide a mobile channel that allows to personalise warnings based on locations, by selecting relevant areas or via GPS-tracking. ¹ In addition, relevant hazard types can be selected, e.g. severe weather warnings, bomb disposal, school closure or policing information. ² As such, they are distinct from cell broadcast, which is used only for severe civil protection warnings and sends alerts only based on the current radio cell of the mobile device. We analysed the literature on warning apps to identify warning and design requirements that might also be relevant for other warning channels. The literature has analysed user preferences regarding warning apps, ^{12 , 14 , 26} and reasons for warning app (non-)adoption and (dis-)continuance of use. ^{4 , 27} Other studies have contributed warning app design requirements ²⁸ and usability guidelines. ¹³ Compared to social media, people value warning apps for their fast and reliable information. ²⁹ For those who have used a warning app in a crisis, this information channel is perceived as more helpful than other online sources. ¹² Users prefer an app that centralises various types of warnings and allows personalisation based on personally relevant warnings. ^{12 , 14} In line with these findings, a study shows that the key factor that affects the duration of use of a warning app is its perceived usefulness. Users are more likely to continue using the app if they feel that it effectively serves its intended purpose – reliably providing critical information in an easy-to-use manner – and is free of errors. ²⁷

A study of app reviews suggests that users of warning apps discontinue using warning apps due to reliability issues, with users not receiving relevant warnings or receiving false alarms at other times. ³⁰ Usability factors also play a role: users discontinue using warning apps that have complex user interface graphics or require too much user input. ²⁷ In addition, warning apps have specific design requirements that differ from the requirements of other apps, which relate to simplicity and ease of use, as well as demanding little energy, bandwidth, and storage space, to account for the specific use case in crises. ¹⁷

Looking at users’ motivations regarding the use of warning apps, Fischer-Preßler et al. ⁴ find that maladaptive rewards – advantages derived from not using a warning app, such as saving battery and data space – have a negative influence on app adoption and continued use. Similarly, inhibition theory posits that potential losses are perceived more strongly than potential gains, ³¹ stressing the importance of effort expectancy and other perceived costs for digital service adoption. ³² Finally, citizens in state-oriented risk cultures have a low motivation to engage in crisis management and preparedness, ^{6 , 7} stressing the importance of low adoption costs.

Bonaretti et al. ²⁸ point towards the importance of timeliness and representational fidelity of warning apps, meaning that warnings need to be easily and quickly noticed and understood by users, e.g. through good visual representations of the crisis and a noticeable auditory and visual alarm signal. Furthermore, the app must be dependable with regard to its warnings, sending an adequate alert level for each relevant and only for relevant emergencies.

Finally, Tan et al. ¹³ present seven higher-level constructs with usability requirements, some specific to warning apps. They suggest (1) app dependability, consisting of instant start, data preservation, error-free operation and clean exit; (2) app design, including branding, orientation, minimal advertising, phone resource usage; (3) app utility, consisting of collaboration, content relevance and search; (4) UI graphics, with the requirements aesthetic graphics, realism and subtle animation; (5) UI input, including control obviousness, de-emphasis on settings, effort minimisation, fingertip size controls; (6) UI output, requiring concise language, standardised UI elements, user-centric terminology and audio output; and (7) UI structure with the elements logical path, top-to-bottom structure and few external links. In addition, Reuter et al. ³³ show that users are reluctant to share private information with state agencies, highlighting the need for privacy by design.

We identify a set of objectives for mobile warnings:

1. Usefulness:

   1. Adaptability: offer many relevant instances of interaction through personalisation ^{12 , 14 , 27}

   2. Representational fidelity: reliably represent the warning status, is free of errors ²⁸

   3. Timeliness: noticeable warnings, fast delivery ²⁸

   4. Criticality: reliably present critical warnings ³⁰

2. Resource efficiency: minimise used data space, energy resources, learning and maintenance effort ^{4 , 17}

3. Simplicity: little user input, simple UI graphics, fast information extraction ²⁷

4. Usability: optimised for critical situations and privacy ^{13 , 33}

2.2 Bot requirements

Chatbots are used in a large variety of fields, particularly in customer service support and e-commerce, but state agencies are also increasingly using them to offer information about regulations and public services. ³⁴ A study analysing the interactions with a public service bot has shown that users largely use very short, utilitarian rather than social inputs and that this brevity is technically justified, as it results in a vastly higher rate of adequate responses by the bot. ³⁴ Another study looking at older adults found that Internet competency had a great effect on whether social-oriented or task-oriented interaction styles were more beneficial in e-commerce, with task-orientation performing better for people with lower Internet competency scores. ³⁵

Within the public service category, a warning bot pertains to the category of content curation chatbots, which “have a chatbot-driven Locus of Control where the user initiative is limited to accepting or rejecting content offers, or requesting specific content types, serving to filter the presented content selection”. ³⁶ These are increasingly developing from one-off interactions to longer-term relationships with regular content provision. Because browsing and searching are less well-suited to dialogue interfaces, these types of chatbots “actively guide users to recommended content” by presenting options, often by using visual elements. ³⁶

Klopfenstein et al. ²⁵ identify that bots are increasingly run within IM apps to serve as functional replacements of mobile applications, e.g. as news bots instead of news apps. Such “botplications” offer a set of advantages for users and programmers: They are instantly available inside IM apps, require little data traffic, their push notifications appear as messages, the messages can be easily shared within the app and beyond, most messaging apps are independent of the operating system and the hosting platform offers user authentication. ²⁵

However, bots have some significant differences compared to apps, especially regarding the User Interface (UI). Conversational user interfaces (CUIs) are sequential, characterised by turn-taking. ^{18 , 37} In addition to text, they support pictures, stickers, videos, audio, generic data files and packaged data, such as geolocations or contact information. ²⁵ In messaging bots, services and information are provided as streams of messages and notifications. Dialogues can take the form of text, voice, video input, or images. ³⁸ They can be deployed on various platforms, including websites, messaging applications or social media platforms. In a bot, each message “should be conceptually seen as a micro application, while the conversation is a timeline of past application screens” 25], p. 559. A challenge also lies in guiding the conversation: “because of the free-form nature of the medium, it is easy for bot users to get lost and not to be certain of what commands or what exact syntax is required to perform the desired action” 25], p. 560. Therefore, the bot should suggest next actions, use buttons or suggested syntax as guardrails. ²⁵ To ensure guidance, simplicity, effectiveness, and to avoid errors, research also suggests avoiding natural language processing 25], p. 559. This rules out unexpected user input ³⁹ and avoids misunderstandings by the bot, ⁴⁰ which would lead to incorrect answers or a lack of responses, and in turn decrease trust in the bot, and thus negatively affect the user experience. ⁴¹ Users also often prefer buttons to entering text as this provides them with clear input options enhancing control 42], p. 257.

Conducting a literature review of task-oriented conversational agents, Göbel et al. ⁴³ identify five design requirements (goal orientation, usability, communication efficiency, human-likeness, user-centric integrity) and eleven design principles (goal extraction, conversation recovery, accessibility, adaptability, feedback and responsiveness, cultural sensitivity, natural communication, adaptive personality, emotional engagement, transparency, ethics and compliance).

The literature points to the following challenges and requirements for bots and CUIs:

1. Onboarding: introducing function and interactions ^{38 , 39 , 44}

2. User support and learning: providing operating aids and allowing users to learn about functions on the fly ³⁹

3. Context maintenance: saving previous inputs and being able to refer to them a at a later point in time ^{38 , 39 , 41 , 44 , 45}

4. Guided conversation: proactively suggesting follow-up actions, offering alternative choices, offering UI enhancements such as buttons or syntax commands ²⁵

5. Social intelligence: dealing with inappropriate and unspecified input ³⁹

6. Manage conversation breakdown: offering options of potential intentions after unfamiliar input ⁴⁶

7. Transparency: being clear about the process, especially for complex actions ³⁹

8. Autonomy: ensuring control over the interactions ⁴⁷

9. Efficiency: satisfying results in short time, ⁴⁸ quick responses to minimise user input ^{44 , 49}

10. Effort minimisation: reducing cognitive effort by structuring process intuitively ^{50 , 51}

11. Privacy: following data protection laws and collecting minimal user data ³⁸

4.1 Crisis context

To ensure reliability and trust, we integrate the API of the official and, in Germany, most widely used warning app NINA, which combines information from several relevant safety and security sources, such as the weather service and the modular warning system (MoWaS), the unified warning system of the German civil protection authorities. ⁵³ The requirements also implied that an already installed application would reduce the resources and maladaptive reward. In addition, the chosen platform should enable auditory and visual alerts. For visibility and quick access, it should be able to send push alerts. Due to the requirement of personalisation, users also need to be able to specify and save their preferences through a transparent process. Analysing the rise of conversational interfaces, ²⁵ finds that bots within messaging platforms fulfil many of these functions, making them attractive platforms for delivering news. ⁹ Additionally, Telegram is widely used as a news outlet in some countries, such as Singapore, with users particularly describing efficiency, convenience and effortless access as key benefits. ⁵⁴ IM apps are widely used. Globally, WhatsApp has 2 billion monthly users, WeChat has 1.3 billion, Facebook 1 billion, Telegram 0.9 billion, SnapChat 0.8 billion, and QQ 0.54 billion. ⁵⁵ While Telegram has fewer users, ⁵⁶ showed that it offers an open-source API ⁵⁷ as well as many features that are important for the usability of bots, ^{25 , 38} such as the possibility to create buttons, define quick replies, customise text messages, and publish media for groups or channels for a large number of members (see Table 7 in Appendix C for a comparison of IM apps). Despite the lower usage, we chose Telegram as the IM to implement our prototype in, as it offers most relevant affordances and thus allows us to build a solution that is more comparable to a warning app. Within the IM, a bot manages the subscriptions and message delivery based on users’ settings.

To implement the system, we selected Python 3 and the pyTelegramBotAPI library to facilitate communication with Telegram. We also use the NINA API to retrieve relevant information on warnings and disasters. The bot’s logic comprises these main tasks: managing the database, generating meaningful responses, providing a user interface, and forwarding slightly edited information from the NINA API. Users can proactively request the warning status of locations or set subscriptions to be automatically alerted by the bot (see Figure 3). The bot guides users through multi-step processes, such as creating subscriptions, by providing appropriate prompts. The bot outputs text messages in the Telegram chat. The bot’s start is managed through the bot_runner, which creates three threads. In the first, the subscription mechanism checks for new warnings every 2 min. In the second thread, the receiver waits for user input in the chat and then calls the appropriate methods in the controller. In the third thread, the warning_handler runs. It processes all active warnings upon the initial start of the bot and then scans for new warnings every 2 min by default. The controller then accesses various other modules, such as place_converter, nina_service, data_service, text_templates and sender. The sender then sends the chat message to the user. In the place_converter, suggestions for requested cities are generated. The nina_service serves as the interface to the NINA API, and the data_service represents the interface with our database. The appropriate text outputs are created by text_templates. For the information architecture, the system employs a database consisting of three JSON files. The first file contains personal information, including the chat ID from Telegram, completed subscriptions, locations saved as favourites, any muted subscriptions and default warning level. The second JSON file stores the postal codes for which active warnings apply, while the third saves which alerts have already been sent to a specific chat ID to prevent multiple warnings.

Figure 3:

Model of the bot’s functional logic.

4.2 Conversational agent

Figure 4 describes the expected user inputs and resulting states based on the results of the workshop. It shows that warnings can be queried by typing the name or postal code of a location or submitting the GPS location. The query can be turned into a subscription for the queried location, or it can be saved as a favourite for future quick access to the location’s warning status. In addition, subscriptions can also be made for specific or all warning types, and all or only severe warnings. The subscriptions can be managed or muted. In addition, data can be managed and deleted and information about the bot can be found. Finally, emergency tips can be downloaded.

Figure 4:

Model of guardrails used to suggest further actions, with button inputs expected for navigation, and text, GPS or postal code for locations.

4.3 Graphical user interface

While offering wide usage and many useful features, some design options within Telegram are limited. The messages have a fixed design, which can be personalised in the app’s settings. The notification sounds can be set individually within the app and specified for different chats, including by uploading ringtones. Contrary to the requirements of warning apps, the latest messages appear at the bottom of the application and not at the top. Each message is automatically timestamped, and the day is also shown. Regarding the text, some options are available to format the text and highlight keywords via the API. Colouring the text is not possible, but messages can be highlighted by using visualisations such as coloured images or emojis. The official icons of the NINA API ⁵³ are used to represent the event code. The warning message contains the title and a description of the warning, including the affected areas. If available, the alert also includes the date and time, severity, and version. At the end of the message, there is a link for more information that leads to the source of the message on the website of the German Federal Office for Civil Protection. When the warnings are transmitted by another agency, such as the weather service, no link is available. After the final round of evaluation and implementation of resulting changes, the bot offered the main features depicted in Figure 5.

Figure 5:

Screenshots of the design and key features of the warning bot, including pulling the warning status for locations and receiving push messages with warnings for saved locations: (a) starting the bot; (b + c) requesting and receiving the warning status for a city or GPS location, (d) saving the city or GPS location into favourites for frequent future warning pulls, (e) setting up locations for immediate warning updates as push messages.

5.1 Sample

The sample of the first evaluation study was based on the researchers’ personal network, with an attempt to recruit a diverse group. Participants’ age, gender, and their affinity for technology interaction (assessed via the ATI, 9 items measured on a 6-point Likert scale, ⁵⁹ German version ⁶⁰ ) are presented in Table 1. It shows that sample 1 represents a large range of age groups and levels of affinity for technology. In the second sample, possibly due to it being an online survey, the sample was younger – with no one older than 39 years – and the median ATI was higher and less varied. While in sample 1, the average ATI score is somewhat smaller than the mean of M = 3.61 found in the German population, that of sample 2 is somewhat higher. ⁵⁹

5.2 Measurement of utility and perceived usefulness

To test the success rate of users performing specific functions with the bot, in both evaluations, users were asked to perform a set of tasks (see Table 2). The selected tasks represented the main functions of the bot and expected challenges in the set-up process. They ensured that users had experienced the available functions before evaluating usability and usefulness. In addition, the tasks can reveal challenges related to specific interactions.

Several approaches exist for measuring whether the artifact is meeting users’ needs and offers a positive user experience. The most commonly used aspect of user experience is usability, defined as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use 61], p. 4. The most commonly used standardised measurement of usability in research and practice is the 10-item System Usability Scale (SUS). ^{62 , 63 , 64} By exploring usability, we can judge whether bot-driven personalisation within messenger channels, despite the existing UI limitations, can be designed in a way that allows for a positive set-up experience.

However, research indicates that usability explains only 39 % of users’ recommendations of a system, ⁶⁵ implying that additional aspects are important. The Technology Acceptance Model (TAM) suggests perceived usefulness as the second main factor. ⁶⁶ It is necessary to account for as it describes whether an artifact fulfils a need related to users’ contexts and environments. ^{67 , 68} In this study, perceived usefulness is measured through elements of the User Engagement Scale (UES), ⁶⁹ which consists of the dimensions reward, perceived usability, aesthetic appeal, and focused attention. ⁷⁰ The scale offers a good addition to the SUS by focusing on aspects beyond usability. ⁷⁰ For example, reward includes the perception that the experience was worthwhile, or the willingness to recommend an application, which is related to perceived usefulness (see Q25 in the Appendix B). Since user experience has some relevance in safety-critical systems but may be less important than usability, ⁷¹ we put usefulness and usability at the centre of our inquiry, including more items on perceived usability (4 items) and reward factors (5 items) and fewer covering aesthetic appeal (3 items) and focused attention (2 items). In addition, we included open questions concerning the reasoning for (not) intending to use the system, as well as perceived advantages and disadvantages compared with other mobile warning channels. Through open questions, we also assessed participants’ feedback about tasks and the artifact generally, as well as their reasoning for their adoption intention and perceived advantages and disadvantages compared to other warning channels. Two quality control questions were added to ensure that the participants were really completing the tasks. The full questions and tasks can be found in the Appendices A and B.

5.3 Data analysis and results

We followed the standard for measuring ATI by offering a 6-point scale, all other scales are 5-point Likert scales. In addition, we used thematic analysis to analyse the qualitative answers ⁷² and deduce any challenges related to handling the bot, its structure, visual interface and conversational interface. To analyse the SUS, following common practice, the negatively framed items are inverted, and the data is normalised to represent a scale from 0 to 100 for the whole score consisting of 10 questions, as this allows for a better comparison of the results with other systems. ⁶³ The SUS results are analysed with non-parametric tests, as the QQ-plot indicates a violation of the assumption of normal distribution, which is corroborated by the positive skewness of the histogram. Despite the Shapiro-Wilk test resulting in a significance of p < 0.01, we decided to use non-parametric tests due to the relatively small sample size, which limits the robustness of analyses against non-normally distributed data.

The Likert-scale data is thus analysed using Kendall’s tau-b coefficient, which is robust to skewed data and suitable for analysing ordinal data on 5-point Likert scales, as it accounts for tied ranks, which are common in such data, and provides a measure of association that aligns with the ordinal nature of Likert responses. ⁷³ The qualitative answers were coded independently by two researchers through thematic analysis, ⁷² with the coding scheme developed inductively from the text.

5.3.2 Online survey evaluation

The survey confirmed that all tasks are rather or very easy to accomplish (M = 4.32). The first task of finding and starting the bot was perceived as the most difficult (M = 4.08). Only one person failed to retrieve the warning status for a given and current location. The mean usability rating, again using the 10-point scale, is 76.88 (SD = 17.58), indicating good to excellent usability. ⁶³ Figure 6 depicts the evaluation of each item on a 5-point scale for easier comparison with the other evaluations. Aspects that received a particularly positive rating concerned being able to use the system right away and without needing expertise or training. Due to the conversational design, which results in previous inputs becoming less visible, we introduced an additional item asking about the transparency of the system’s status, resulting in a relatively good rating (M = 3.88, SD = 1.10).

Figure 6:

Mean (orange, middle line), median (blue, outer line) and standard deviation (black, inner line) of each SUS item’s 5-point scale rating, *inverted items.

Regarding privacy, users perceived that the bot’s handling of private information seemed trustworthy (M = 3.69, SD = 1.30), that they did not have qualms using the bot with regard to data security (M = 3.73, SD = 1.31) or privacy (M = 3.73, SD = 1.06). The download of the official prevention brochure as crisis prevention advice was perceived as helpful by all but two participants (M = 3.88, SD = 1.41). Most users appreciated the offline availability, although they would have preferred a more concise summary (the used one has seven pages). The new design iteration included emojis to denote the warning topics and severity. These were largely seen as helpful (M = 3.88, SD = 1.08). Asking about improvements, 10 users made comments which related to small design improvements (such as in the way the names of locations were presented) and to a technical issue that resulted in longer loading times (which was fixed in the course of the study).

Regarding the adoption intention, 20 % fully agreed that they would like to use the bot frequently and almost half rather or fully agreed. If the bot were available on users’ favourite messenger app, 20 % more planned on using the bot to make subscriptions. Fewer people would use the bot to query the warning status of specific locations. Finally, another important indicator of the overall usefulness of a system is the likelihood of recommending a system to friends or family. The results are depicted in Table 3 and show that almost half of the participants would recommend the bot to friends and family.

Table 3:

Usage and recommendation intentions, rated on a scale from 1 (fully disagree) to 5 (fully agree).

Intention to	M	SD	Fully agree	Agree or fully agree
…use frequently	3.25	1.22	20 %	44 %
…use on favourite messenger
…for subscriptions	3.83	1.37	40 %	68 %
…for inquiries	3.25	1.15	12 %	44 %
…recommend system	3.50	1.10	20 %	48 %

In order to explain these preferences, we correlated the intentions to use the system with the individual technology affinity, the usability rating and the dimensions of the user engagement rating through a Kendall tau-b test. The results of the correlation analysis are depicted in Table 4. It shows that the intention to use the bot (item 1 of the SUS) correlates with the ATI, SUS and the reward dimension of the user engagement scale. It also correlates with the intention to use the bot in the future if available on users’ favourite messenger app.

Table 4:

Kendall tau-b correlation coefficients between usage intentions and evaluations.

Intention to	…use for subscriptions	…use for inquiries	…recommend	ATI	SUS	UES	UES RW
…use frequently	0.366*	0.359*	0.233	0.305*	0.356*	0.131	0.404*
…use on favourite messenger
…for subscriptions	1	0.285	0.412*	0.259	0.297	0.351*	0.470**
…for inquiries		1	0.139	0.368*	0.116	0.236	0.251
…recommend			1	0.400*	0.510**	0.664**	0.724**

1. ATI = affinity for technology interaction, SUS = system usability scale, UES = user engagement scale, UES RW = reward dimension of the UES. ^* p < 0.05; ^** p < 0.01.

People who would be willing to use the bot for its core function, warning subscriptions, rate it high in the reward dimension and would recommend the bot. People who would use the bot to inquire about the warning status, a secondary function introduced mainly to familiarise users with the bot, does not correlate with the perceived usefulness or the intention to recommend the bot. However, it correlated with technology affinity, further stressing that this is a feature for those who like to engage with technology. Affinity for technology, usability, and all dimensions of the user engagement scale (UES Perceived Usability τ _b = 0.410, p = 0.005, UES Aesthetic Appeal τ _b = 0.531, p < 0.001, UES Focused Attention τ _b = 0.615, p < 0.001) all affect the intention to recommend the system.

Looking at users’ reasoning, those planning to use the bot in the future mentioned the ease of use and not requiring the download of an additional app. People who do not plan on using the bot mentioned not needing a mobile phone to be made aware of severe crises and rather using different sources, such as browser searches, indicating a perceived low need to be informed in a timely manner, or a general disinterest in crisis warnings, e.g.: “I think that more serious matters would somehow reach me, and I wouldn’t need a phone for that. I have never thought about it”. Another group who are less interested are those people who are already using a warning app, who described little added value and some disadvantages regarding the user interface, particularly the display of the preparedness information.

Asked about the advantages and disadvantages of the bot compared to a warning app, users mentioned specifically the flexibility offered by the bot and not requiring another app compared to warning apps. One participant stated: “If I’m honest, I haven’t used any of the apps yet because I didn’t want to install another app and familiarise myself with it. Lack of storage space was also a thing a while ago…. A Telegram bot is more useful for me than another app” (P24). However, other users mentioned that the bot required a messenger app, particularly Telegram. Another perceived disadvantage is the lack of flexibility with regard to the user interface design in the messenger bot. While the handling of the bot received positive evaluations and was perceived as intuitive, one user mentioned that it might pose a challenge for older people: “[…] if the buttons are gone at the bottom, you have to press on a quite small icon to call them up again” (P16). Another disadvantage of the bot compared to apps or CB is that the latter require less or no set-up effort, but they can operate simply based on the live location, which is not the case for the bot (see Table 5 for the full list of themes).

Table 5:

Users’ reported advantages and disadvantages (n) of the warning bot compared to other warning channels, N = 20.

	Advantages	Disadvantages
App requirement	No additional app needed (6)	Requires Telegram app (8)
Interface	Easy to use and to learn (4) Topic engagement through interactive design (1)	More difficult set-up compared with one-click GPS-based warnings in warning apps (2) – Less attractive and flexible interface (2) – Better presentation of prevention information (1) Display of maps (1)
Perceived usefulness	Subscriptions for different locations (4) Warning status queries (6)	Disinterest in warnings (4) Use of other channels, such as Internet search (4)
Resilience	–	Less resilient against Internet outages than CB (2)
		No warnings when in silent mode (1)
Privacy	Better data protection (1)	–

6.1 Warning “botplications” requirements

Based on the requirements for warning apps, the study has shown that messaging channels offer some, if not all, relevant functions that are central to delivering timely, personalised warnings.

Bonaretti et al. ²⁸ analyse warning apps with regard to different layers. Looking at these layers (see Table 6), we find that warning apps and warning messaging channels are identical in that they run on the same physical layer on smartphones. However, if an IM app has already been installed, users do not need to invest in additional storage space, and messages typically require a small amount of Internet data. In addition, although not widely available, messenger apps can use peer-to-peer communication, making them, to some extent, resilient to Internet outages. On the deep layer, concerning the alerting object, the bot can send the same alerts as the app, due to the availability of an API of the official alerting app. The crucial difference is at the software level, where one uses a warning app and the other an IM app, whereby the choice of messaging app should depend on the number of users. The surface layer, too, is highly dependent on the software used. Generally, options for sound and visuals are limited by the IM apps’ functions and design.

The software layer also influences the provided warning and crisis functions. Regarding the functions, an important difference is that currently, location tracking is not possible within the chosen IM app. Therefore, warnings cannot be made based on the current location of users. However, users can allow short-term access to the location and the bot can use this information to suggest a warning subscription for the respective location. Due to this process, the set-up takes longer than in some warning apps, where the location-based warning is enabled as soon as users grant access to their location information. In addition, the bot cannot send warnings based on the current location, only based on predefined locations of interest. This leads to the bot requiring more set-up than a well-implemented warning app, which in its initial start can ask for the permission to use GPS, enabling warnings with only one click. Another difference is that it is currently not possible to show an interactive map within the IM app. Therefore, it is more difficult to show users that the app is active and to give them some insights into the warning status. We, therefore, implemented a feature that allows users to query the warning status of locations, which offers an immediate way for users to interact with the bot and verify its functionality. This feature is not common in warning apps but may provide additional benefit for people who enjoy engaging with technology – especially as “site-seeing”, i.e. digitally visiting locations affected by crises, is a common online activity. ⁷⁴ Another important function is offering additional advice. However, the conversational interface appears inadequate for displaying longer texts. Therefore, we opted to offer advice through a file that can be downloaded. While users prefer to have all information in the warning app rather than through external links, ¹³ this allows saving the file on the device, making it available offline.

With regard to the conversational design, the implementation process has shown that many challenges arise from the onboarding process. This is in line with research on voice-based interfaces for visually impaired people, which finds that discoverability of the available functions is a challenge. ⁷⁵ For onboarding, we have chosen to let users query locations’ warning status, as this provides an immediate insight into the capabilities of the bot and the type of output to be expected. In this way, users can experience the bot’s warning function before setting up subscriptions. This interaction is used to invite users to set up warning subscriptions for the queried location, which the study confirms to be the key function of the bot. For the set-up process, the chosen settings need to be transparent. In addition, buttons can serve as guardrails to give guidance about the available options. The evaluation has shown that efficiency is a key element with the warning bot. Efficiency was improved by displaying colours and emojis to give immediate visual cues, reducing time spent reading. In addition, texts were shortened substantially. Providing sufficient information to navigate the bot while keeping text to a minimum is another challenge that is intensified in the messaging platform. Users expressly stated that the bot was easy to use. This indicates that the recommendation of buttons to guide the conversation ²⁵ can be confirmed in this context.

While many studies indicate that human likeness increases positive evaluations of bots, this was not mentioned by users in our study. In a study of a municipal service bot, the relevance of human likeness was regarded very differently between users. This may be due to the character of the bot. While the municipal service navigation bot used natural language to support users, the bot developed here is closely related to an application. This supports the differentiation of botplication from other types of conversational bots, with botplications aiming for simplicity and effectiveness, whereas human likeness is irrelevant or secondary. ²⁵ The analysis, discussion, and evaluation of chatbots’ maturity is mainly based on the use of natural language processing. A typology by ⁷⁶ differentiated between first level bots that facilitate service triage (matching user request with rules and linked data sources); second level bots for service information gathering and analysis that access data from multiple sources to personalise the service; and third level bots for service negotiation, which entail predictive analytics and neural networks to evaluate customer circumstances based on various sources such as user profiles and service profiles and providing multiple personalised service options for the customer in their natural language. However, the task-oriented botplication described in this study does not fit into the typology. For example, in the typology, accessing user profiles or continuing service interaction occurs along with NLP components such as sentiment analysis. The bot used here shows that personalised service delivery does not require large language processing capabilities. Indeed, the design recommendations for CUIs within chatbots suggest a limited use of text and an extensive use of graphical UI elements to steer the conversation in an efficient manner. ²⁵

6.2 The role of usability and perceived usefulness for intended future use

Looking at the utility of the warning bot, we find that 68 % of study participants stated that they would use the system if it was provided through their favourite IM app. In addition, almost half stated that they would recommend the bot.

The reward dimension of the user engagement scale describes aspects such as whether the time spent with the system was worthwhile and emerged as the most significant, impacting also the intentions to use the bot generally and specifically for warning subscriptions. The intention to recommend the warning bot only correlates with the intention to use it for warning subscriptions in a preferred IM app, indicating that this aspect is perceived as the most useful feature for other people. Thus, this underlines the added value of bots when coupled with proactive notifications, as this is not achieved through other sources, such as Internet searches.

The study suggests that usability, in the context of warning channels, is less relevant for predicting the intention to adopt a system. Instead, the intention to use the bot is more closely related to the utility of a system – aspects that are described in the reward dimension of the user engagement scale. ⁷⁰ This supports the value of the user engagement scale and its different dimensions, which has also proven useful in the health and news domains. ^{77 , 78 , 79} Therefore, future representative user studies that evaluate chatbot designs or compare them to alternative warning channels should include such aspects in addition to usability aspects. This result is in line with previous findings on e-government non-adoption. One study found that users preferred to remain with their status quo, interacting with humans: “The less need citizens feel for public services or the more convenient they find conventional ways of contacting administrations, the less they are inclined to use public e-services”. ⁸⁰ Adapted to the use of warning apps, we also find that many respondents, even when indicating that the tool is useful and has high usability, state that they would continue using public media for warnings. Similar to another qualitative study on e-government, we find that a perceived lack of need or infrequent usage is one of the main barriers. ⁸¹ In the open statements, it also became evident that, at least for some of the participants, it had not become clear that the subscriptions would automatically alert them in case of a warning. As a result, two participants stated that they might not be querying information just as a crisis was impending, likely due to the unfamiliarity with bot-based subscriptions. Still, more attention should be paid to ensuring that users understand the scope of the offered function, e.g. through a short introductory video. ^{82 , 83}

In this study, we sought to investigate the usability of a warning messaging bot as an alternative warning channel to warning apps. Therefore, we attempted to emulate features that are offered by the official German warning apps. However, warning apps could conceivably offer additional safety and security functions or include more accessibility features, such as haptic instead of auditory signals or video emergency calls for people with hearing impairments. The structure of the conversational interface introduced in this study can be used to design a fully voice-based process for people with impaired vision, allowing them to query information through natural speech or to set up automatic warning subscriptions. A study shows that conversational assistants provide a range of useful functions for blind people. ⁷⁵ With added natural language processing functions, the developed conversational design could be used to make personalised warnings accessible to further user groups, who tend to prefer to be able to use mainstream channels rather than specifically designed ones. ⁸⁴ However, moving towards accessibility would require extensive user studies and changes in the way that information and alert signals are presented. ⁸⁵

While the additional steps needed to set up a subscription may be perceived as a disadvantage, one user reported that the process increased engagement with crisis management, possibly by confronting users with the available selection of warning types. This finding may echo an increased engagement with municipal services that was achieved through the use of a chatbot. ⁵¹

6.3 Limitations and future work

The paper has several limitations regarding the interactions that were analysed and the design of the warning bot. Firstly, due to ethical concerns related to ensuring the reliability of the bot “in the wild”, the evaluation did not encompass real-world use, restricting the measurement to participants’ experiences in the set-up and their intentions for future use. Future work should create an experimental setting that allows the usability of the alerts to be tested. One question addressed the advantages of a messenger bot compared to other warning channels. This question was answered by both current users and non-users of warning apps. Future work should explore the affordances of a messenger warning bot with the affordances of warning apps and cell broadcast, similar to a study which has analysed affordances of news consumption via messenger bots. ⁵⁴ In addition, the evaluation is biased towards younger users, who appear to be more willing to use the bot in the future and who are likely to be experienced messenger app users. ⁸⁶

With regard to using messenger apps as the containers of bots, we find that some messenger apps offer significant features for open source, low-cost implementations. However, a disadvantage is that certain aspects cannot be adapted. In this prototype, a significant usability limitation is that the text input fields cannot be hidden or disabled. This suggests to users that text inputs may be required. In addition, activation of the text input field can lead to the disappearing of the buttons, requiring some familiarity to recuperate the buttons. In addition, other research has also explored the relevance of humanness of bots with inconclusive results and suggested a dependence on the task and domain ⁸⁷ . Inspired by the usability requirement of simplicity, we did not implement any humanness in the bot in the warning context. Future research could explore the relevance of this attribute through a comparative experiment.

Regarding the conversational design of the bot, due to the importance of achieving correct results in a safety-critical domain, the current prototype does not use natural language processing (NLP). While this does justice to the requirement of minimal input, especially for mobile use, and reduces errors, NLP could enable users to interact more freely with the bot like ⁸⁸ have put forth. However, our system’s backend, functions and conversational design structure could be used for future accessibility studies, where the messenger or smart speakers may be used to set up the warning bot preferences and where a smart home environment may be used for visual, haptic or auditory signals adapted to users’ accessibility needs. This could enable users with visual and hearing impairments to set up and receive noticeable alerts. A previous study has already suggested different signals and intensities to communicate varying warning levels. ⁸⁹ The use of natural language for output would also allow conveying more extensive information regarding crisis preparedness. Using NLP, the bot could answer specific questions related to crisis preparedness upon request. Previous research suggests that such information should include statistical information that stresses the benefits of crisis preparedness. ⁹⁰