Cyber hate awareness: information types and technologies relevant to the law enforcement and reporting center domain

2.1 Defining online hate speech

HS is a contested concept, ³⁴ which is rarely systematically conceptualized despite extensive research on its causes and harms as well as adequate responses. ³⁵ In academia, HS is often defined in relation to the respective research motivation. ³⁵ For computer science, this means that research datasets are often underpinned by differing conceptualizations of it. ³⁶ This creates challenges for the evaluation and generalizability of detection models trained on them. ^{37 , 38} In law, definitions and thus the sanctionability of different types of expressions as prohibited forms of HS vary by national jurisdiction. ³⁹ Finally, online platforms also follow varying definitions that inform their content moderation practice. ⁴⁰

Appropriately defining HS can be challenging. First, the term is often used as an umbrella term for numerous negative or aggressive statements, including harsh criticism and simple insults. ⁴¹ Second, it is often insufficiently distinguished from broader concepts, e.g., abusive or offensive language, as well as specific forms of group-related hostility, e.g., racism. ⁴² Third, even though there is some agreement that it targets groups or individuals based on their group membership or identity, ^{35 , 37} there is disagreement about whether HS can only target pre-determined ‘protected’, i.e., minority, groups. ⁴¹ The group-relatedness of HS becomes apparent when it is situated within the taxonomy of harmful online content of Banko et al. (see Figure 1). ³⁶ With the category Hate and Harassment they cover content that torments, humiliates, undermines, or frightens victims. While they recognize that content types are not mutually exclusive, they differentiate Doxing, Insult, Sexual Aggression, Threat of Violence, Identity Attack, and Identity Misrepresentation. Since HS can involve not only overt group-related attacks but also derogatory stereotypes, ²⁰ it covers both of the latter two content types. Fourth, HS definitions often focus on one of three aspects: The intention of the person expressing it (1), the content of the speech and its form (2), or the effect on the target (3). ^{41 , 42}

Figure 1:

HS within the taxonomy of harmful online content by Banko et al. ³⁶ Adaptation of the taxonomy by the authors.

Developing a novel HS definition goes beyond the scope of our work. However, based on the narrative review of conceptual literature above and given our domain of interest, we regard the following aspects as critical for the selection of an adequate definition:

1. HS should be defined with reference to its content. This seems suitable for LEAs and RCs since they also identify it on this basis. Articulators’ intentions and effects on victims may be difficult to ascertain. ⁴²

2. It should also cover subtle forms of HS that work, e.g., with negative stereotypes, metaphors, irony, or humor, ^{43 , 44} as these can also justify or reinforce discrimination against groups. ²⁰

3. It should not narrow down targeted groups, as new targets can emerge over time. ²⁰ This also seems important regarding the work of LEAs and RCs.

4. Since HS is not defined in German criminal law, ⁴⁵ LEAs and RCs may also be concerned with cases below the threshold of criminal liability.

The HS definition by Fortuna and Nunes [20], p.5] corresponds to these considerations but needs to be adapted to suit the law enforcement and RC domain. First, it should be narrowed down to online content. Second, it should be defined irrespective of data modality, as not only textual but also visual, audio, and audio-visual material can constitute HS. Third, reference to exemplary targeted group characteristics should be omitted to ensure brevity. Accordingly, we adopt the following definition in this work:

Online HS is internet content, irrespective of data modality, that attacks or diminishes, that incites violence or hate against groups, based on specific characteristics. It can also be expressed in subtle or humorous ways.

2.2 Situational awareness

In order to prevent and respond to HS effectively, LEAs and RCs need to establish situational awareness and undertake informed decisions. Stanton et al. distinguish three major conceptions of situational awareness in research. ⁴⁶ These focus on the interaction between individuals and the environment, ⁴⁷ individuals’ cognitive sub-systems, ⁴⁸ or individuals’ perception and understanding of the environment with progressive degrees of insight. ⁴⁹ As we want to investigate practice-relevant information types within the environment of LEA and RC staff engaging with HS, the latter conception with its three awareness levels is particularly suitable as a conceptual framework. Specifically, Endsley [9], p.36] differentiates “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future”. In an effort to adopt the concept for cyberspace, the term cyber situational awareness was established. It refers to the state of knowledge of actors that enables them to perceive the relevant elements in the cyber environment (e.g., current situation, impact of the attack, adversarial behavior, as well as quality and trustworthiness of available information) within a particular volume of time and space, to comprehend their meaning, and to project their status in the near future. ^{14 , 15 , 50}

Although it is conceptualized as a subset of situational awareness, it cannot be considered in isolation because events in cyberspace often impact the physical world, e.g., financially, socially, or politically. ^{51 , 52 , 53} Its growing importance within public administration is reflected in numerous national cybersecurity strategies. ¹⁴ Especially professionals in formalized security organizations like computer emergency response teams (CERTs), whose work is particularly challenging due to the necessity of coordination and information exchange, benefit from continuously improving their situational awareness. ⁵⁴ The term cyber situational awareness is often related to knowledge about occurrences in one’s own network, but CERTs have to look way further “to gain a common operational picture of the threat environment in which the constituency is operating” [55], p.17]. Due to the growing number of attacks and security breaches, the volume and variety of potentially relevant data and data sources are increasing, and thus, maintaining adequate cyber situational awareness is increasingly dependent on the properties and capacities of the tools used. ^{56 , 57}

However, due to its focus on cybersecurity threats, the notion of cyber situational awareness does not account for the specifics of managing cases of hateful content, such as in the HS response activities of LEAs and RCs. Still, some works examine situational awareness of LEAs, including a mixed-method study for identifying themes of police-specific situational awareness ⁵⁸ and an interview study on its role in everyday and high-risk tactical police interventions. ⁵⁹ Moreover, to enhance situational awareness, researchers propose an ecosystem for the combined use of augmented reality and AI in patrolling and tactical scenarios, ⁶⁰ an interactive dashboard for fire and police departments, ¹³ and a mobile visual analytics approach for criminal, traffic, and civil incidents. ⁶¹ Yet, none of these conceptual, empirical, and technical studies focus on the specifics of situational awareness in HS response. Finally, the HS classification scheme for German LEAs and RCs of Bäumler et al. that differentiates HS by targeted group, the conveyance of an immediate threat, and criminal relevance (cf. Figure 2), ³⁰ is of interest to our work but does not adopt a situational awareness perspective.

Figure 2:

HS classification scheme developed in Bäumler et al. ³⁰ The grey labels characterize the classification tasks. PMK = politically motivated crime.

2.3 Hate speech response technologies

Computer science researches various technologies that may be leveraged to assist LEAs and RCs in online HS response. Concerning HS gathering, there are three major research directions. First, there is user-centered research in HCI on tools that support victims or witnesses in documenting and reporting hateful content. Whereas LEAs and RCs usually offer e-mail contacts and self-administered web portals for reporting, ^{3 , 26} one center provides a mobile app whose development, however, was not accompanied by research. ⁶² Yet, there is user-centered research on tools to support targets of gender-based harassment in evidence documentation and report generation. ^{16 , 17}

Second, detection algorithms can identify HS content as part of proactive monitoring. There exists extensive research in AI, machine learning (ML), and natural language processing (NLP) on its binary detection. ^{63 , 64 , 65} While most approaches focus on textual HS, ^{18 , 20} there is some research on its detection in visual ^{21 , 66} or audiovisual data. ¹⁹ While so far most HS detection models are implemented as black boxes, i.e., without disclosing the reasoning behind algorithmic decisions, ⁶⁷ some works investigate how interface design can improve algorithmic transparency and explainability for content moderators, ^{67 , 68} LEA staff, ²⁷ or internet users. ⁶⁹

Third, beyond content detection and reporting tools, open-source intelligence (OSINT) can be leveraged for gathering information on HS. OSINT refers to “the collection, analysis, and use of data from open sources for intelligence purposes” [70], p.677]. Among other use cases, OSINT methodologies are crucial in monitoring alternative social media and understanding the spread of hateful content across platforms and channels. ⁷¹ In context of HS, integrating them with advanced computational techniques, e.g., NLP, ML, or graph neural networks, can enable the recognition of perpetrator, target, and topic relationships, ⁷² the identification of malicious users, ⁷³ and the tracking of hate-related keywords. ⁷⁴

Concerning analysis, there is research on sub-type classification and visual analytics. For content moderators, LEAs, and RCs, the type, target, severity, or legality of HS may be relevant. Thus, granular HS classification attracts increasing attention. AI and NLP research often addresses this as a multi-class problem. ⁶⁵ However, this assumes mutually exclusive labels. Individual content may include varying hostility patterns and target different group affiliations simultaneously. ^{22 , 75} Moreover, in some jurisdictions, several criminal norms may apply to HS content simultaneously. ³⁰ Multi-label HS classification addresses this issue but still receives limited attention. ⁶³ Models are available for hate types, ^{23 , 24 , 76} targets, ^{22 , 76 , 77} and hate patterns, ^{78 , 79} while for criminal norms, only one training dataset exists. ²⁹

Beyond that, there is research on the visual analysis of online HS data. Some works investigate the visualization of temporal and geographic HS diffusion with maps. ^{80 , 81} Others propose dashboards to examine its dissemination with social network analysis, e.g., by visualizing trends, illustrating associations with real-world events, or highlighting perpetrator relationships. ^{12 , 25}

Most of these works do not involve users. At the same time, those with an empirical grounding are mostly unrelated to the law enforcement or RC domain, instead targeting internet users or content moderators. Thus, a research gap exists regarding the identification and systematization of information types relevant to this domain that can inform the design of HS response technologies.

3.1 Data collection: expert interviews

Following a targeted convenience sampling approach, we contacted German LEAs and RCs accepting reports on online HS from Germany. Twelve individuals from eleven LEAs and 17 individuals from eleven RCs agreed to participate, of which 14 were male and 15 were female. We use identifiers to refer to the interviews throughout this paper (L1–12; R1–17). From LEAs, we interviewed officials from five state (L1–5) and three federal (L6–8) police authorities, as well as public prosecutors from four attorney general’s offices (L9–12). From RCs, we interviewed staff of three centers focusing on online HS (R1–7), three centers accepting reports on different types of illegal content, including criminally relevant HS (R8–11), and centers that handle reports of antisemitic (R12–13), antiziganist (R14), or queer-hostile incidents (R15) and digital violence (R16–17). We involved organizations with varying scope, as they all engage with at least some types of online HS. Table 1 provides detailed information on the interviewees and organizations. After obtaining their informed consent, we held all interviews digitally between March 2023 and August 2024. On average, they lasted 60 min. We audio-recorded and held them, with one English exception (R7), in German. Participation was not compensated.

3.2 Data analysis: thematic analysis

After transcribing and anonymizing the interviews, we performed a thematic analysis following Braun and Clarke ³¹ with the software MAXQDA 24 to systematize the empirical data. Given the flexibility of the approach and the exploratory character of our research, we considered it well-suited for elaborating information types relevant to acquiring situational awareness of HS. As there is no previous work on situational awareness in this domain, the themes guiding our analysis were identified inductively during the exploration and coding of the empirical material without a pre-defined coding frame. For inductive theme discovery, we followed Mayring’s processual approach for category formation. ⁸⁵ Based on our research question, we defined our selection criterion for relevant content and categories’ level of abstraction. Then, one author analyzed the material in an initial iteration, coding all text that matches the selection criterion. He formulated initial thematic categories for concrete relevant information types during this process. If the selected text matched an already established category, it was assigned to that category. Otherwise, a new category was created. After coding a quarter of the interviews, the authors reviewed the categories and decided that the level of abstraction and selection criterion were adequate. Then, the rest of the material was coded. This resulted in a coding scheme that the authors reviewed a second time. During this step, the categories and their definitions were refined. In addition, meta-themes that reflect not concrete information, e.g., perpetrators’ identities, but high-level awareness sub-types, e.g., perpetrator awareness, were inductively formed through thematic clustering. Two researchers then used the revised coding scheme to code all material in a second iteration. They discussed difficulties and borderline cases to enhance consistency and coded six interviews parallely to assess coding quality (∼20 %; L2, L4, L8, L10–12). MAXQDA was used to check intercoder agreement. The resulting kappa coefficient of 0.78 following Brennan and Prediger ⁸⁶ indicates substantial agreement. ^{87 , [1]} The final coding scheme consisted of six meta-themes and 23 categories and can be found in the Appendix, whereas Table 2 provides an overview of the categories and their occurrence in the individual interviews. In the subsequent section, we structure our results by the identified meta-themes and use core statements for illustration.

4.1 Victim awareness

Victim awareness refers to the perception of information relating to those targeted by HS and the comprehension of its significance in the situational context. High awareness may further permit projections of future developments regarding hate targets. Victim-related information can further be significant for establishing context or evidence awareness and may be of value in assessing the legal and threat dimension of hate. Four information types can primarily be assigned to this sub-type.

As HS attacks or denigrates targets based on their group membership or identity, information on hate types, i.e., which abstract group affiliations are targeted by it, has significance for several LEAs and RCs. The same applies to the specific targeted groups. These differ from hate types, as they do not refer to the kind of group affiliation, e.g., religion, but to the specific group attacked, e.g., Muslims. Individual HS may be attributed to several hate types and target several groups at once, as many victims are attacked in connection to more than one group-related attribute:

That is often the case, especially when it concerns women, that there is not only one angle. So it is often the case that one says: Based on skin color and from a gender perspective, women are an affected group (R1).

Particularly, when slang or veiled group-related slurs are used, it may be necessary to consider the individual context when establishing which hate types and targeted groups are present. Both information types have minor relevance for prosecution but are an invaluable component of situational pictures on the dissemination of hateful online content within specific jurisdictions or online spaces. Their cross-case analysis allows to comprehend the HS exposure of different social groups over time and an anticipation of upcoming trends. In addition, some RCs leverage the information to recommend tailored, e.g., group or hate type specific, external counseling services to victims.

Information on victims’ specific identity, contact details, and location is, by contrast, of greater significance for the day-to-day operation of LEAs and RCs, especially for prosecution. However, this information is only available for content that is targeted at concrete persons. It is particularly important for instances of complaint offenses (Antragsdelikte), such as defamation or insults (cf. Section 4.5). Here, a criminal complaint must be obtained from affected individuals, which is only possible if their identity and contact details are known:

In these cases, it is simply not possible for the victims to remain anonymous. This means that we actually need the contact details (L12).

In the case of ex-officio offenses (Offizialdelikte), such as incitement, this information may be helpful but is not essential. Accordingly, most RCs allow anonymous reporting. Knowledge of victims’ location, e.g., their place of residence, is furthermore relevant in three regards. First, many organizations use it to assign a case to a regionally responsible police station or public prosecutor’s office if the perpetrator’s place of residence is still unknown (cf. Section 4.2). Second, in some RCs, victims are referred to local counseling services if they know their place of residence. Third, it is relevant for all cross-case situational pictures on the dissemination of online HS that encompass a spatial dimension, e.g., on targeted groups or criminal offenses within a certain jurisdiction. For instance, hate crime hotspots may be identified on that basis:

Of course, it’s nice for us to be able to recognize areas of concentration. Is it a regional hotspot and what is the reason for this? Perhaps it’s due to a perpetrator who is a persistent offender (L2).

4.2 Perpetrator awareness

Perpetrator awareness refers to the perception of information related to perpetrators of HS and the comprehension of its significance in the situational context. Information on this can also be useful for understanding the broader context of hateful content and may constitute evidence. Moreover, it often contributes to comprehending the legal and threat dimension of hate. Five information types can be primarily assigned to this sub-type.

Considering a perpetrator’s online activities can be essential to comprehend the scope and dynamics of HS. Information on a perpetrator’s online behavior, such as the frequency and nature of their posts, affiliations to groups, and interactions with others, provides valuable insights for LEAs. Such information helps to identify patterns, assess the threat level, and determine the appropriate legal response. Repeated HS dissemination amplifies severity and may influence the outcome of criminal proceedings:

Repetition of the actions is an important criterion. Individual incidents can also be serious, but repeated behavior strengthens the case as a hate crime (L6).

Alongside online activities, perpetrator’s social media profile(s) and IP-Address are information types that help LEAs and RCs determine the identity of anonymous perpetrators. For LEAs, this information is crucial for initiating criminal proceedings, assessing potential threats, and developing prevention programs. However, uncovering perpetrator identities in the digital realm can be challenging. Perpetrators frequently use pseudonyms, faked e-mail addresses, virtual private networks (VPNs), and disposable accounts, rendering identification complex:

The challenge lies in determining who is behind the account. That is our everyday work…. Many use VPN tunnels and fake emails that they use for 10 minutes to register an account and then delete (L3).

LEAs employ various methods to overcome identification challenges, including OSINT investigations and filing data requests to platform operators. Some also rely on cross-referencing information from multiple cases to detect patterns and connections. As part of this, they prioritize data that may uniquely identify an account, such as account IDs, account URLs, IP addresses, and timestamps, as those usually remain constant. Especially IP addresses are often a key piece of data for identification, but obtaining and utilizing them can be challenging due to legal and procedural constraints.

Finally, information on the perpetrator’s location, i.e., their place of residence, is of importance for both criminal proceedings and operational purposes in LEAs. As hate speech often transcends regional boundaries, knowing where a perpetrator resides allows assigning HS cases to LEAs with appropriate regional competencies:

Of course, if you have a perpetrator, or if it comes through a reporting office and you can identify a perpetrator, then it goes to the station where the perpetrator is based (L1).

When the perpetrator’s location is unknown, LEAs may resort on the victim’s location to determine the responsible authority. In cases where a physical threat is likely, the location is also relevant for initiating emergency measures, e.g., sending police officers to the perpetrator’s address.

4.3 Context awareness

Context awareness refers to the perception of information related to the social contexts in which HS is disseminated, the comprehension of its situational significance, and the anticipation of developments in this regard. It intersects with victim and perpetrator awareness as information associated with it may contribute to those sub-types and vice versa. Contextual information can be evidence and crucial for determining content’s legality and threat potential. It encompasses four primary information types.

Information regarding the platform or website on which HS content was published is of significance to LEAs and RCs. First, for many RCs and police agencies, knowledge of this is a prerequisite for notifying operators about such content and thus initiating its deletion. For LEAs, this is also important if user data has to be requested from operators to identify anonymous perpetrators. Second, the information is also relevant when generating situational pictures. In light of CHA levels, this relates not only to understanding the current situation by pinpointing temporal trends, but also projecting them into the future. Changes may indicate that hate dissemination is shifting to particular platforms:

Things are often promoted via well-known platforms, but then the users migrate to others and that’s where all the criminal activity actually happens. So it’s always important to recognize these trends and developments early on (L8).

Such insights allow LEAs and RCs the adaption of monitoring activities and the demand-oriented design of prevention and awareness-raising measures.

Several LEAs and RCs emphasize that knowledge on the relation of HS content to specific events, such as criminal incidents or political events, is highly relevant. This can not always be objectively determined. Often an interpretation within the conversational context is required if references are only implied or emerge from other content. Some LEAs and RCs conduct HS monitoring in context of specific events to understand the dynamics of HS dissemination in their vicinity:

We also do some monitoring and check whether you can really pin it down to one trigger, one date, one event and what effect it has. For example, state elections. (L2)

Some also record event-relations of processed cases and present corresponding data to decision-makers. Since knowledge on respective trends may allow a projection of incident numbers, event attribution also informs the projection level of CHA. This can influence resource planning, case prioritization, and the anticipation of potential threats (cf. Section 4.6) necessitating immediate action:

I think an acute situation like this has the most potential for something to happen. In other words, that some kind of action will follow (R4).

For many LEAs and RCs, the relation of HS to broader social or political discourses, e.g., on the war in Ukraine or the Covid-19 pandemic, has comparable relevance. Such links are not always apparent in the content itself, but often stem from its conversational context and may thus require interpretation. Links to discourses are particularly relevant for situational reports on case volumes, as it allows conclusions about the evolution of topics on which HS is disseminated. Such insights are not only valuable for understanding the current situation, but also its projection. In some instances, discourse-related trend data not only serves as a basis for decision-making within the organization itself, but is also provided to other administrative or political decision-makers. It can inform the initiation of measures, such as increased protection of facilities, necessitate upgrading technologies such as AI detection algorithms, affect the allocation of resources, and trigger the initiation of targeted monitoring:

We always have to be up to date with current events. There’s a weekly report, a quick monitoring on current cases. For example, the Ukraine war, the coronavirus pandemic (R2).

Finally, to some LEAs and RCs the extent to which HS content or its creators are connected to anti-constitutional activities, i.e., whether there is an extremism relation, is important. Often, this cannot be determined from the content itself and requires interpretation. Respective information may influence case prioritization and the forwarding of content and profiles to domestic intelligence services, which focus on monitoring and countering extremist efforts. In context of CHA, it is also used to anticipate future extremist activities:

You have to look at this separately from criminal law, because that is often not even about specific offenses, but about activities that can then lead to them. A storming of the parliament or whatever (R1).

4.4 Evidence awareness

Evidence awareness refers to the perception of information that is relevant as evidence during the investigation and prosecution of potentially illegal HS and the comprehension of its significance in the context of the current situation. It intersects with victim, perpetrator, and context awareness, as information pieces from those often constitute evidence, and contributes to legal and threat awareness. Six information types can be primarily assigned to it.

A direct link to specific HS content is required by most RCs and LEAs. This type of information primarily refers to the content’s URL. For certain content types, e.g., longer texts or video and audio files, additional information may also be valuable:

If a video has thousands of comments and we get the URL, it takes us a long time to find the comment. If the video is two hours long and there is a potentially relevant minute, then a time code helps us in addition to the URL (R6)

Direct links are particularly important during evidence documentation, for RCs before a case is forwarded to other organizations and for LEAs in the context of investigations. If they are unavailable, a time-consuming search may be necessary, or it may be impossible to proceed. Furthermore, direct links are also relevant for requesting content deletion from platform operators. Therefore, some RCs require the indication of a URL in their reporting portals.

As HS content may be deleted or hidden before or during its processing by LEAs or RCs, (audio-)visual evidence is required for a successful assessment and subsequent investigations and criminal proceedings. In case of textual or visual content, this usually entails screenshots. In case of audio-visual content, it can also be recorded and saved as a file. Three different types of (audio-)visual evidence are considered particularly relevant. First, evidence documenting the specific HS content. Second, evidence documenting the immediate conversational context, e.g., an initial post or thread. Third, evidence documenting the perpetrator’s profile. As regards screenshot quality, it is emphasized that they should ideally include the URL of the depicted content or profile and a time stamp:

We always need the criminally relevant comment or post. Then the initial post, the context in which it was posted, and then a screenshot of the user’s profile. The screenshots should all contain the URL and preferably the time (L9).

Visual evidence should be documented as early as possible. RCs often offer uploading options for screenshots on their reporting portals. In addition, RCs and LEAs conduct separate evidence preservation for both reported and self-identified content, if possible.

Information on the reporter’s identity and contact details is only available if HS content has been externally reported. Since many RCs also allow anonymous reporting, they and subsequently LEAs often do not have this information. The data can be identical to the identity and contact details of the victim (cf. Section 4.1). During case processing it is helpful but not essential to have a contact option, e.g., by e-mail. Especially if reported HS has already been deleted or is not publicly accessible, evidence and contextual information may be requested:

We have cases were things are reported from closed user groups …and if we say there is not enough for us to evaluate, we ask if there is anything that can be sent afterwards (R1).

If criminal proceedings are initiated, the reporter might also appear as a witness. Furthermore, RCs can only provide feedback if contact information has been provided.

The incident time, i.e., the date and time HS content was published, has significance for LEAs and RCs as it represents the time of offense for criminally investigated cases. In some RCs, it further influences the case processing order. Across cases, the information is relevant for all analyzes, reports, and situational pictures on the dissemination of online HS that cover temporal trends or specific time frames.

The documentation time of HS content, i.e., the concrete date and time evidence was preserved, is instead only relevant for individual investigations and criminal proceedings. It is important to enhance the admissibility of (audio-)visual evidence in court. Ideally, it is incorporated directly into the evidence:

So I would want a piece of evidence. With a good time stamp. Because we have people claiming that their account was hacked. That they weren’t responsible (L7).

The availability of HS content, i.e., whether it was and still is publicly accessible, is relevant in two regards. First, it is important information during evidence documentation. Publicly accessible content can be documented by RCs or LEAs themselves. In the case of unavailable content, it is necessary to contact the reporting party and request evidence. On the other hand, the availability and therefore potential reach of content can be relevant in criminal proceedings, e.g., when determining the sanction.

4.5 Legal awareness

Legal awareness refers to the perception of information concerning the legality of HS, the comprehension of its case-specific and cross-case significance, and the projection of the dissemination of illegal HS into the future. It is obtained by interpreting information from the other CHA sub-types and comprises three main information types.

For almost all RCs and LEAs, the general criminal relevance of online HS constitutes important information. An often preliminary assessment of criminal relevance determines how cases are handled. Most RCs forward potentially criminally relevant cases to responsible LEAs or support victims in filing a criminal complaint. In non-criminal cases, victims are nonetheless often supported, either in content deletion, by providing counseling, or by referral to external counseling services. Public prosecutor’s offices initiate an investigation if there is an initial suspicion of a criminal offense, in the course of which they and supporting police authorities gather evidence, clarify the perpetrator’s identity, and specify criminal norms. If the suspicion is confirmed, they will file charges. Otherwise, the investigation will be discontinued. Determining whether content is criminally relevant requires an interpretation of various information from other CHA sub-types. Accordingly, extensive efforts are usually required to generate this information:

In the best case, I have all this information…. Otherwise, the identity of the suspect must be established, then the statement and the context must be evaluated (L10).

Whereas RCs’ determination of criminal relevance is more preliminary, LEAs undertake a more extensive, rigorous, and evidence-bound assessment, as criminal proceedings may initiated on this basis:

This can sometimes be very difficult, extensive. A statement that the perpetrator made in one minute may require a lengthy examination by the public prosecutor, who may also have to consult case law (L10).

Beyond that, data on criminally relevant HS can also be used to deduce and project trends. This can, e.g., inform resource planning within the organization.

When examining criminal relevance, most LEAs and RCs check the applicability of individual criminal norms. Several German criminal norms are potentially applicable to online HS. ³⁰ While it may be possible to assess candidates for some offenses solely based on content, e.g., public incitement to commit offenses (§ 111 StGB), it is particularly important to consider contextual factors for others. For instance, when assessing cases of defamation (§ 187 StGB) it is crucial to consider the (un-)truthfulness of person-related statements. Generally, several criminal norms can apply to individual HS simultaneously. Information on potentially applicable criminal norms is particularly important for RCs, as different types of offenses are forwarded to separate authorities. The differentiation is also important for LEAs, since in the case of complaint offenses, the victim of HS must file a criminal complaint for further prosecution, which is not required in the case of ex-officio offenses. In the former case, the authority must have knowledge of the victim’s identity and contact details (cf. Section 4.1).

The applicable criminal norms are naturally also relevant for subsequent criminal proceedings. Beyond that, aggregated data can also provide a valuable basis for comprehending and projecting trends:

We are also heavily indicator-based. To manage our resources and identify deficiencies, but also to provide our partners with key figures regarding offenses, which offenses are in focus, and the success of our measures (L8).

Another information that should be considered separately is the attribution of criminally relevant HS to a politically motivated crime (PMK) type. While this information has only relevance for one RC, interviewees from several LEAs stated that they evaluate HS according to discernible political motivations, primarily to provide data for crime statistics. There is a uniform federal framework with the following mutually exclusive PMK-types: PMK – right, PMK – left, PMK – foreign ideology, PMK – religious ideology, PMK – other/not assignable. Within CHA, aggregated data on this is essential for understanding and projecting trends regarding the political motivations of perpetrators. It can inform future crime prevention efforts and internal training:

If there has been an extreme increase, for example in the area of PMK – right, you can say: Okay, we need to do more in prevention, perhaps make the departments more aware, and also look into why there has been this increase (L3).

4.6 Threat awareness

Threat awareness refers to the perception, comprehension, and projection that an immediate physical threat to individuals, groups, or institutions is conveyed by HS content. We consider it separately because, although it involves the interpretation of information from other sub-types, it does not match any of them. In addition, LEAs and RCs often take exceptional measures in case of a corresponding characterization of content. Several of them review reported or identified HS early to determine whether a threat is conveyed:

The first thing you look at is whether measures need to be taken immediately. Does it involve any threats, any specific dangers for someone? And that is of course always the first thing that is checked (L1).

If a potential threat can be deduced, e.g., in case of direct or veiled calls for violence, these organizations immediately inform the responsible sections within the state criminal police office (LKA) or federal criminal police office (BKA) so that those can initiate security measures. This may involve sending police units to threatened actors or perpetrators:

With experience, you know what tends to indicate lethal or serious violence. We immediately have in mind that there is a risk aspect…. And we approach the authorities in the respective state quite quickly. If it’s acute, they can visit the house. Or talk with the threatening person (L7).

This awareness sub-type is heavily dependent on interpretation. Perpetrators’ intentions must be comprehended and available information must be evaluated to project the likelihood of a threat.

5.1 Implications for reporting tools

The majority of HS cases that LEAs and RCs handle are based on reports. To ensure that transmitted data can contribute meaningfully to CHA, in particular evidence awareness, both the scope and quality of information must satisfy the recipients’ requirements. If this is not the case, a time-consuming inquiry for additional information or manual search and documentation effort may be necessary. ²⁶ This has implications for reporting tools.

Reporting tools should account for both reporter and recipient perspectives (D1): Both LEAs and RCs offer public channels for reporting HS, mostly e-mail contacts and web portals. ^{3 , 26} However, their appeal and usability remain uncertain as there is a lack of user-centered research on their design. Previous HCI research involved targets of gender-based harassment in developing documentation and report creation tools. ^{16 , 17} However, none of these tools allow the submission of reports to LEAs, RCs, or similar organizations, and the requirement elicitation was limited to the reporting side. Researchers designing such tools should thus engage with both reporters and recipients to ensure good usability, user experience, completeness and actionability of transmitted data, and data privacy and security.

Reporting tools should facilitate evidence documentation (D2): The fact that LEAs and RCs frequently receive insufficient URLs and (audio-)visual evidence suggests that securing and submitting such information can be challenging for the reporting party. This could be addressed by providing easy-to-understand instructions within reporting tools that explain step-by-step how to create sufficient screenshots, i.e., with timestamp and URL, and generate URLs pointing directly to HS content. Technical support for evidence documentation would be even more convenient, e.g., by enabling reporting tools to capture sufficient (audio-)visual evidence and automatically retrieve correct URLs. Previous research on online harassment documentation tools offers suggestions to this end. ^{16 , 17}

5.2 Implications for OSINT tools

If reported information is not sufficient for handling cases or initiating criminal proceedings, or if LEAs proactively examine cases, OSINT investigations can be utilized to gather additional information alongside traditional investigative approaches and data requests at platforms. We can formulate two implications for OSINT tools that assist with this.

OSINT tools should allow evidence documentation (D3): Some interviewees, especially from police agencies, use OSINT to collect perpetrator-related information. Our findings suggest that OSINT tools for detecting and characterizing hate networks, ⁷² tracking hate-related keywords, ⁷⁴ or identifying malicious users ⁷³ presented in recent research may be beneficial for enhancing perpetrator, victim, and context awareness in LEAs and RCs. However, what these tools have in common is that they have no documentation functionality for acquired information. Yet, evidence is essential in our domain, especially in preparation for criminal proceedings. Thus, OSINT tools should provide documentation functionalities, such as generating (audio-)visual evidence or saving URLs and metadata.

OSINT tools should follow privacy by design principles (D4): With OSINT tools, LEAs and RCs can effectively collect relevant information and thus strengthen their CHA. However, authorities are subject to particular restrictions regarding the collection and processing of personal data, especially from uninvolved third parties. State-of-the-art tools for hateful content have no safeguards in this regard. ^{72 , 73 , 74} Riebe et al. further show that privacy issues are rarely considered for similar tools in cybersecurity contexts. ⁸⁹ Privacy by design principles should thus inform the development of novel solutions to ensure their applicability and the legal admissibility of gathered evidence. Research on OSINT in other law enforcement contexts can provide guidance. ^{70 , 90}

5.3 Implications for AI-classification tools

In LEAs and RCs, the assessment of HS with regard to its relevance under criminal law, present hate types, targeted groups, extremist relevance, and immediate threats is essential for the initiation of responses. Especially if large data volumes induce information overload, ^{4 , 5} there is potential to use AI to evaluate corresponding content according to these aspects. In light of our findings, there are several implications for designing such tools.

Classification tools should be capable of multi-label classification (D5): Some of the most important assessment tasks in the investigated domain can entail the simultaneous assignment of cases to several subcategories, e.g., extremism types or criminal norms. However, multi-label HS classification has generally received little scientific attention. ⁶³ Available models for hate types or targeted groups are limited to a few classes. ^{22 , 23 , 24 , 76 , 77} For classification by extremism type or criminal norm, no models are available, and end user-centered tools are limited to binary classification. ^{12 , 67 , 68 , 69 , 91} Thus, designers of classification tools should put a strong emphasis on implementing models for practice-relevant multi-label tasks.

Classification tools should only perform a pre-assessment (D6): The assessment of HS is often complex, ⁹² highly context-dependent, ²⁷ and influences far-reaching decisions, e.g., on criminal proceedings or security measures. Erroneous decisions may thus have serious consequences. Therefore, within classification tools, AI should only perform a pre-assessment to support human decision-making, e.g., as part of a preliminary case prioritization or allocation. To avoid algorithmic over-reliance, user interfaces should highlight the preliminary nature of the assessment, communicate limitations and error rates, and encourage users to question and alter the assessment. As part of this, explainable artificial intelligence (XAI) approaches may render decisions more comprehensible. Benefits of XAI in HS classification have been researched, ⁹³ but only for binary tasks.

Classification tools should be adaptable to evolving requirements and contexts (D7): Within the law enforcement and RC domain, assessment criteria for HS can evolve over time. For instance, the assessment of content’s criminal relevance may be affected by the advancement of the legislative framework and case law, or novel hate types or targets may emerge due to a shift in discourse. In addition, some LEAs and RCs conduct monitoring in context of specific events or discourses, where generic HS classification tools may only be of limited use. Designers should, therefore, implement steps to ensure the long-term applicability and short-term adaptability of detection tools. One way to increase model flexibility might be to tailor pre-trained large language models (LLMs) to the application domain. ⁹⁴ These could then be fine-tuned for specific tasks or contexts, i.e., by using few-shot learning approaches that require only a small amount of training data. ⁹⁵ Data augmentation, i.e., the generation of artificial training data, ⁹⁶ might be particularly helpful when fine-tuning such models if only limited data for novel hate types, events, or discourses is available.

5.4 Implications for visual analytics tools

The goal of visual analytics research is to turn information overload, which might be induced by the large volume of reported incidents or irrelevant public data considered for OSINT, into an opportunity. Decision-makers, such as in LEAs or RCs, should be enabled to examine large-scale, multi-dimensional, multi-source, and time-varying information to achieve a situational overview and make effective decisions in time-critical situations. ⁹⁷ From this perspective, our research suggests three design implications concerning data collection, data modeling, and analysis of hate speech incidents.

Visual analytics tools should facilitate the modular integration of reporting and OSINT data sources (D8): While reporting tools are the backbone for data collection in LEAs and RCs, our interviews indicated that OSINT tools help to enrich reported data or allow for the proactive identification of additional cases. Both channels rely on diverse information sources, such as apps, e-mail, social media, and web portals. Combined with the issue of regularly changing application programming interfaces (APIs), ^{98 , 99} it seems important to provide opportunities for the modular integration of data sources. While results from reporting and OSINT might be shown in separate feeds with individual cases, the interface should facilitate relationship awareness ¹⁰⁰ if, e.g., OSINT data is required for the handling of a reported case or a cross-case analysis is conducted to identify connections and patterns.

Visual analytics tools should enable the analysis of context, perpetrator, and victim information (D9): The execution of visual analytics tasks requires the definition of a proper data model. ⁹⁷ By conducting our interview study, we established a nuanced understanding of context, perpetrator, and victim information, which is required to develop such a data model in future work (see, e.g., a data model for cyber situational awareness ¹⁰¹ ). This is also a prerequisite for the display, aggregation, visualization, and exploration (e.g., zooming, filtering, and details on demand ⁵⁶ ) of CHA data on the interface level. Based on aggregated temporal data on incidents, affected social groups, and used platforms, visualizations should enable the analysis of specific timeframes (see, e.g., [102]), temporal trends, and future projections as a foundation for mitigation and response strategies. Furthermore, a map view might be a promising complement to highlight the locations of victims, perpetrators, responsible authorities, or counseling services (see, e.g., [13], 103]).

Visual analytics tools should integrate XAI into their processing pipeline (D10): Our results suggest that AI should be used to enrich factual information with predictive information to allow for a rapid prioritization of incoming reports, especially if an immediate physical threat risk was predicted. While our interviewees acknowledged the importance of human decision-making, the integration of visual analytics and XAI ¹⁰⁴ seems promising to facilitate the understanding of algorithmic classifications. This allows for a more effective model reconfiguration, e.g., by leveraging data augmentation ⁹⁶ and few-shot learning, ⁹⁵ if the performance is insufficient or legal conditions change. Another potential of visual analytics lies in the visualization of AI-detected discourses, topics, ¹⁰⁵ and events. ¹⁰⁶ This could help in the detection of temporal trends for projections and reactive measures.

5.5 Limitations and future work

Our research is subject to some limitations. First, the findings of qualitative research are influenced by subjectivity. Our interviewees’ responses are shaped by their individual perspectives and experiences. Moreover, our interviewing style can influence response behavior, and our data analysis depends on our interpretations. ¹⁰⁷ Therefore, a standardized quantitative survey on relevant information types with staff at LEAs and RCs could be implemented in future research. Triangulation with our data could decrease the impact of subjectivity, while in-depth insights into rationales and practices are still feasible. ¹⁰⁸ Second, our sample exhibits shortcomings. On the part of the LEAs, we only interviewed mid-level or senior staff, often with a coordinating function. This was not intentional, but resulted from contacted organizations’ suggestions of interview partners. Given the lack of representation of case managers or staff with other non-coordinating roles, certain information types of relevance to their work may have been missed. Future research could uncover potential biases by obtaining a more heterogeneous sample. Third, while we also cannot claim validity for all German LEAs and RCs, the generalizability of our findings outside the German context is particularly difficult to assess. Since we only interviewed one RC with an international scope and no non-German LEAs, some information types and CHA sub-types might be specific to the national context, especially those centered around criminal investigations. Cross-country research could thus examine which information types are of international relevance. Fourth, we specifically focused on the law enforcement and RC domain to approach situational awareness of online HS. We did not consider the perspectives of other stakeholders engaging with online hate, such as content moderators, intelligence agencies, and civil society monitoring projects. While there is user-centered work on combating HS with these actors, situational awareness perspectives are missing. Therefore, we see potential in following the example of cyber situational awareness research, ^{14 , 15} pursuing a sound conceptualization of CHA in future work.