Four eyes, two truths: Explaining heterogeneity in perceived severity of digital hate against immigrants

1 Introduction

In recent years, hateful online communication has become so widespread that it is now part of users’ everyday lives. For instance, Castellanos et al. (2023) showed that 65 % of their student sample had witnessed online hate speech during 2020 and 2021. Numbers for adults (36.2 % and 68.8 % for online hate and cyberbullying, respectively, Rudnicki et al., 2023) and, most importantly, subtler hate (e.g., dark inspiration, Frischlich, 2021) may be even higher. While repeated exposure to digital hate content has serious implications for people’s mental health (Lo Moro et al., 2023), it also affects societal debates (Brüggemann and Meyer, 2023) as well as societal cohesion given that (digital hate) content exposure exerts significant cognitive, attitudinal, emotional, and behavioral effects (e.g., concerning immigration, Ziegele et al., 2018). Immigrants are a persistent target of digital hate, where they are often described as a threat to national, cultural, economic, and personal security (Costello et al., 2021). This negative sentiment towards immigrants is widespread in the European Union (EU), as reflected in a current EU-wide representative survey finding that 43 % of EU citizens and even 49 % of Austrians perceive immigration as the primary challenge facing the EU (European Commission, 2021). Although numbers for the prevalence of digital hate against immigrants are sparse and should be considered with care, figures between 16 % and 24 % and 4.4 % and 9.5 % are reported in Czech and German anti-muslim Facebook groups, respectively (Hanzelka and Schmidt, 2017). Other research across European countries reports even higher numbers in a sample of immigration-related Tweets (e.g., 88 % in Lithuania in 2018 or 53 % in Cyprus in 2020; Arcila-Calderón et al., 2022). Such prevalences are alarming and require further in-depth analysis.

What reported prevalences tend to obscure is that people substantially differ concerning whether they experience supposedly hateful content as such. In other words, what may be categorized by researchers, for instance, as online hate speech might not be considered as severe or even hateful at all by some social media users. Differences in users’ severity perception of digital hate have been documented for different contents (Kümpel and Unkel, 2023) and across dispositional and sociodemographic variables (such as extraversion and social background; Akhtar et al. 2020; Sang and Stanton, 2022). Heterogeneous perceptions are highly problematic for social media platforms which are obliged to remove certain content as well as law enforcement who aim to oversee and track digital hate moderation practices, complicating their efforts and deteriorating outside plausibility. This perceptual heterogeneity has significant implications for computational approaches which are necessary to digital hate detection and moderation at scale given the importance of consistently labeled training data (Waseem, 2016). Here, research has consistently revealed disagreements among anonymous annotators concerning what is hate speech and what is not (Yin and Zubiaga, 2021). While some data-driven approaches were designed to account for these disagreements and, as a result, showed improved algorithm performance (Akhtar et al., 2020), there is a pressing need for empirical insights into what determines regular users’ varying sensitivities about the severity of digital hate content (i.e., subjective evaluation of seriousness, harm, or impact associated with instances of hateful online content before higher-level categorization or volition comes into play). Such insights would provide another layer that might help improve training data and, subsequently, algorithms that aim to be sensible to perceptive nuances and contingently engage in moderation best suited for certain content. Such algorithms are supposed to follow a human-centered approach to increase well-being, aligning with central values of the European Union (i.e., fundamental rights, human agency and oversight, transparency, nondiscrimination and fairness, societal wellbeing; see European Parliament, 2023) in order to “serve the needs of the society and the common good” (p. 9), as described in the AI Act. This recently introduced legislation on artificial intelligence (AI) in the EU aims to provide a unifying framework for the adoption of human-centric and trustworthy AI that ensures the protection of health, safety, fundamental rights, and environmental concerns through harmonized rules for AI systems’ market placement that prohibit certain practices, impose requirements for high-risk AI systems, establish transparency regulations, regulate general-purpose AI models, implement market monitoring and enforcement measures, and support innovation (European Parliament, 2023).

Consideration given to different perceptions when creating AI-driven hate detection models denotes a prerequisite to coincide with the values of the EU and, subsequently, the AI Act. This approach would acknowledge the complexity of human interactions and ensure that AI models account for a wide range of cultural, social, and contextual factors, thereby promoting fairness, accuracy, and effectiveness in identifying digital hate while minimizing biases and potential harm. Furthermore, by incorporating and considering interindividual differences, such algorithms would likely increase transparency, interpretability, and reproducibility by enabling a thorough understanding, contextual adaptation, personalized detection strategies, explanatory feedback, user-centric evaluation metrics, user-driven model development, and community engagement. By accounting for individual variations in perception, algorithmic detection models can better align with user expectations and societal norms, thus leading in all probability to increased public support through higher accuracy (Nussberger et al., 2022).

This paper provides a threefold contribution to the existing body of research. First, by directly investigating heterogeneity of perceived severity of digital hate against immigrants in a quota-based sample from Austria, a Central European country with comparatively firm digital hate regulations yet below-average scores on migrant empowerment indices (Solano and Huddleston, 2020), we address one of the most problematic issues in both conceptual discussions about digital hate typologies and practical applications of algorithmic detection and moderation systems by testing a current classification taxonomy. Second, we do so by accounting for personality traits and attitudes (as well as their interaction) with predictions being derived not from an overarching theoretical framework but instead from domain-level approaches contributing different pieces of the puzzle that facilitate a better understanding of digital hate perception. Third, we employ an advanced methodological approach by conducting an autoregressive path model for a non-student population sample that is less likely to suffer from various common methodological biases.

2 Theoretical background

Digital hate as an umbrella term encapsulates hostile behaviors that have been characterized by ambiguity in conceptual boundaries (Matthes et al., 2023). More specifically, defined as all types of norm-violating hostile expressions disseminated digitally and directed either against a specific person or groups, digital hate comprises constructs like online hate speech, cyberbullying or cyberaggression, as well as other more vaguely defined phenomena such as diverse forms of hostile actions and modes of offensive speech whose conceptualizations typically overlap up to the point where differentiation occasionally becomes arbitrary. As such, it is not necessarily limited to hate based upon group characteristics (e.g., gender, ethnicity, sexual orientation; see Gelber, 2021) but also includes other online hostilities that share an ordinary understanding of hate (see Brown, 2017) in an attempt to partially unify partially overlapping constructs in this broad field. With this overarching conceptual perspective as a starting point, we more specifically follow a recently introduced approach by Rossini (2022) who differentiates between two distinct categories of hateful speech: Incivility, characterized as violations of interpersonal communication norms, and intolerance, which pertains to expressions that threaten democratic principles and values such as equality, diversity, and freedom. These two overarching categories can be further operationalized into profanities, insults, expressions of outrage, and character assassination as subcategories of incivility, and discrimination and hostility as subcategories of intolerance (Bianchi et al., 2022).

This set of hateful behaviors is reflected in computational detection approaches, which are typically centered around phenomena, such as hate speech (Yin and Zubiaga, 2021) or cyberbullying (Perera and Fernando, 2021), but can also be more holistic, for instance, by using toxicity labels (Sheth et al., 2022). Complex machine learning (ML), which requires labels and annotated features for supervised approaches (i.e., support vector machines, naive bayes, logistic regression, decision trees, k-nearest neighbor; Mullah and Zainon, 2021) and deep learning (DL) algorithms, which only require labels (i.e., convolutional neural networks, long short term memory, bi-directional LSTM; Malik et al., 2022), are the most widely established tools for these tasks. ML is a field of AI that enables systems to statistically learn from data, while DL is a subset of ML that involves neural networks for pattern recognition (Dargan et al., 2020). DL text representation advancements have resulted in transformer-based embedding techniques that fuel developments in the field; however, performance is still improvable (Yin and Zubiaga, 2021).

Model performance is inherently dependent on the underlying dataset (Hettiachchi et al., 2023). Dataset creation often follows a top-down methodology, whereby researchers select data, inadvertently introducing bias (van Rosendaal et al., 2020), which is then annotated by crowdworkers (Akhtar et al., 2020) adhering to predetermined definitions. This approach tends to overlook individual perspectives. Once user ratings are aggregated for subsequent analysis, lower inter-rater reliability emerges together with unknown biases (Kocoń et al., 2021). Current research accounts for annotators’ topical knowledge, their rating confidence, as well as their demographics, leading to improved models (Akhtar et al. 2020); however, personality traits and attitudes are still being neglected (Kocoń et al., 2021).

Varying severity perceptions of both incivility and intolerance can be theorized through different theoretical lenses. Social constructionism broadly argues that sociocultural and historical factors shape individuals’ conceptions of reality (Burr, 2015). From this perspective, digital hate content is subject to varying interpretations because of individuals’ socialization and enculturation. This broad theoretical position can be supplemented by micro-level theories from psychology. Cognitive appraisal theory (Watson and Spence, 2007) helps elaborate how these individual realities vary from person to person. The theory posits that any stimulus evaluation varies as a function of subjectively perceived (mis-)alignment with personal goals and expectations, controllability, and causal attribution. These propositions can be transferred to perceptions of hate stimuli both offline and online, opening up the field for personality and attitudes as determinants of varying cognitive appraisals. Together, heterogeneous constructions of reality, along with their subsequent distinctive appraisals, may provide a sound theoretical background for examining varying perceptions.

Research has already identified a few factors influencing severity perceptions of digital hate incidents, including sociodemographic (e.g., age, Sang and Stanton, 2022; gender, Binns et al. 2017; education, Cowan and Hodge, 1996; culture, Lee et al., 2023; race, Sap et al., 2022; political leaning, Hettiachchi et al., 2023) and dispositional factors (e.g., extraversion and agreeableness, Sang and Stanton, 2022; moral integrity and sexist attitudes, Hettiachchi et al., 2023; knowledge about hate speech, feminist and anti-racist attitudes, Waseem, 2016; and altruism, free speech attitudes, racist beliefs, traditionalism and language purism, Sap et al., 2022). Given the challenges with heterogeneous perceptions in annotation, it is necessary to further advance through theoretically rigorous social scientific investigations. The present study addresses this need by examining how severity perceptions are predicted by personality traits and attitudes (see Figure 1 for the theoretical model).

Figure 1:

Theoretical Model.

3 Method

This project was part of a comprehensive two-wave panel survey amongst Austrians that was conducted between July 27 and August 5, 2023, for Wave 1 (W1) and September 27 and October 6, 2023, for Wave 2 (W2). The entire survey was screened to be of minimal ethical risk by the IRB of the Department of Communication at the University of Vienna (ID: 20230705_029). Additional material (including datasets and analysis scripts) are available at https://osf.io/ynuxq/.

4 Results

Zero-order correlations are displayed in Table 1. Both empathy dimensions as well as anti-migrant attitudes and the SDO dominance subscale were highly correlated (r > .5). Since predictors were entered simultaneously into the path model, shared variance between them has been partialized out, which, at least to some extent, may alter construct meanings. Notably, assumption checks indicated some residual normality issues and decreasing variances in residual errors, possibly introducing bias (see OSF Appendix Figures 3a–e)

5 Discussion

This study’s objective was to investigate varying severity perceptions of digital hate against immigrants by assessing the impact of personality traits (i.e., trait empathy and identity insecurity) and attitudes (i.e., anti-migration attitudes and SDO) based on domain-level theories in order to advance human-centered classification algorithms as the AI Act by the EU requires. Results revealed that empathic suffering displayed the predicted positive association with perceived severity, whereas feeling others did not. These differential findings emphasize the importance of considering empathy facets when exploring perceptions of digital hate (Wachs et al., 2023). It is evident from our data that individuals with a heightened propensity to experience genuine emotions because they perceive the suffering of others are more likely to perceive digital hate directed at immigrants as severe. This suggests that emotional activation (rather than emotion mirroring) plays a more vital role in shaping one’s perception of digital hate. Together, these results prompt further investigation into the intricate relationship between empathy and varying digital hate perceptions, particularly when directed at vulnerable groups.

Conversely, we found no support for a relationship between identity insecurity and perceived severity. This may carry implications for uncertainty identity theory, since it postulates that individuals experiencing insecurity align with their in-groups to mitigate uncertainty, potentially leading to phenomena like xenophobia (Wagoner and Hogg, 2017). Given this lack of alignment, it might be that perceptions, such as the perceived severity of digital hate, may only be partially influenced by bystanders’ identity security, perhaps particularly when it is directed at marginalized groups like immigrants. Further research is needed to explore possibly more nuanced mechanisms underlying the proposed relationship, for example by considering potential moderating variables.

When it comes to the prediction via attitudes, anti-migrant attitudes turned out as expected, negatively predicting perceived severity, suggesting that topically coherent prejudices (here against immigrants) appear to color how digital hate is perceived. This aligns with prior research, which has demonstrated a connection between negative attitudes toward immigrants and their dehumanization (Markowitz and Slovic, 2020). When immigrants are perceived as lesser humans, reduced perceived severity of digital hate might be the logical consequence. Future research could explore the generalizability of this finding to other target groups of digital hate and may properly test for potential mediators being at work.

Findings partially supported the predictive role of SDO as we only found a significant positive relationship for egalitarianism. These results again highlight how differently constructed subdimensions may affect evaluations of digital hate targeting immigrants. Dominance’s statistical meaninglessness suggests that the active subjugation of outgroups may not be as prominent as one might expect. This challenges the assumption that individuals endorsing this aspect of SDO would consistently perceive digital hate targeting immigrants as less severe (Ho et al., 2012). Conversely, the positive relationship between egalitarianism and perceived severity indicates that less confrontational hierarchy-enhancing ideologies may be an important driver for increased severity perceptions. In other words, individuals who subscribe to egalitarian but still socially stratified systems may be potential allies when it comes to digital hate against immigrants – a proposition, however, that requires more evidence.

No significant interactions between personality traits and anti-migrant attitudes were identified. This has two key implications. First, it highlights the enduring positive influence of empathic suffering on digital hate perception, regardless of contradictory attitudes. In essence, individuals with higher levels of empathic suffering tend to consistently perceive digital hate as more severe, irrespective of their anti-migration attitudes. Secondly, identity insecurity consistently did not affect digital hate perception, independent of anti-migrant attitudes.

Finally, results provided partial support for interaction effects between SDO and personality traits. Again, dominance did not turn out meaningful even in interaction with empathy and identity insecurity. In contrast, a significant negative interaction between empathic suffering and egalitarianism was observed. Broadly, this result indicates that greater or weaker egalitarian attitudes are less relevant in determining whether or not one evaluates hateful communication as problematic for those people who have a strong dispositional tendency to share the suffering of others, thus partially explaining attitude-incongruent evaluations and perhaps even behaviors related to severity perceptions (e.g., bystander interventions). If this could be replicated on a state level, it might also further explain the varying effectiveness of messaging strategies. That is, triggering or inhibiting empathic suffering via certain message cues may make anti-egalitarian audience members more or less aware of harms related to digital hate, respectively, presenting an intriguing avenue for future research. Given that this interaction can also be read the other way around, our finding might also suggest that cultivating egalitarian attitudes may be crucial for countering detrimental but hardly changeable personality traits. This is essential because empathic suffering’s positive relationship with severity perception also means that low-scoring individuals are more likely to disregard attacks on immigrants. Egalitarian attitudes may overwrite this socially undesirable disposition.

Regarding theoretical implications, our findings demonstrate that social constructionist (Burr, 2015) and cognitive appraisal lenses (Watson and Spence, 2007) provide solid frameworks for predicting perceptual differences of digital hate. Both personality traits and attitudes selectively influence perceived severity, occasionally interacting with each other. Concerning previously proposed typologies (Bianchi et al., 2022), we show that people do not seem to differentiate between the different types of hate when targeted against immigrants, at least not similarly as researchers do based on fine-grained conceptual distinctions.

We further advance our understanding of heterogeneous digital hate perceptions which yields critical implications for detection methods, public policy, and social media platform governance. More specifically, we show that not only sociodemographics but also personality traits and attitudes can predict between-person differences in perceived severity. These findings offer implications for strategies aimed at sensitizing individuals to digital hate. Notably, emphasizing the role of empathic suffering, which might be triggered as a temporary state, may contribute to enhancing people’s capacity to recognize instances of digital hate as problematic and encourage intervention. The other way around, providing egalitarian framings may be promising to counteract the impact of inhibitory triggers of empathic (non-)suffering. In the context of dataset creation for detection tasks, it appears, based on our findings, increasingly necessary to carefully choose annotators by also considering personality and attitudinal characteristics to account for perceptual heterogeneity. Moreover, there is a necessity for scholarly and public discourse concerning the selection of appropriate thresholds for automated detection tools, as there exists considerable variability in individuals’ sensitivities to digital hate.

6 Limitations

From a theoretical perspective, forthcoming research could consider incorporating additional dimensions of empathy (e.g., emotional attention, Alloway et al., 2016) and identity insecurity (e.g., insecurity due to dissimilar others; Massey and Cionea, 2023), which we only covered selectively. Further, a variety of other personality traits and attitudes (e.g., toward free speech) might be relevant for severity perceptions. Additionally, we examined de-contextualized severity perceptions of social media users as potential bystanders, which might differ from when participants are instructed to imagine themselves in other roles (e.g., as moderators or annotators). Another limitation relates to our focus on attitudes and traits that naturally disregards other important factors determining severity perceptions (such as, e.g., social norms).

Methodological constraints include the assessment of perceived severity through verbal descriptions rather than tangible examples and the reliance on self-reports. While using descriptions instead of stimuli is a typical approach in sensitive contexts where forced exposure comes with ethical concerns, gauging perceived severity without direct exposure may introduce bias and might hurt external validity. On a related note, the collapse of the two-factor solution might also be induced by the specific target group investigated, showcasing that digital hate against disempowered targets is not dependent on the specific content type. Additionally, self-reports are commonly challenged regarding whether participants can (or want to) accurately specify their perceptions in such a manner. To improve, it is imperative to sensitively expose participants to authentic digital hate and consider psychophysiological or behavioral measurements in addition to self-reports. Besides measurement limitations, cross-construct dynamics, especially regarding anti-migration attitudes and SDO, might have impacted our model.

7 Conclusion

Social media companies increasingly rely on automated detection methods for digital hate and, by doing that, struggle with heterogeneous perceptions among annotators. In this study, we demonstrate the predictive value of empathic suffering and egalitarian attitudes on users’ perceptions of digital hate targeting immigrants. Our findings can be read as a call for data annotation efforts to consider these factors more thoroughly for instance, by grouping and weighing individual responses accordingly in ML and DL experiments. In this way, the goal of increasing social acceptance of algorithmic moderation and improving detection performance might be closer in reach.