Generative AI and Contestation and Instrumentalization of Memory About the Holocaust in Ukraine

The recent rise of generative artificial intelligence (AI) can lead to fundamental transformations in the field of Holocaust remembrance. Earlier non-generative forms of AI, which focused on identifying and retrieving historical information via search engines and content recommenders systems^[1] have already caused profound changes in how individuals learn about the Holocaust. By deciding which information sources and individual content items to prioritise to their users (Makhortykh et al. 2023a), these systems shape the visibility of individual Holocaust heritage institutions in digital environments and influence how successful or unsuccessful the online educational and commemorative campaigns are. At the same time, occasional evidence of non-generative AI systems amplifying the visibility of antisemitic and denialist content^[2] also raises concerns about the possibility of these systems undermining the institutional efforts to counter Holocaust denialism and hate speech, in particular in the light of the low levels of Holocaust knowledge around the world (e.g. Claims Conference 2020, 2021).

In contrast to non-generative AI systems, foundation models, such as GPT or Stable Diffusion, allow their users not only to find and retrieve relevant content but actually generate it from scratch. It enables new possibilities for Holocaust remembrance, including a representation of the past from a multitude of perspectives and multiple scales, as well as a broader access to lesser known aspects to it (Kansteiner 2022), in addition to new possibilities for detecting distortion and denial of historical facts (Makhortykh et al. 2023b). However, there are also growing concerns about generative AI contributing to the distorted representation of the Holocaust due to non-intentional hallucinations and fabrications of the AI models (Alkaissi and McFarlane 2023) or intentional jailbreaking by malicious actors (Tucker 2023). These concerns are amplified by the difficulties in differentiating between human- and AI-made content (Susnjak 2022) that can facilitate the production of non-authentic historical content (e.g. fake Holocaust testimonies) that can not only mislead individual users of AI systems but also contribute to the manipulation of the public opinion, in particular, if such content is used for instrumentalising the past.

The concerns are particularly pronounced in the case of Eastern European countries, where Holocaust memory is often instrumentalised by a diverse range of political actors. Many of these countries constitute fractured memory regimes (Kubik and Bernhard 2014, 17), which are distinguished by the tendency of their political elites to monopolise public memory practices which are then used to legitimise elites’ decisions, including potentially unpopular moves. Often, such instrumental uses of the past are accompanied by the use of online media for reinforcing dominant representations of the past and suppressing alternative interpretations to eradicate memorial dissent. While most existing studies looking at these “web wars” (Rutten et al. 2013) focus on human users and their efforts to contribute to the instrumentalisation of the past or counter it (e.g. Barna and Knap 2023; Gaufman 2015; Kalinina and Menke 2016; Khlevnyuk 2019; Kulyk 2016), a few recent studies also highlight the growing role of AI systems in this context (Makhortykh, Urman, and Ulloa 2022; Zavadski and Toepfl 2019).

Among many instances of instrumentalisation of Holocaust memory in the region, the use of it by the Kremlin in the context of the ongoing aggression against Ukraine stands out. Since the beginning of the aggression in 2014, the pro-regime Russian institutions and activists extensively relied on Second World War references for mobilising the Russian public and constructing negative identities of Ukrainians resisting the aggression (Gaufman 2015; Makhortykh 2018; Pakhomenko et al. 2018). The instrumental use of war memories, including the ones related to the Holocaust, has increased following the large-scale invasion in 2022 (Ferraro 2023; Shevtsova 2022) with the primary aim of demonising Ukrainians among Russian and Western audiences. Often, such instrumentalisation involves the distortion of historical facts related to the Holocaust in Ukraine: a telling example is the speech of Vladimir Putin in September 2023, in which the Russian president claimed that Ukrainian collaborators were the main perpetrators of the Holocaust in Ukraine and performed cruelties of which SS soldiers were not capable of, which resulted in the death of a quarter of all Holocaust victims (Meduza 2023).

Despite the intense instrumentalisation and distortion of Holocaust memory by the Kremlin, there is limited understanding of how it can be amplified or countered by generative AI. Recent studies (e.g. Urman and Makhortykh 2023) provide evidence that in some cases, outputs of generative AI-powered conversational agents, such as Google Bard, can be systematically skewed in favour of Putin’s regime, including the case of the prompts dealing with the Russian aggression against Ukraine. However, it is not clear to what degree these observations are applicable to Holocaust memory and its instrumental distortion by the Kremlin. In order to address this gap, we audited the performance of two major AI-powered conversational agents – ChatGPT and Google Bard – regarding information on the Holocaust in Ukraine.

For this aim, we used a selection of 74 statements translated into English, Russian, and Ukrainian languages; the statements were inquiring information about the aspects of the Holocaust, which are commonly instrumentalised and often distorted by pro-Kremlin propaganda. Examples of such aspects include the attribution of blame for the Holocaust and its individual episodes to Ukrainians in general, and to specific Ukrainian wartime groups (e.g. the Organization of the Ukrainian Nationalists or the Nachtigall battalion) and the details regarding specific instances of the Holocaust (e.g. the number of victims and the ways of killing). We also included a few prompts dealing with the mass atrocities targeting non-Jewish communities in Ukraine, in particular ethnic Ukrainians and Poles, for comparative purposes.

To evaluate the performance of the two conversational agents, we compared the responses provided by them to the predefined baseline (i.e., what we would perceive as the correct answer to the question based on existing research regarding the Holocaust in Ukraine). The baseline for the prompts varied from simple “yes” or “no” answers to the range of possible options (e.g. in the case of the debated number of victims for specific instances of the Holocaust). For some questions, where there is no consensus among historians, we identified as a baseline an answer mentioning that the question is unclear or debated.

While we still need to finalise the analysis and ensure the agreement between the coders evaluating the outputs of conversational agents, our preliminary results indicate several troubling findings regarding the performance of agents in relation to information about the Holocaust in Ukraine. Overall, our analysis shows a relatively low accuracy of the agent outputs, with the proportion of correct answers varying between 30 % and 55 % depending on the conversational agent and language. In addition, we considered about 18–30 % of the outputs to be partially correct. We found that Google Bard was approximately 1.5 times more accurate than ChatGPT in English and Ukrainian, but not in Russian. The latter can be explained by a striking amount of non-responses for Bard in Russian (over 30 % of cases), while in the other languages, this number was almost negligible (around 1 %).

The proportion of outputs considered inaccurate varied between 20 % and 45 %, with the largest number of inaccurate responses produced by ChatGPT in Ukrainian.^[3] In a number of cases, these inaccuracies were related to Ukrainians and Ukrainian organisations being incorrectly presented as Holocaust perpetrators. In certain cases, the attribution of blame was rather absurd: for instance, conversational agents claimed that the mass murder of Ukrainians in Koryukivka in 1943 was conducted by Ukrainian nationalistic groups despite it being the sole responsibility of the Hungarian occupation forces allegedly assisted by Russian Hiwis. Another example relates to outputs (e.g. of ChatGPT in Russian) to prompts regarding the Nachtigall battalion, which persistently claimed that the battalion did not exist during the Second World War, and was instead formed in 2014 and participated in the Russian-Ukrainian war.

In addition to the fundamentally incorrect responses (i.e. the ones directly contradicting the baseline established by human experts), we observed multiple instances when conversational agents distorted historical details regarding the Holocaust. In some cases, such distortions were related to factual errors such as incorrect naming of the location of the atrocity (e.g. a Ukrainian village, Koryukivka, was referred to as a location in the Kursk region of Russia), or the way Jewish victims were killed in the course of a specific atrocity (often, such details were based on a few well-known instances of the Holocaust, such as Babyn Yar massacres, and then were attributed to other Holocaust episodes by the agents). However, in other cases, agents went as far as inventing historical personalities (e.g. non-existing Ukrainian and German perpetrators who assumingly committed atrocious actions) or even quotes from eyewitness testimonies hallucinated by the models powering the agents. In a number of instances, these testimonies included fake claims about the involvement of Ukrainians in the Holocaust, thus aligning with the distortion of the historical facts by the Kremlin.

This small-scale investigation of the performance of generative AI-powered conversational agents has several implications. The first is the rather urgent problem of the poor quality of information generated by AI models regarding the Holocaust in Ukraine. While the distortion of historical facts does not always align exclusively with the Kremlin’s interest and, in some cases, mistakenly denies the involvement of Ukrainian perpetrators in the Holocaust, it also often amplifies the Russian regime’s efforts to use the past to demonise Ukrainians. Even in those cases, when distortion of the historical facts does not directly serve the instrumental purposes of external actors, it amplifies the epistemic uncertainty regarding the Holocaust in Ukraine, and can facilitate the manipulation of facts in the long term.

The second implication relates to the data the agents are trained on and their ability to assess their own limitations. The large number of inaccurate responses can be explained by the limited access of conversational agents to information about certain episodes of the Holocaust, as well as their inability to correctly retrieve information in relation to more niche Holocaust-related topics. Under these circumstances, agents often produce inaccurate descriptions of events that may include details mistakenly drawn from other Holocaust-related events or even invented by the agent. Instead of pointing out their limited knowledge of the subjects, about which they are prompted, conversational agents in most cases either refuse to respond without providing a clear reasoning for it or, which is more concerning, still provide an inaccurate answer with a large degree of confidence.

The third implication concerns the importance of following the earlier calls (e.g. Walden 2023) to conceptualise the role of digital technology in general and generative AI in particular for the future of Holocaust memory both in Eastern Europe and worldwide. The inability of some of the most used AI-powered conversational agents to provide accurate information about the historical mass atrocities, including the well-documented instances of the Holocaust, is concerning. Not only can it enable new possibilities for distorting historical facts and manipulating the public opinion, but also it can challenge the core mission of Holocaust heritage institutions aiming to preserve memories of the victims and prevent instrumental uses of the past. In addition to exploring possibilities for improving the performance of generative AI models on the level of their design (e.g. by improving the training data regarding information about the Holocaust and other genocides, and optimising guardrails to minimise the likelihood of the model to invent historical facts), it is crucial to increase awareness about the possible risks of AI, and create possibilities for the general public and heritage practitioners to develop AI literacies required to address these risks.