AI-based character generation for disease stories: a case study using epidemiological data to highlight preventable risk factors

2.1 Narrative medical visualization

Narrative visualization blends storytelling with data visualization to craft data-driven stories. ⁸ In medicine, storytelling adapts to healthcare’s unique challenges, employing diverse data while respecting patient privacy and sensitivities. ⁹ Medical data stories frequently appear in the news and health media. The methods for crafting these stories are an ongoing topic in narrative medical visualization research, ^{10 , 11} which explores how visualization can facilitate physician-patient interaction and motivate care. ¹² McCurdy et al. ¹³ employed visual storytelling for patients to express health issues, using a timeline-based template to show symptom extent and duration. Meuschke et al. ⁹ established a template for crafting data-driven disease stories for patients and the general public based on analyzed web blogs. Mittenentzwei et al. ¹⁴ provide a design process for narrative medical visualizations where a character is one of the four main ingredients for a narrative. The book chapter by Garrison et al. ¹⁵ aligns closely with our work, highlighting the importance of narrative in health literacy. Their focus on narrative techniques, personalization, and data integration offers a theoretical and practical framework that complements our character-driven storytelling approach.

Kleinau et al. ¹⁶ applied a template in a case study to enhance engagement and knowledge retention through structure, personalization, characters, and metaphors. Building on this, Mittenentzwei et al. ¹⁷ studied the effects of narrative genres (slideshow and scrollytelling) in disease communication for usability and aesthetics. Later, they examined the impact of different story protagonists on engagement and memorability ¹ and found that a human protagonist exerts various influences on story perception, especially improving engagement. While these stories are data-driven, the characters they feature are hand-crafted and designed based on discussions with clinicians. In our work, we propose a way to derive characters directly from epidemiological data.

2.2 Towards data-driven storytelling with characters

Characters enhance a story’s narrative, particularly in the media and entertainment industry, with the gaming sector leading character-driven storytelling research and innovation. Cavazza et al. ¹⁸ focus on the dynamic interaction of autonomous characters. More recently, Mariani et al. ¹⁹ and Sheldon ²⁰ investigated character-driven storytelling. Mariani et al. focus on developing characters for interactive storytelling, exploring how richly crafted characters with complex traits and backstories can drive narratives, engage audiences, and deepen immersion in story worlds. Katherine Isbister ²¹ illustrates the complexity of the design space for characters by exploring game character design from a psychological perspective. Notably, the concept of nonplayer characters (characters not controlled by the user) shares similarities with characters in data-driven storytelling. In game design, decisions primarily revolve around social roles and integrating characters into gameplay, emphasizing that characters with a greater presence should be developed with more depth. However, unlike typical nonplayer characters in games (e.g., traders, guides, or enemies), the characters in data-driven storytelling lack interactivity with the user. Both data-driven stories and games share the common goal of fostering empathy for their characters. Katherine Isbister ²¹ specifically discusses the concept of Social Syncing, highlighting how emotions can be “contagious.” This underscores the significance of emotions in character design, as users can internalize and adopt the emotions conveyed through a character’s expressions. Faces play an important role in conveying the emotions of a single character, while body language is especially useful for communicating connections with others. Chen et al. ²² investigates the user’s identification with the protagonist, concluding that a high degree of identification can lead to self-referencing (meaning that users connect the story’s content to their own lives).

In game development, narrative design (story and character roles) and visual realization by concept artists are typically handled by two distinct individuals. ²³ We utilize GenAI to assist the concept artist’s role, while the narrative design is driven by data. We have also included additional context, such as emotions, in our GenAI prompts, as they also play an important role in character design. In data-driven storytelling, characters play a more nascent role. Dasu et al. ²⁴ introduced a framework to improve data storytelling by integrating characters, making visualizations engaging and accessible. They analyzed 160 data stories to outline character roles – main, supporting, and antagonistic – and their interactions within story plots, demonstrating how characters can effectively bridge complex data and audience understanding through visual metaphors and narrative structure. While this work demonstrates how characters can clarify and humanize complex data, they remain limited to manually defined character roles and lack automation or design aspects like emotional expression or visual coherence.

2.3 Generative AI art

The recent popularity of openly available (latent) diffusion models like Stable Diffusion, ⁴ along with user-friendly applications such as Midjourney ³ and Leonardo AI, ²⁵ has brought GenAI art to the forefront. These models and tools enable image creation from text across various domains.

While the training of diffusion models may require large amounts of training data and compute resources, a technique called Low-Rank Adaption ²⁶ allows for efficiently fine-tuning existing models for specific domains and applications, e.g., stylized or near photo-realistic images. Community-driven platforms like Civitai ²⁷ allow requesting and sharing such models, while tools like the stable-diffusion-web UI by Automatic1111 ²⁸ allow users to easily use them. In addition to creating images from text, diffusion models can also be used for inpainting (replacing content in a masked image), sketch-to-image (conditioning the creation of images based on a simple sketch), and tasks such as up- or down-sampling. These advancements in GenAI art offer various opportunities to support design processes. So et al. ²⁹ introduced a GenAI tool for generating narrative medical visuals from social media posts on conditions like diabetes, based on the ‘bio-psycho-social model’ encompassing medical, psychological, and social factors. ³⁰

Existing research in interactive storytelling and game design reveals a gap in leveraging GenAI to synthesize narrative roles, emotions, and data-driven contexts. Our work directly addresses this void by exploring how GenAI tools can assist in creating emotionally resonant, visually impactful characters tailored for medical storytelling – a domain where characters can bridge technical data and audience empathy but remain underexplored.

5.1 Extract character from data

Our character design process is built upon high-quality longitudinal data. We propose a query-filtering approach to extract a targeted subset of data for subsequent image generation using the SHIP dataset. This filtering applies two sets of criteria, aligned with design principles C1, C2, C7, and C8, see Figure 1 (Step 1.2. Extract Character from Data). Hard criteria select candidates embodying the core medical message and story intent, such as advocating for positive lifestyle changes. These include individuals with profiles aligning with disease risk factors, who have either overcome health issues or developed diseases due to lifestyle choices. Soft criteria consider candidates with notable lifestyle improvements or deteriorations, requiring background knowledge of disease risk factors for identification.

For the narrative context, we chose additional variables that can help describe the personal background and personality of the protagonist, e.g., gender, family, education, and occupation. This shows many parallels to the creation of personas, where user data is gathered and analyzed for demographic or behavioral patterns. ⁴⁴ The resulting personas are often enhanced with, e.g., pictures or names to create a comprehensive profile. We added emotional context based on additional resources (e.g., documentaries about patients with non-alcoholic fatty liver disease ^{45 , 46} ). Other additional resources may include information provided by medical experts or internet sources about the disease and research into character design, e.g., appropriate names that fit the age and background of the characters.

Algorithm 1 shows our hard and soft criteria for extracting the female character recovered from NAFL, while Algorithm 2 depicts the extraction of the male patient that develops NAFL. The hard criteria fall into three groups, focusing on variables strongly correlated with NAFL identified from SHIP studies: ⁷

1. Diagnose-related variables: For the diagnosis of NAFL, only subjects with fatty liver combined with no alcohol problems and either recovered or developed the disease were considered.

2. Demographic variables, i.e., gender, family, education, and occupation to give context and history to the character.

3. Risk factors or lifestyle-related variables for NAFL, e.g., age, BMI, alcohol use, physical activity. ⁷

Soft criteria identified those participants with positive lifestyle changes: weight loss, exercise, smoking cessation, and reduced alcohol consumption.

5.2 Text-to-image generation

We employ a text-to-image generation method to create characters based on hard and soft criteria, see Figure 1 (Step 2.1. Text-to-Image Generation). Budich et al. ⁶ utilized Leonardo AI, ²⁵ a web-based GenAI tool with pre-trained models for various image styles. However, users have limited control over the models, as regular updates may alter the functionality of older prompts. Given this significant drawback, both for the long-term application of the tool in supporting character design for data-driven medical stories and for research studies, we adapted our image generation pipeline to locally run Stable Diffusion version 1.5. This decision was based on the following criteria:

1. Reproducability: Stable Diffusion is widely recognized as a benchmark for evaluating generative AI techniques, including diffusion models, and is frequently cited in academic papers and studies as a comparative baseline. Its fixed releases ensure that researchers can utilize the exact same version of the GenAI model that we employed.

2. Open source: Stable Diffusion’s open-source nature allows users to thoroughly examine its model architecture and algorithms, providing insights into its inner workings, an essential aspect of ensuring reproducibility. The absence of licensing fees for non-commercial use makes it a cost-effective choice for institutions, researchers, and developers, especially in academic or non-commercial settings.

3. Data privacy: Stable Diffusion can be executed locally, ensuring that both the input and output data remain exclusively accessible to the creator. This is particularly critical for applications in sensitive areas such as medical topics, where data privacy is paramount.

4. Multimodal capabilities: Stable Diffusion offers versatile functionality essential for character creation, including text-to-image generation, image-to-image translation, and tools for refining outputs, such as specifying areas of an image to be altered or preserved. This adaptability makes it suitable for a wide range of GenAI tasks. Its all-in-one functionality streamlines the pipeline, eliminating the need for multiple platforms and enhancing accessibility for character creation workflows.

5. Available resources: As an open-source platform, Stable Diffusion provides access to a diverse range of models and a flexible toolchain developed by its active community. This supports fine-tuning the model and storing optimization results, including network weights and learning rates, e.g., as model checkpoints tailored to various image styles and user-friendly GUIs. Its high degree of customizability makes Stable Diffusion a preferred choice for creators seeking adaptability and control.

6. Relevance: Stable Diffusion is a leading diffusion-based generative model that creates high-quality, photorealistic, and stylized images from text prompts. Competing with proprietary models such as OpenAI’s DALL-E and MidJourney, it leverages advanced latent diffusion models (LDMs) to deliver high-quality outputs with optimized computational efficiency and resource utilization.

1. Model Configuration. We utilized the Stable Diffusion web UI, ²⁸ offering a graphical, browser-based interface to specify parameters like model selection, number of output images, and prompts. Customized model checkpoints can be developed by retraining neural network weights using a small set of images with consistent styles. Pre-trained model checkpoints (files containing the weights of a Stable Diffusion model specialized in a particular style, subject, or aesthetic) are available on the Civitai website, ²⁷ with our selection being “HT Photorealism v4.1.7”, ⁴⁷ tailored for portrait photography. Table 1 outlines our parameter setup, predominantly using default settings from the Stable Diffusion web UI v1.6.1.

2. Positive and Negative Prompts. Positive prompts indicate the desired image outcome, while negative prompts specify undesired features. Image generation models may lack regulation of their generated output, potentially generating not-safe-for-work (NSFW) content, including nudity. To encourage safe results, we included “NSFW” as a negative prompt for all images. Additionally, “text” and “deformed” were added as negative prompts, as models often struggle with text and complex structures like hands.

3. Prompt Adaptation due to Model Specifics. Using Stable Diffusion instead of Leonardo AI requires adjusting previous prompts used by Budich et al. ⁶ for optimal outcomes. Style-specific keywords (e.g., semi-realistic or photorealistic) were removed from the prompts as they are now defined by the model checkpoint. Additionally, character names were removed from the prompts, as they no longer significantly affect their appearance.

4. Effects of Different Keywords. The keyword “kindergarten teacher” introduced occupation-specific backgrounds like classrooms and blackboards in some images, often altering clothing to blouses or plain long-sleeved shirts, occasionally with floral patterns. However, the occupation’s influence on output varies, and it may perpetuate stereotypes and clichés for certain jobs. Keywords like “educated,” “mother,” and “married” had minimal impact on output, mainly altering characters’ clothing and jewelry in some images. For instance, “educated” occasionally produced more elegant looks with prominent jewelry and blazers. Despite their minor influence, these keywords remain in the prompt scheme as they are derived directly from data. Additionally, even if keywords have minimal effect, the model can still generate believable characters consistent with patient data.

Certain keywords, like “platinum blonde” or “wearing glasses” have a more significant impact than others, overriding descriptors such as “mother,” “kindergarten teacher,” and “educated” from the original prompt. However, terms like “open-minded” have negligible influence on results. The influence of keywords on generated images varies based on the model’s training data. Reducing keywords in a prompt enhances control over the output by decreasing the weight of each keyword. Consequently, we eliminated keywords not derived from patient data and precise appearance descriptions, as they override data-derived keywords.

5. Keyword Weights. Prompt weights can be modified to adjust specific character features. The default weight is one, which can be increased by adding + or a number between 1.1 and 2 or decreased by − or a number between 0 and 0.9. + stands for a weight of 1.1, ++ adds a weight of 1.1², and so on. The same scheme applies for reducing the weights where – equals 0.9 and −− equals 0.9². For example, “A beautiful, friendly, 65 years old, (overweight)+, educated woman” assigns a weight of 1.1 to “overweight”.

6. Base Prompt Input. We modified the prompt scheme from Budich et al. ⁶ to suit model specifics, resulting in the following base prompt input: <age> <gender> <body weight> <family> <occupation> <education> <emotion> <ethnicity> <picture section>.

All keywords, except for emotion and picture section, were derived from data. The base prompt scheme yielded satisfactory results, though additional keywords could further modify the images, such as altering hair color or style. We generated images for each character at time point s0, selecting emotions based on their health status. The prompts used were:

• 65 years old women, overweight, mother, kindergarten teacher, educated, scared, European, portrait

• 32 years old man, normal weight, married, salesman, strong and focused, European, moderate drinking, portrait

7. Refinement. In the initial phase, a range of output images is generated, providing the designer with options. Illustrated in Figure 1, even with the same prompt, Stable Diffusion produces diverse characters. From here, designers can adjust prompt component weights to experiment further.

5.3 Image-to-image translation

The character is finalized using Image-to-Image translation, ⁴⁸ refining the design by generating similar images. This process preserves key features like pose, hairstyle, and clothing, allowing the selection of the best character variant. Refer to Figure 1 (Step 2.2. Image-to-Image Translation) for the finalized female base character. Our pipeline generates character variants from this final image, covering diverse attributes such as emotions, body weights, and ages crucial for depicting long-term medical narratives.

Image-to-image translation alters an image based on a text prompt. The inpaint approach allows users to specify regions for alteration by drawing on the image, as illustrated in Figure 2. To modify emotions, we painted within the facial region. Adjusting weight and age involved including the outer boundaries of the head, considering potential changes in head size. Using inpaint helped to preserve features affected by unstable diffusion. We maintained consistency by employing the same negative prompt as in text-to-image generation and adjusted the positive prompt by modifying emotion, age, and weight keywords.

Figure 2:

The original image (a) and an inpaint area (b) are combined with a text prompt for image-to-image translation. Different emotions, like scared (c) and happy (d), are generated within the painted area.

We used image-to-image translation to show the character’s evolution from time step s0 to s2 in the SHIP study. Over ten years, the female character lost weight and recovered from fatty liver, while the male character gained weight and developed liver inflammation. Their facial expressions were adjusted, with healthy characters showing positive emotions and diseased characters showing negative ones. See Figure 1 for the female character’s changing expression, body weight, and age.

5.4 Generalizability

There are multiple epidemiological studies that offer their data for research purposes, such as the UK Biobank, ⁴² Rotterdam Study, ⁴³ and German National Cohort (NAKO). ⁴⁹ Our pipeline can be applied to any epidemiological study, but hard and soft criteria need to be adjusted based on the story intent. Cohort data, collected through studies such as the ones mentioned above, usually consists of sociodemographic data (employment, education, ethnicity, etc.), lifestyle information (preventive and risk factors, e.g., smoking, diet, activity, etc.), and health status and physical measures/biological samples (body weight, blood pressure, blood samples, etc.). These are the most important aspects of developing a narrative medical visualization focusing on one disease and featuring an individual health journey.

In our proof of concept, we created characters that have either a poor health status that improved due to lifestyle improvements, or a good health status that worsened due to poor lifestyle. Based on diseases tracked in the epidemiological study, concrete hard and soft criteria can be defined to find fitting patients. Sociodemographic data helps shape the AI-generated character’s appearance, such as age in the prompt. Lifestyle information can be used to visualize lifestyle aspects relevant to a certain condition, e.g., a character can be depicted as smoking. Lifestyle information visualizes relevant aspects, like depicting a character smoking. Health data, physical measurements, and biological samples are used to represent the disease, its causes, symptoms, or consequences, such as body weight.

It is important to emphasize that since individual patients are used as models for the GenAI characters, consultation with a clinician – e.g., someone who works with the cohort data – is essential. This will ensure that the patients selected are representative and that the final character matches the physician’s approval, especially with regard to risk factors for certain diseases.

6.1 Participants

To answer Q1 and Q2, we conducted a quantitative evaluation using an online questionnaire over six weeks. 29 participants completed the questionnaire, while 14 participants dropped out before completion (their responses were not included). The questionnaire and the responses are available as Supplementary Material. While we covered a range of different backgrounds, most participants are under 30, have a university degree, and live in Germany, see Table 2. Gender distribution is nearly equal between males and females.

6.2 Results of Q1 – authenticity

Select keywords for an image. We explore the authenticity of the images in representing the underlying data. Participants were tasked with assigning keywords to the images, covering six categories corresponding to each keyword used in the image generation prompt:

Age: young adult (18–29), middle-aged (30–59), mature (60+).

Body Weight: overweight, normal weight, underweight.

Family: married, widowed, separated, divorced, single, parent.

Occupation: Military, Agriculture, Manufacturing, Construction, Natural Sciences, Traffic/Logistics, Sales/Trading, Business Organization, Health Care, Social Sciences (short forms based on the occupational sectors defined in the German Classification of Occupations 2010 – Revised Version 2020 ⁵⁰ ).

Education: No qualifications, Secondary school-leaving certificate, High school diploma, Completed apprenticeship, University degree.

Emotion: angry, afraid, sad, calm, strong/focused, happy (Based on the “Feelings Wheel” developed by Willcox ⁵¹ for aiding individuals in identifying and articulating emotions. It is a structured framework that categorizes and subcategorizes different emotions. To keep the identification of emotions simple, we chose to only use the six top categories.)

Age, body weight, and education were single-choice questions because each participant could only select one correct answer per category. In contrast, family, occupation, and emotion were multiple-choice questions, allowing participants to select all plausible options, as multiple options could apply to a person simultaneously.

In Figure 3, participants’ keyword selections reveal that most (n = 21) perceived both characters as Middle Aged (30–59), aligning with our prompt for the male character’s age. Although the female character is intended to be 65 years old, most participants selected an age range only slightly above this. Overall, participants tended to view the male character as younger and the female character as older, highlighting a noticeable age gap between them. Regarding body weight, participants categorized the male character as normal weight (n = 28) and the female character as overweight (n = 23), consistent with our original prompt. While participants mainly interpreted the male character’s emotion as calm (n = 27), which was not in our original prompt, they consistently chose positive or neutral emotions, reflecting our intention for the healthy character to be depicted positively. Calm was also the most frequently selected emotion for the female character (n = 26). Few participants opted for afraid (n = 3), the emotion we used in our prompt, or another negative emotion like sad (n = 3). We often had to increase the keyword weights for emotions in our prompts to elicit more pronounced facial expressions. However, our findings suggest that the model struggled to produce strong negative emotions, as these were rarely identified by participants.

Figure 3:

The bar chart displays participant responses for male and female characters. Colors represent different categories: age (single choice), body weight (single choice), emotion (multiple choice), occupation (multiple choice), family (multiple choice), and highest level of education (single choice). Keywords from the original prompt used to generate the images are underlined for the male (blue) and female (red) characters.

Regarding the characters’ occupations, the most common responses for the male character were Business organization (n = 24) and Sales/Trading (n = 24), the latter aligning with our original prompt input. For the female character, the predominant response was Occupations in health care, the social sector, teaching, and education (n = 27), which corresponds to our original prompt featuring the keyword kindergarten teacher. While participants may have based their choices on visual cues like clothing, these occupations are also heavily gender-typical, suggesting that gender alone could strongly influence the results.

The most frequently selected family term was Married for both the male (n = 19) and female characters (n = 23). Many participants also thought of the male character as being single (n = 18). While the family was not specified for the female character, we used the keyword married in our original prompt for the male character. About half of the participants also thought that the male (n = 14) and female characters (n = 16) were parents, which is the case for the latter according to the underlying SHIP data. The male (n = 22) as well as the female character (n = 13) were assumed to have a high educational level, i.e., a University degree, by many participants. While the education of the male character was not specified in the data, the female character has a High school diploma, which was only selected by one participant. When asked whether the characters in the static images felt like real or computer-generated people, participants frequently chose both options without a noticeable tendency.

Select images for keyword combinations. We showed the participants a subset of keywords used to generate the image variations for s0 and s2 for the male (A and B) as well as female (C and D) characters. The participants had to select all the images matching the keywords. The keyword combinations are:

1. 32 years old, Strong/Focused, Normal Weight

2. 65 years old, Scared, Overweight

3. 42 years old, Sad, Overweight

4. 75 years old, Happy, Normal Weight

The results are shown in Figure 4. Nearly all users chose character images with the correct age and weight. For A, the image generated with the keyword “angry” was selected most frequently (n = 24), with the image generated with “strong/focused” as the second most selected (n = 18). The angry expression, characterized by a furrowed brow, may also be interpreted as concentration or focus, indicating the ambiguity of this facial expression.

Figure 4:

Top three images selected for each keyword combination. Below each image is the number of responses for that image and the emotion keyword we used in the prompt. The task was multiple-choice. Each participant could select all the images they found that matched the keywords. Images matching our prompt are highlighted with a dashed frame.

For B, C, and D, participants most commonly selected the correct images. However, for character C, the top image received only 17 votes (the top images for A, B, and D received at least 24 votes) indicating more difficulty in recognizing this emotion. Although correct images generally dominated the votes, participants often interpreted other images as depicting the same emotions. Furthermore, the model’s subtle alterations in the images made distinguishing emotions challenging, as seen in the comparison of strong and calm emotions in Figure 4(A).

Select images for character descriptions. This approach assessed whether participants correctly identified emotions, but we also explored whether the selected emotions resonated with the audience or if they would choose images depicting other emotions. We provided the participants with these short stories:

1. “Thomas Schmidt is a 32-year-old salesman. He is married and has a secondary school level. Although he has a poor diet, he has a normal body weight and moderate alcohol consumption but no alcohol problem.”

2. “Thomas Schmidt is a 43-year-old salesman. He is married and has a secondary school level. Due to his poor diet, he became overweight and developed critical stomach fat and liver inflammation.”

3. “Emma Winter is a 65-year-old woman. She is married and a mother. After graduating from high school, she became a kindergarten teacher. She is overweight and has been diagnosed with a fatty liver. However, she has no alcohol problem and is an ex-smoker.”

4. “Emma Winter is a 75-year-old woman. She is married and a mother. After graduating from high school, she became a kindergarten teacher. She started exercising and lost weight during the last years and successfully recovered from her fatty liver.”

For each story, they should select one image that best fits the character described in the text.

The most chosen image for each story is shown in Figure 5. Participants generally selected images with accurate age and weight. For the male character, the top images for both the (a) healthy and (b) diseased versions depicted anger. While anger in the diseased character could reflect frustration about his condition, the reason for its prevalence in the healthy character is less clear. It might be that participants interpreted the expression not as anger but as focus, similar to observations in a previous task. For the female character, participants predominantly chose scared for the diseased version (c) and happy for the healthy version (d), aligning with the images selected for our stories later in the evaluation. This suggests participants often use negative emotions to depict disease. The task required a single image selection per character. Notably, options (a) and (c) received fewer votes, 6 and 7, respectively, compared to (b) and (d), which garnered 13 and 12 votes, respectively.

Figure 5:

Most voted images for every story including the frequency of votes it received and the emotional expression shown. On the right side of every image, the frequency of self-reported criteria that were most influential for the participants’ choice is depicted.

We asked the participants which criteria they considered for their answers. Body weight, age, and emotion were frequently selected. We offered participants to enter their own criteria. Five participants used this option and stated that the following criteria played a role in their decision:

1. Alcohol consumption, Dietary, Occupation

2. Signs of health from poor diet such as eye bags, wrinkles, unhealthy skin color

3. Bad teeth

The image generation model sometimes incorrectly rendered teeth, suggesting that the normal-weight male character had poor dental health when smiling. Although unintended, this coincidentally aligned with the character’s backstory of having a poor diet that might affect his teeth. Participant responses indicate they also look for subtle health indicators like eye bags.

6.3 Results of Q2 – resemblance

We assessed whether participants viewed images from image-to-image translation as depicting the same person as the original. They rated the resemblance of each altered image on a 5-point Likert scale; results are shown in Figure 6.

Figure 6:

Frequency of participants’ responses on the similarity of two images, aggregated by gender and weight to the statement “both images above show the same person, probably at different ages, weights, and moods.”

Participants rated images where only the emotion was altered, but age and body weight were constant, as most similar. However, in cases where age and body weight changed, images of the female character transitioning from overweight to normal weight were seen as more similar than those of the male character going from normal to overweight. Emotional changes did not significantly affect the similarity ratings. In the image-to-image translation process, the model frequently altered the noses of characters, especially giving overweight characters more upturned noses. This suggests that the model often applies learned features associated with certain body weights instead of preserving the original features, leading participants to see them as different individuals.

7.1 Refinement

The main feedback of the participants regarding which cues are helpful to identify a disease was (1) showing more than a portrait section to enable the audience to assess body weight better and (2) showing more subtle signs of disease such as eye bags, unhealthy skin color, or bad teeth.

While the former was easy to implement by generating torso shots instead of portraits, the latter could not be achieved as the generative model was not able to realize these subtle changes in a sufficient way. For example, even when using inpaint to define that only the area below the eyes should be changed to make eye bags more prominent, it generally led to deformed eyes. Therefore, instead of depicting the male character’s poor diet using bad teeth, we chose to place him in a fast-food restaurant. This framing also allows us to discuss the choice of backgrounds with the design experts in the next step by comparing the female character with a neutral background and the male character with a non-neutral background. Based on the torso shots, we generated comic variants of the characters. Therefore, we used the model checkpoint revAnimated V1.2.2. ⁵² As there is a huge body of medical comics used for health education, ⁵³ our intention is to investigate whether photorealism or comic style is preferred by design experts in certain aspects. We created six different pairs of images (three for each character, including the ones used in the previous user study). Each pair shows a character at time point s0 and s2 in the SHIP study, see Figure 7.

Figure 7:

The refined images.

We used the following prompts for the generation of photorealistic torso shots:

• 65 years old women, overweight, mother, kindergarten teacher, educated, scared, European, waist and torso shot, standing, solid color background

• 75 years old women, normal weight, mother, kindergarten teacher, educated, happy, European, waist and torso shot, standing, solid color background, sports

• 32 years old man, normal weight, married, salesman, focused, European, waist and torso shot, standing in a fast-food restaurant

• 42 years old man, overweight, married, salesman, sad, European, waist and torso shot, standing in a fast-food restaurant

Using image-to-image translation, we generated the comic adaptions by adding the keyword “comic character” and including the LoRA (Low-Rank Adaptation) inkSketch V1.5. ⁵⁴ LoRA is a fine-tuning technique that modifies a pre-trained model without requiring full retraining. The LoRA was activated using the command “lora:inkSketch_V2:1” in the prompt window.

7.2 Qualitative evaluation

We gathered feedback from four design experts (D1 – D4), one of them being a co-author of this paper.

• D1 holds bachelor’s degrees in design and computer science. They are familiar with medical visualizations and narrative medical visualization, and have practical experience in designing them.

• D2 has a bachelor’s degree in pharmacology and a master’s degree in medical illustration. They have practical expertise in illustration, especially portraits, and working experience in a public health agency.

• D3 has a master’s degree in interaction design and holds a design professorship. They are familiar with illustration, character design, and storytelling and have practical experience in designing them.

• D4 has a graduate degree in medical illustration and has experience as a healthcare communications designer.

We used Miro ⁵⁵ as a virtual whiteboard to show background information about SHIP, NAFL disease, patient backstories, and data. We prepared a tier list on the Miro board for each of the ten design criteria from Section 4. The design experts were individually asked to rank all the image pairs (one pair being the character at s0 and s2 in the same style) from one (very good) to six (very bad) based on how well the image pairs fit each individual criterion. This was achieved by the design experts dragging the generated character images into the tier list, see Figure 8. Furthermore, they were asked to express their thoughts and reasons for their decisions, which were noted down according to the think-aloud method.

Figure 8:

This screenshot illustrates the setup of one of the Miro boards used during the qualitative evaluation with design experts (specifically D4). On the left, orange boxes provide information about the data and the disease. On the right, a ranking list is presented for each criterion (outlined in blue boxes, similar to Section 4). Here, design experts were tasked with ranking the character images for each criterion by arranging them based on their assessments. Images placed higher in the column were deemed to better fulfill the respective criterion. The detailed results can be seen in Figure 9.

7.3 Results of Q3 – refinement

While all design experts have a background in illustration or design and narrative medical visualization, they showed different preferences when discussing the generated images. Their rankings can be seen in Figure 9. It became apparent, that despite their professional background, personal experiences and preferences played an important role in their assessment. In the following, a detailed overview of the design experts’ feedback regarding each criterion is presented. A concluding overview can be found in Section 9.1.

Figure 9:

The design experts (D1 – D4) ranked the generated images regarding each individual design criteria (C1 – C10). Similar to a tier list, the design experts were asked to assign the character images on a scale ranging from 6 (very good) to 1 (very bad), while it was possible to skip ranks and assign the same rank to multiple images. The original ranking from the Miro boards is shown along with an aggregated score for each image style (RP, RT, C) per category (C1 – C10). The scores are calculated by adding up the ranking results, where each image could receive 1–6 points from each design expert (D1 – D4), and allow for a comparison of how the image styles performed per category.

C1. Relevance to Medical Context. D1 favored the torso shots as they show the stomach fat, which is an important indicator for NAFL. D2 highlighted positively that the change in posture in the torso photo of the woman indicates a change in lifestyle. The portraits of the woman also show that she glows more in the second image, but because the rest of the body is not shown, the stomach fat is not visible. The comic version of the male character was ranked low by D1 because the style felt inappropriate. D2 said that using comics would give the creator the opportunity to exaggerate more on, e.g., symptoms. However, especially the female comic character does not look like the same person in the two images. D3 thinks that the comic version of the male character looks too strong and confident like he is in control of his life, which does not fit the story. D1 appreciated the male torso shots’ background for reflecting the character’s poor diet, while D3 found it too bold and stereotypical, preferring the less intrusive grey background.

C2. Informative and Educational. D1 argued that having portraits that focus on the face leads the focus of the story toward the emotions of the characters. However, for NAFL the body shows important indicators, in particular stomach fat. D2 and D4 agree, adding that it is also possible to guess the body weight based on the portraits but it is much harder. While D1 sees a lot of potential in the comic style, but especially the images of the female character do not fit the story as she looks too young in the second image. Additionally, D2 noticed that the facial expression for the comic version of the male character stays the same, and that shifts the focus of the viewer to the changes in the body. However, emotion can show the social implications of the disease, so it depends on the intent of the story if it should be shown or not. D3 thinks that the emotions of the comic versions and the photorealistic body shot do not match the story’s intent. D1 liked that the background of the male torso shots shows context relevant to the story. D2 liked that the female character glows and shows positive emotions after recovering from NAFL. However, the deterioration is more stark in their personal opinion.

C3. Memorable. D1 argued that memorability, in general, depends on the ability of the individual viewer to memorize faces, which differs per person. For the photorealistic images, the portraits might be more memorable as they show the face in more detail. D3 thinks that the expression of the photorealistic torso shot of the female character is especially memorable. D4 thinks that the portrait of the female character conveys a warm and motherly personality while the portrait of the male character shows no traits that stand out in particular. Furthermore, the male torso shots look too cliche. D1 said that the comic version of the female character has a lot of expression, making it more memorable. D3 agrees, but thinks that the increase of memorability might come at a loss of conveying the relevant content. The male comic character is described as being too sleek which reduces memorability. D2 thought that comics are, in general, more memorable. The change of the posture and clothing of the female character is done nicely in both comic and photorealistic torso shots. The stark shading and general art style of the male comic character make him a distinguishable character and not just a random person. In general, D2 said that changing clothing and backgrounds for the individual time steps would be better. The portrait photo of the female character has a very good contrast between the clothes and the background.

C4. Empathy and Emotional Connection. D4 highlighted that a general issue might be that all kinds of characters might act as projection areas, where viewers, despite recognizing that they are close to their own demographics, will think, “This happens to other persons; this will not happen to me.” According to D1 and D2, the portraits are better at conveying emotions as the face is visible in more detail. D2 added that while the male character’s portrait evokes empathy, it does not really look like the same person and the character ages too much. The torso shots of the male character do not evoke much emotion, according to D2. The facial expressions are neutral and the overweight level does not look unusual. D3 mentioned that the photorealistic torso shot of the female character feels more distant to them than the portraits and the photorealistic torso shot of the male character. D1 said that the photorealistic torso shot of the female character shows a large emotional difference between the time steps. In the second time step, she appears proud of overcoming NAFL. On the other hand, the comic version of the male character looks unemotional. His aggressive glance establishes a certain distance from the viewer, and the story is not conveyed that well. However, the background is praised again. D2 said that the style of the comic version of the female character is too different to feel like this is the same person which prevents the viewer from feeling empathy. Furthermore, the comic style looks more abstract, like taken from a textbook, which builds distance between the characters. On the other hand, the photorealistic images feel less distant and give the impression of being real people. D4 favored the female portrait as it looks motherly and warm, but showing the character in a fitting environment, e.g., at home, would increase the emotional connection. While the male torso shots successfully convey the character’s lifestyle choice by placing him in a well-known setting, such as a fast food restaurant, the character itself does not evoke empathy.

C5. Credibility. D1 appreciated that in the photorealistic torso shots of the female character, similarities but also changes are visualized in a credible manner. The character in the second image (S2) looks a bit taller than in the first (S0), which should be improved. D1 said that the portraits of the male character do look like the same person, however, the healthy version (S0) should look happy instead of neutral. The healthy version of the photorealistic torso shots of the male character looks good, but the diseased version does not seem to show the same person. In general, it is important to show the visual and emotional changes of the characters. Changing the clothing style in different images of the same character would add to this. D2 said that in the case of the male character, the portraits, and the comic torso shots are identifiable as the same person. D4 agreed in the case of the comic versions. There, the nose and jawline look similar. The background conveys the eating habits of this person. In the portraits of the female characters, most of the facial features are similar, however, the eyes appear to be too different. D3 and D4 said that the comic versions of the female character and the photorealistic torso shot of the male character do not look like the same person at all as their facial features are very different. D3 only recognized a consistency in the portrait photos. While the poses in the torso shots were praised, the comics did not give D3 a sense of credibility. D4 said that the portrait of the female character looks most consistent.

C6. Personalization and Audience Relevance. D1 and D2 thought that the female character matches the main risk group very well, as women tend to be affected more often than men. Additionally, the demographics in western Pomerania, where SHIP takes place, show that more than every third person is 65 years or older. Furthermore, D2 and D3 said that the male character is too much of a businessman which might not resonate well with a large part of the audience. D3 says that the male character looks too helpless in the portrait photo, which does not invite the viewer to resonate with him. D2 assumed that a face might be more relatable than, e.g., body or pose. The portraits leave more room for imagination, potentially addressing a wider variety of body types. D1 and D2 assumed that comics might not resonate well with the older target group, but this would need further investigation. In the case of the torso shots for the male character, the context in the background increases personalization, according to D1. D3 suggested that another strategy could be to not visualize detailed individuals but instead use archetypes. This way, a larger part of the audience might be able to resonate with one archetype as it is more general. However, it is difficult to create archetypes that are not too stereotypical. Additionally, it would be great for the audience to show both progressions for the same character. Starting with a healthy character, the story can show how an unhealthy lifestyle leads to NAFL and how to recover from the disease by making lifestyle changes. D4 does not see the need to apply this criterion to the characters as they are based on real persons and thus are relevant real-world examples of the disease.

C7. Behavioral Change and Motivation. D1 was unsure if a positive or negative example is more motivating for the audience but has a personal preference for the positive example, comparing it to the psychological concept of positive reinforcement. D2 said that both examples can motivate, and it might also depend on the disease and what works best. For example, there might be a difference in the case of diseases where risk factors are hard to quit due to an addictive factor such as smoking, where an example of a character that quickly progresses towards a healthier lifestyle might not be appropriate. D4 assumed that what works best for an individual viewer might be the character that is closer to the viewers on demographics and lifestyle. D3 argued that the portraits have a different level of meaning, focusing on the emotions instead of on the physique. D1 argues that the comic version of the male character looks cool, not diseased. Thus, the gravity of the disease is not conveyed. D2 agrees, adding that the character does not look like he wants to change which will not be motivating for the audience. D3 sees similar issues in the photorealistic torso shot of the male character.

C8. Data-Informed Design. D1 said that the torso shots are crucial to show the critical stomach fat of the male character as this was one important aspect in the data. For both characters, the change of age and weight is visible in all the torso shots, except the comic version of the female character where the age does not seem to match the data. Concerning the portraits, only age is conveyed prominently, neglecting body fat. D2 agrees, highlighting that the poor diet of the male character is also visible in the torso shots due to the background. For D3 the age was not correctly conveyed in the comic versions. The female character looks too young in the second image, while the male character looks too young in the first image. D4 thought as well that the photographs are generally better at conveying the data.

C9. Ethical Considerations. D1 argues that the SHIP participants’ privacy is guaranteed in all cases. As the portraits do look like real people, D2 assumed that it is hard to get away from any resemblance with real people. D2 added that in the comic versions, an accidental resemblance of a real person is unlikely. The comics are furthermore portraying the characters in a sensitive way as the proportions are still good looking, e.g., going too much in the direction of a caricature might ridicule certain conditions. Portraits are more likely to avoid fatphobia, as it is mostly concentrated on the body. But since weight is an important aspect of the disease, D2 thought it reasonable to show obese bodies in the story. However, designers should think about what visual cues are crucial to show and focus on these. D3 argued that it is not possible to avoid stereotypes completely, as there will always be some aspects that match certain stereotypes. D4 said that they think the comic versions do not get the necessary sensitivity of the topic across. They questioned if the comic characters were respectful to the people from which the data was taken. D1 assumed that it might be possible that images showing an expression that is too bitter might convey the impression of the character being isolated or stigmatized potentially triggering viewers with similar experiences.

C10. Iterative Design and Evaluation. For this criterion, design experts were asked to consider using one of the characters in an NAFL story. They were tasked with selecting images for the iterative design process and suggesting improvements for the first step. D1 preferred the photorealistic torso shot of the female character but suggested reworking the facial expression on the left side, such as reducing the shadow contrast. While portraits alone would not fully represent NAFL disease, a mix of portraits for emotions and torso shots for physique or lifestyle would be ideal. D2 preferred the male character’s portrait, finding it most relatable. A stressful job with long hours leading to poor diet and NAFL was seen as realistic. However, D2 suggested adding more intermediate images to show the characters’ gradual journey and depict how they could recover by adopting a healthier lifestyle. D3 thought the portraits are looking the most natural and credible. It is also a good start for the story to first create empathy so the audience cares for the character and only in the second step show them in the context of the disease. Similar to D2, D3 also liked the idea of showing how a person develops and recovers from NAFL. D4 liked the story of the female character better. However, they disliked the concrete style of the comic. For a medical topic, they would opt for a softer style, e.g., inspired by watercolor.

9.1 Conclusions

In the following we will provide a concluding overview of our results in the context of our research questions.

Q1 – Authenticity: Is the character authentic to the underlying data? The generated characters were found to authentically reflect the underlying epidemiological data in terms of key attributes like age and body weight. Interpreting emotional states, however, proved to be more challenging. Our evaluation results show, that users searched for additional clues for diseases, such as unhealthy skin color, eye bags, or bad teeth. Including these visible symptoms in the character generation process would create a stronger impression. Employing eye-tracking technology in such assessments can reveal which parts of the image capture users’ attention the most.

Q2 – Resemblance: Do the altered images resemble the same person? The resemblance of altered images to the original character was a common critique, though it varied depending on the factor being modified. Changes in emotion did not negatively impact resemblance, whereas modifications to body weight and age often resulted in new character images that lacked sufficient similarity to the original. Notably, with the settings used, image-to-image translation tended to alter not just skin features (e.g., adding wrinkles to signify aging) but also other attributes such as eyes, noses, and occasionally even hair color. While image-to-image translation demonstrated potential for maintaining character consistency, it requires further refinement to ensure coherence during transitions. Specialized tools for ensuring character consistency, e.g., face swap models, ⁵⁸ could be used in case the character’s face is to be preserved. Changing features like the bodyweight may also impact facial features to some degree.

Q3 – Refinement: How to refine the characters to meet design standards? Discussions with design experts highlighted that character design has many variables and preferences, such as artistic styles, facial expressions, and body poses, that differ per person. The evaluation highlighted key strengths and limitations across the styles and suggested additional criteria for designing characters that effectively resonate with target audiences. Torso shots were praised for their ability to visually depict critical medical indicators, such as stomach fat, a significant marker for NAFL (higher scores, e.g., in C2: Informative and Educational and C8: Data-Informed Design). On the other hand, portraits were preferred for the depiction of showing emotions (higher scores, e.g., in C4: Empathy and Emotional Connection). Contextual backgrounds, such as a fast-food restaurant, further enhanced the narrative by illustrating lifestyle factors. However, the background can also be distracting. Comic styles often struggle with maintaining consistency in conveying age and emotional states. This affected audience recognition. Concerns were raised regarding sensitivity and suitability for the target group. On the other hand, comic styles provide a broad range of possibilities to depict different aspects of a disease (e.g., emotions can be emphasized more clearly than with photorealistic representations) and comic characters might increase memorability (higher scores in C3: Memorable) but the current images have several design flaws as described before. Additionally, the evaluation highlighted new considerations for character design:

Comprehensibility: To effectively convey the character’s health journey (or the portion thereof), it is important to go beyond presenting just two images at the beginning and at the end of the journey. Including transitional images that illustrate gradual changes in between can improve narrative clarity and engagement. Combining photorealistic portraits with torso shots provides a more comprehensive narrative. Portraits foster emotional connection, while torso shots highlight physical health markers, creating a balanced and engaging depiction.

Appeal: The appeal of character design should be considered alongside C6: Personalization and Audience Relevance, which focuses on sociodemographic factors. Elements like comic style may vary in appeal depending on the target audience, influenced by factors such as age and culture. Different comic styles might be perceived as more or less appropriate and appealing. Experts recommend exploring archetypes that strike a balance between relatability and generalization while being cautious to avoid stigmatized stereotypes.

Context: Integrating relevant settings, such as a home environment for a female character or a fast-food restaurant for a male character, can enhance narrative immersion and effectively communicate lifestyle factors. However, the chosen background should avoid being overly complex, distracting, or stereotypical to maintain clarity and relevance.

9.2 Future research opportunities

Medical character creation provides individual challenges due to sensitive data in medical contexts. Medical topics evoke emotions, necessitating appropriate, reassuring characters. A character’s reflection of the psychological and physical changes caused by disease is vital for aligning with the narrative, as demonstrated in our case. Based on our proof-of-concept, where we utilized GenAI to create data-driven characters for the application case of NAFL, we propose the following future research directions:

1. Generalizability: Validation against diverse datasets and character sets can verify the generalizability of our approach. Expand the study to a wider range of disease data, identifying classes of diseases and personas for characters.

2. Broaden the application area: Our approach is tailored to epidemiological study data, where the creator can choose from a broad set of patient data entries to find an appropriate base for a character for a data-driven disease story. However, these stories are also relevant beyond epidemiology. Exploring how the pipeline can be applied to other types of data could expand the scope of our approach. In situations where patient data is unavailable, representative personas could be developed in collaboration with physicians.

3. Explore archetype-based designs: To support creators with limited experience in character design, providing generalizable archetypes that avoid stigmatized stereotypes while resonating with broader audiences, would be very helpful. These archetypes must be tested across multiple narratives to determine their versatility and impact on audience engagement.

4. Evaluate style variations: Identifying appropriate styles for different target groups, such as photorealistic and different artistic styles, can guide the character design process in the future. If a set of popular styles can be identified, model checkpoints can be trained and published that are specialized for these styles.

5. Providing context: The influence of context, such as backgrounds in character images, needs to be investigated more thoroughly to identify appropriate settings and designs.

6. Expanding to video and voice generation: Lastly, our approach can be expanded to video generation, including the use of an artificial voice.

7. Tool building and pipeline automation: In its current form, our proof-of-concept requires user input at several stages of the content generation pipeline. Reducing the manual effort in crafting narrative visualizations will be a key focus of future work. For example, using large language models for generating story arcs and matching characters at several stages of their disease.

8. Personalized storytelling and digital twins: Given the current approach one story is crafted for informing a set of patients. With the increasing interest in personalized medicine, crafting stories that match a single patient’s data may be of interest. Creating a prototype story that is automatically completed using the patient’s data may be a first step in providing a more personalized experience.