What kind of technology transparency do users appreciate? Comparison of textual and graphic cues in app design

1.1 Transparency cues as determinants of user mental model

A mental model is a conceptual representation of how a system functions, constructed by users and based on their understanding of the system. ¹⁹ A correct mental model allows users to adequately assess the reliability of a system, while insufficient mental models can result in wrong conclusions or evaluations from system outputs. They can lead to users distrusting or overtrusting a system instead of trusting it to an extent that matches its capabilities. ²⁰ For instance, when users underestimate the capabilities of a research tool, they may distrust and take over the research themselves, even though they could use the technology as support and it could take work off their hands. Conversely, if they overestimate its capability, they may rely too much on the tool and not question the results when necessary. Thus, the accuracy of users’ mental models is crucial for an appropriate and adapted use of the technology.

Research suggested that transparency cues can improve users’ mental models. For example, Gunning et al. ²¹ postulated that an explainable system enables users to revise their mental model, which then can lead to appropriate trust, use, and better performance. Likewise, the Human-Automation System Oversight (HASO) Model ²² states that the system’s transparency influences the mental model and thereby promotes situation awareness. Situation awareness, in turn, helps users to monitor the system and intervene if necessary to avoid unwanted consequences. Such effects of transparency cues are also empirically supported. As an example, Ramaraj et al. ²³ showed that natural language and visual explanations of a robot’s behavior led to more accurate mental models of users. Similarly, Anderson et al. ²⁴ demonstrated that more extensive explanations were associated with better mental models of users who interacted with an AI during a game – though at the cost of higher cognitive loads and partly lower performance. In the context of mobile apps, higher transparency in a mobile sensing app significantly affected users’ understanding of data usage in the app but not their understanding of data logging. ²⁵

Additionally, the felt and actual accuracy of the mental model can diverge: Users might subjectively feel like they understand how an app functions even if their explanations are objectively wrong. ^{18 , 26} This can lead again to inappropriate technology use since users rely only on their subjective understanding of the system in their judgements. Therefore, in our study, we investigate both users’ subjective and objective mental model accuracy to identify corresponding challenges and risks in transparency design. We thus aim to inform design solutions that support appropriate and concurring subjective and objective mental models. Yet, based on the research reviewed, ^{21 , 22 , 23 , 24 , 25} it can be generally assumed that technology outputs with transparency cues result in a more accurate mental model than outputs without transparency cues:

It is yet unclear how exactly explanations should be designed to benefit user understanding instead of being perceived as redundant or overly complicated. ^{27 , 28} Additionally, little is known yet about the generalizability or context-sensitivity of the effect of transparency cues on users’ mental model. We aim to contribute to a more nuanced understanding by comparing the effects of two different types of transparency cues across two different user scenarios.

1.2 Transparency cues as determinants of user experience and explanation satisfaction

User experience describes the users’ perceptions and responses resulting from interacting with a system, such as the perceived attractiveness, efficiency, perspicuity, dependability, stimulation, and novelty of a product. ²⁹ Also focusing on the experience of users, explanation satisfaction represents how content users are with explanations respective transparency cues provided by a system. ³⁰ In contrast to the mental model accuracy determining how a system may be used, the experience of users may influence if it is used at all, making this perspective equally relevant for the evaluation of transparency design. ^{31 , 32 , 33}

Referring to Gunning et al.’s ²¹ model of explainability, transparency cues should affect users’ subjective evaluation of the system, including how satisfied users are with specific explanations. Moreover, Shin and Park ³⁴ postulated in their FAT (fairness, accountability, and transparency) model that technology transparency increases satisfaction and supported this assumption empirically. Various empirical findings, also integrated through reviews, showed similar effects. ^{8 , 35 , 36} Furthermore, models of information processing, such as the Heuristic-Systematic Model/Elaboration Likelihood Model ^{37 , 38} or the cognitive effort perspective ³⁹ , imply that information that is more easily processed by the respective users may be preferred by them. Likewise, Grice ⁴⁰ not only developed maxims related to the content of messages but also one maxim dealing with their presentation to define good communication. According to this maxim of manner, communication can be improved by creating messages with high clarity for recipients. Applied to transparency cues, this may imply that the presentation format impacts the perceptions of users. Accordingly, participants of Vermeulen et al. ⁴¹ specified that they prefer visualizations over textual explanations because the former are easier to understand. Similarly, participants with little technological knowledge favored visual transparency cues over other cues like counterfactual explanations. Yet, Wastensteiner et al. ¹³ found that a textual explanation was better understood but less preferred by subjects. In sum, we assume that even though textual cues might be easier to understand, users may experience graphic cues more positively than textual cues, leading to the following hypotheses:

2.1 Participants

We initially received responses from 334 participants. After excluding participants who completed less than 80 % of the survey (n = 41), our final sample consisted of 293 participants. No further participants had to be removed for answering the attention check incorrectly (“For this question, please mark the answer ‘strongly agree’.”) or for responding more than 2 SD quicker than the average processing time. This number exceeded the required sample size of 147 participants based on a priori power analyses with G*Power 3 (see preregistration at https://osf.io/mh938/?view_only=cb4d737a87a445f7b549be570d436eea for details). ⁴² Participation requirements were an age of majority and proficiency in German. The participants were recruited via university mailing lists, social media, and snowball sampling. As an incentive, they could choose between taking part in a raffle of four coupons of €25 each and getting course credit.

The sample included subjects with ages from 18 to 67 years (M = 24.3, SD = 7.3). Of the 293 participants, 70.3 % identified as female, 27.7 % as male, and 2.1 % as having a diverse gender. Moreover, most participants were students (87.0 %) or employees (10.2 %) and had a high school diploma (65.5 %) or a university degree (29.0 %).

2.2 Design and material

The study was conducted as an online experiment. We implemented a mixed design with transparency cue as a between-subjects factor (no cue vs. textual cue vs. graphic cue, see Figure 1) and the smartphone app type as a within-subjects factor (weather app vs. sleep tracking app). The transparency cue was manipulated by a varying interface design of the two apps. To avoid careless responding due to fatigue, ⁴³ order effect biases, ⁴⁴ and carry-over effects of app understanding from one transparency cue condition to the next one, ⁴⁵ each participant experienced both apps but only with one transparency condition. As a control variable, the suggested reliability of each app was also varied in the conditions with a cue to check the robustness of the results, independent of the transparency cue content. Therefore, the sample contained n = 62 participants in the no cue condition, n = 119 participants in the textual cue group, and n = 112 participants in the graphic cue group.

Figure 1:

Example interfaces of the three transparency cue conditions for the weather app: (A) no cue condition, (B) textual cue condition, and (C) graphic cue condition. Text passages translated from German into English.

During the study, participants should imagine themselves using the apps. For the weather app, participants envisioned that a temperature-sensitive tree, which they are taking care of for a traveling friend, is standing on their balcony, and they are using the weather app to infer whether they should leave the tree out overnight. For the sleep tracking app, the scenario described waking up and checking the sleep data gathered by a sleep tracking ring worn the previous night. The app interface presented a recommendation to get up half an hour earlier in the morning based on the data collected.

Depending on the experimental condition, a different type of transparency cue about the app’s output was displayed (see Figure 1 for examples; all scenarios can be found in Appendix A):

1. In the no cue group, no information on the functioning of the app was given.

2. In the textual cue group, information on the functioning of the app was provided by a natural language, sentence-based explanation. This included a sentence informing users that the forecast is determined in such a way that the displayed temperature can deviate from the actual one by a certain number of degrees (for the weather app) and a sentence explaining sources of information that were recorded and used to compute the recommendation (for the sleep tracking app).

3. In the graphic cue group, information on the functioning of the app was provided by a visual, graphic-based explanation. This included a graph showing an area around the predicted temperature which might also be possible (for the weather app) and graphs for each source of information on which the app’s recommendation is based with symbolic markers indicating why the specific recommendation was made (for the sleep tracking app).

We developed the transparency cues in all conditions as an explanation of how the app output arises and aimed for a realistic design, like it might be implemented in real user settings. To this end, the textual cues were designed to be concrete, sound, and complete ⁴⁶ while still being concise enough (two sentences) that real users would take the time to read them. The graphic cues were designed to contain the same information as the textual cues. Line graphs were chosen since this type is frequently used in weather and sleep tracking apps, is relatively simple and should therefore be familiar and understandable to participants. Similar graph types for both apps facilitate comparisons across the two different contexts.

Additionally, regarding the control variable, the transparency cue suggested that the app has either low or high reliability. In the weather app, this was done by varying how much the actual temperature might deviate from the prediction and, in the sleep app, by varying on how many types of recorded data the app based its recommendation. This should create the impression that the app output is not reliable, giving information on shortcomings of the app, versus that it is reliable, making it safe to consider. Since this is not the focus of this paper, we will not go into this in more detail below.

2.3 Procedure

The study was preregistered, approved by the university’s ethics committee (ethics vote: EK-MIS-2024-239), and implemented as an online survey. First, participants provided written informed consent to the privacy policy and were randomly assigned to one of the five conditions for the weather app (see Section 2.2). They had time to read the scenario and get familiar with the app interface and subsequently answered a questionnaire including the subjective and objective mental model accuracy, user experience, and explanation satisfaction (see Section 2.4). The questions regarding explanation satisfaction were not shown to the control group since they did not receive an explanation respective transparency cue. The same procedure was then repeated for the sleep tracking app, in that participants first received one of the five app interfaces and then responded to the same measures tailored to the sleep tracking app. In the end, questions about demographic information and covariates were posed (see Section 2.4).

2.4 Measures

Subjective mental model accuracy was measured by a single item (“I have understood the functioning of the app.”) on a 7-point scale from 1 (strongly disagree) to 7 (strongly agree). Objective mental model accuracy was assessed using the nearest neighbor method, ¹⁹ in that participants should select the response option that most precisely describes the functioning of the respective app. The four options to choose from varied between the two apps (see Table S1 of the Supplementary Material at https://osf.io/d3g4s/?view_only=967d302afa574792a52b80ee8284eef6). A correct response was coded with 1 and an incorrect one with 0.

As a variable of the experience perspective, user experience was evaluated via the short version of the User Experience Questionnaire (UEQ). ^{47 , 48} The short version contains eight semantic differential items with seven response options that can be divided into the pragmatic facet (including the perceived perspicuity, efficiency, and dependability) and hedonic facet of user experience (including the perceived stimulation and novelty; Cronbach’s α = 0.82, Cronbach’s α_pragmatic = 0.82, Cronbach’s α_hedonic = 0.78). For explanation satisfaction, the Explanation Satisfaction Scale ⁴⁹ was used. One item was removed due to an inapplicability to the described scenarios, and the scale was adapted to the study content (e.g., “system” was replaced by “app”), resulting in six items rated from 1 (strongly disagree) to 7 (strongly agree; Cronbach’s α = 0.92).

Furthermore, covariates that were assumed to influence the results were surveyed. These included the affinity for technology interaction, ⁵⁰ previous experiences with both apps, attitude towards both apps, frequency of smartphone use, and an attention check. All items were surveyed in German, and their English translations can be found in Table S1 of the Supplementary Material (see https://osf.io/d3g4s/?view_only=967d302afa574792a52b80ee8284eef6).

3.1 Descriptive analyses

Table 1 shows the descriptive values of all dependent variables. We found positive correlations between the subjective measures, namely user experience, its pragmatic and hedonic facets, explanation satisfaction, and to a lesser degree subjective mental model accuracy. Subjective mental model accuracy showed only a small correlation with objective mental model accuracy, which is in line with previous research ^{18 , 26} and highlights the relevance of separately investigating both constructs. Objective mental model accuracy did not correlate significantly with any other measures of subjective experience.

3.2 Effects of transparency cues

3.3 Comparison of apps

Mixed-design ANOVAs for all dependent variables showed that significant interactions of transparency cue and app occurred for subjective mental model accuracy (F(2, 290) = 3.63, p = 0.028) and explanation satisfaction (F(1, 229) = 15.57, p < 0.001). For both variables, the difference between transparency cues was larger for the weather than the sleep tracking app, with the graphic cue resulting in lower ratings than the textual cue (see Figures 2 and 3). Interaction effects were not present for objective mental model accuracy (p = 0.105) and user experience (p = 0.446, p_pragmatic = 0.212, p_hedonic = 0.972).

Figure 2:

Interaction of transparency cue and app on subjective mental model accuracy. Means and standard errors are shown for the six conditions resulting from transparency cue (no cue vs. textual vs. graphic) and app (weather vs. sleep tracking app).

Figure 3:

Interaction of transparency cue and app on explanation satisfaction. Means and standard errors are shown for the four conditions resulting from transparency cue (textual vs. graphic) and app (weather vs. sleep tracking app), since explanation satisfaction was not measured in the no cue group.

3.4 Analyses of interindividual differences

Considering the covariates, attitude towards the apps had an influence on objective mental model accuracy (F(1, 56) = 10.94, p = 0.002) and on pragmatic user experience (F(1, 56) = 6.04, p = 0.017). Apart from that, just the frequency of smartphone use showed a main effect on pragmatic user experience (F(1, 56) = 4.12, p = 0.047), while all other interindividual differences did not significantly influence the dependent variables (see Tables S4 to S6 of the Supplementary Material at https://osf.io/d3g4s/?view_only=967d302afa574792a52b80ee8284eef6).

4.1 Advantage of textual cues to users’ mental model

The results support existing models and previous research in that transparency cues had an impact on users’ mental models. ^{21 , 23} While previous research focused mainly on the objective mental model, ^{23 , 24} we demonstrate that both the felt and objectively measured understanding of the technology were influenced by the transparency design. Yet, only textual cues showed a positive effect on the participants’ mental models, while graphic cues did not result in significantly different mental model ratings than interfaces without transparency cues. This was against our assumptions since we proposed that both transparency cues should improve the users’ understanding of the apps. From a theoretical perspective, the result could be explained by different functions of graphic and textual explanations: A previous study on mental model formation found that participants consulted textual explanations as an initial “default” option to acquire a general subject understanding, whereas graphic explanations were only additionally reviewed “on demand” if specific knowledge was required. ⁵¹

Possibly, our results also depend on the concrete operationalization of the graphic cue, which could have been more difficult for participants to understand quickly than the textual cue. This explanation would align with guidelines developed by Xu et al., ⁵² stating that graphic cues should only be used if they are easy to understand. Yet, this suggests that differently designed graphic cues could have caused other responses from users.

As an alternative explanation, the ways to convey mental model information (e.g., textual, graphic) might be differently effective depending on the respective user. Thus, our found advantage of textual cues could imply that textual cues are generally well understood by most users in contrast to graphic ones, which might be better comprehensible only for specific user groups. It could also suggest that mental model information might be typically memorized in a verbal modality and that textual cues, which are closer to this modality, may generally convey such information better. Nevertheless, it cannot be stated at this point that textual cues are superior to graphic ones, making it necessary to investigate user responses to different designs of such cues by varying them, for instance, in the time needed for processing.

4.2 No found superiority of graphic cues on explanation satisfaction and user experience

Looking at the experience perspective, explanation satisfaction was significantly higher for textual than graphic cues. This finding contrasts with our expectations and previous evidence from other contexts where users were more satisfied with graphic cues, ¹³ showing that graphic cues are not per default superior to textual cues. Nevertheless, this may not directly imply that graphic cues cannot result in more satisfied users, since the results on explanation satisfaction corresponded to those on mental model accuracy. It could be that the graphic cues used were perceived as more complex than the textual cues, suggesting that other graphic realizations might lead to opposite effects. According to Kulesza et al., ⁴⁶ a good explanation is concrete, sound, complete, and not overwhelming. While we aimed to portray these characteristics equally in the textual and graphic explanation, the results hint that our concrete graphic cues might have been more abstract or overwhelming, impairing users’ satisfaction.

Contrarily, the transparency cues did not significantly affect the overall and pragmatic user experience. There were also no differences between textual and graphic cues in terms of hedonic user experience, but graphic cues resulted in higher ratings on the hedonic facet than interfaces without transparency cues. Thus, user experience may be differently influenced by transparency cues than the other dependent variables and may depend more on other design factors. Moreover, against skepticism in previous works, ⁵³ transparency cues appear not to have a negative effect on users’ experience and may even be hedonically useful. However, since interacting with the technology can be important for evaluating user experience, ⁵⁴ it is also likely that user experience was not measured reliably and that, therefore, no differences in user experience were found.

4.3 Stronger difference between graphic and textual cues in the weather app

Comparing both apps investigated, the results pointed in a similar direction for the weather app and the sleep tracking app. However, the differences between the textual and graphic cues were more pronounced in the weather app than in the sleep tracking app scenario, due to low ratings of the graphic cues on subjective mental model accuracy and explanation satisfaction. This finding may support the above explanation that the graphic cues might have been more difficult for participants to understand immediately than the textual ones, since the graphic presented more complex information for the weather app than the sleep tracking app. For the weather app, a confidence interval of the temperature forecast was shown, whereas the sleep tracking app simply displayed graphs with smileys to make interpretation easier.

Alternatively, Hoffman et al.’s ¹⁸ model of the explanation process in explainable AI (XAI) might explain why the results were more pronounced for the weather than the sleep tracking app. The authors stated that users have an initial mental model of a system and that they revise this existing mental model based on transparency cues received. In our study, participants had less experience with sleep tracking apps than with weather apps, suggesting that previously existing mental models might have been more developed for the weather app. This, in turn, could have made them more prone to potential expectancy disconfirmations by the cues, causing the results.

Thus, the stronger differences between transparency cues may be either content-related, pointing at the concrete operationalizations of the cues, or context-related, pointing at differences between the apps. This signals that precise implications are not reasonable at this moment and that the need to search for the sweet spot for transparency design prevails.

4.4 Practical implications

While the inconsistency of our results does not allow for any definitive conclusions yet, some practical implications for app developments can still be derived:

1. Our findings suggest that the incorporation of transparency cues does not seem to generally have a negative effect on user experience, as discussed in previous research.

2. More precisely, textual cues were presented as a promising approach to integrate transparency into mobile apps, enhancing users’ mental model and experience.

3. However, special attention may be required for the detailed design of graphic cues. The fact that graphic transparency cues did not increase users’ mental model underlines the need for carefully and purposefully designed cues that correspond to users’ needs. This may apply especially to the portrayal of complex or multi-dimensional information, where graphic explanations are frequently used ⁵⁵ but not necessarily promote user understanding.

4. The context-dependency of our results points to no “one size fits all” solution for enhancing user understanding in mobile apps. This highlights the importance of evaluating each design individually in its specific context to find out which transparency cue conveys relevant information most effectively. Therefore, a target-performance investigation with validated scales (e.g., the TechTra scales ⁵⁶ ) is essential. Yet, it should be beneficial to investigate general factors that make a specific transparency design more effective (e.g., context characteristics, target group characteristics), as previously done for cross-technology factors influencing users’ transparency perception and need. ⁵⁷

4.5 Limitations and future research

The study is constrained by several limitations, offering perspectives for future research. First, it was conducted online without real interaction with the app, which possibly impaired participants’ involvement and reduced the external validity of the results. ⁵⁸ Yet, we chose this approach to enhance the internal validity and obtain meaningful results on the impact of transparency cues that can be further tested in a real setting where users directly interact with a technology and in other contexts (e.g., different app domains). Nonetheless, Babel et al. ⁵⁹ have found that participants’ evaluation of robot characteristics did not differ significantly between online and laboratory contexts, thus providing evidence that external validity is not always impaired in online research on technology interaction.

Second, a cross-sectional design was applied, assessing short-term reactions of the participants. However, as Hoffman et al. ¹⁸ suggest by their explanation process model, effects of transparency cues might alter over time depending on the existing mental model of users. Thus, longitudinal studies are crucial to examine potential changes in users’ responses and identify optimal recommendations for transparency designs.

Third, comparisons between graphic and textual information are inherently challenging due to the amount of information that can be conveyed. For example, the information contained within a single graph might take many sentences to be accurately described in textual form. While we aimed to maximize comparability by keeping the content of explanations as similar as possible across both cue types, an optimal recommendation may be to combine multiple cue formats (e.g., textual and graphic). Previous studies implied that such a combination may yield better results in user understanding than just relying on one format. ^{60 , 61} Therefore, future studies should include an experimental condition in which the different cue types complement each other.

Fourth, as previously discussed, the results point to different factors that can be relevant for textual and graphic cue design, such as the cues’ complexity. ¹² Future research should examine those and search for mechanisms that make some transparency cues more effective than others by implementing, for example, more gradual variations in cue designs (e.g., less complex to complex graphs, texts with neutral wording to evaluative content).