The Influence of Computer-Related Attributions on System Evaluation in Usability Testing

1 Introduction and Related Work

Usability testing is often complemented with standardized questionnaires measuring different usability criteria, e.g. effectiveness, efficiency, controllability, learnability, and the like. Measuring personality traits and other user characteristics is less common. However, prior research shows that users’ personality traits, namely the ‘Big Five’ personality traits of openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism, influence the results of usability tests as well as drop-out rates [9]. In this work, we present a novel approach of including user characteristics in the overall picture of usability testing by investigating the relationship between computer-related attributions and users’ system evaluations. The goal of our research is to understand better how user characteristics influence usability and user experience and, in the end, design systems that fit their users’ need better.

Attribution Theory is rooted in Social Psychology and describes a well-known everyday phenomenon: Individuals perceive their environment and their own behavior according to typical patterns and strive for general and consistent explanations for external events. Therefore, attributions describe individual and subjective causal explanations [3] and cognitions regarding the extent of control people perceive to have over external events [19].

A distinctive aspect of attribution is Locus, i.e. whether people perceive that certain events are caused internally (i.e. by one’s own actions) or externally (i.e. by other people or influences) [5]. Besides locus, three further attributional dimensions have been described: Stability, Controllability, and Globality.

Stability differentiates between temporally stable and instable causes. Temporally stable means that causes are of a permanent nature (such as knowledge, intelligence, abilities) – for example, attributing successful computer usage to one’s solid computer knowledge (or attributing computer problems to a lack thereof). Instable causes are more singular or unique (e.g. a momentary power breakdown) and less likely to occur again [1].

Controllability refers to the amount of control a person feels he or she has in a certain situation (e.g., the aforementioned power breakdown would likely be perceived as an uncontrollable cause, whereas misspelling a search term might rather be judged as a controllable cause) [18].

Globality describes whether a certain cause is perceived to have similar effects in a wide range of different situations (i.e. aspects which are believed to affect one’s computer use in general) or to apply only to a specific situation (e.g., when a user experiences issues with a certain application but does not generalize this to computer use in general) [14].

Attributions have been shown to influence emotions, motivation, as well as behavior in a decisive way [18]. For example, attributing success internally to one’s own abilities may spark positive emotions such as satisfaction, pride, and confidence. On the other hand, internal attribution of failure may induce resignation or feelings of guilt, shame, or even depression [1].

Regarding Human-Computer Interaction, prior research has identified several specific computer-related attribution styles, showing some notable differences regarding general computer-related attitudes and behaviors (e.g. [10]). Furthermore, there is research investigating gender differences regarding computer-related attributions. Some authors found that girls tend to explain successful computer use with external factors (simple tasks, being lucky) while blaming failures on their lack of competencies. Contrary, boys tend to attribute success to their own skills and failures to external circumstances (e.g. bad usability) (e.g. [2, 8, 16]). However, detailed analyses by Niels et al. [12] indicate that women and men have essentially the same attribution styles. Unlike prior studies, they did not find women’s attribution styles to be less favorable than men’s.

So far, there is little research how different attribution styles influence system evaluations and user experience. For example, it is feasible to assume that people experiencing different levels of controllability (as an attributional dimension) might evaluate the controllability of a computer application (as a system quality defined by e.g. ISO 9241-110 [6]) differently, irrespective of the system design. Likewise, users with internal or external locus of causality, respectively, might evaluate effectiveness and efficiency of a system differently. Furthermore, users with predominantly stable and global attribution patterns might generalize prior use experiences to current use situations, affecting their system evaluation.

The results of a first study by Niels et al. [13] indicate that indeed attribution patterns influence systems evaluations. However, their sample size was relatively small and some of their findings were inconsistent. The current study is designed to replicate, confirm and deepen these findings using a larger sample. Furthermore, we aim to investigate possible gender differences.

The goal of our research is to provide empirical results regarding the relationship between users’ attributions and their system evaluations to provide a basis for future research and practical recommendations for conducting usability tests.

2 Research Methods

3 Results

4 Discussion

The results of our study indicate that users’ attributional patterns do indeed influence their system evaluations, as was first reported by Niels et al. [13]. Especially regarding situations of success we found notable correlations between attributional dimensions and the attractiveness of the system as well as its goal-directed usability: Users who attribute their success externally to characteristics of the interactive system, who feel more in control when using the system, who perceive causes of their success to be temporally stable and likely to recur in other situations assess the respective application more positively.

It is not surprising that external locus of causality is associated with positive system evaluations: Users who attribute their success to the system design rather than their own skills consequently rate the qualities of the system – which in their view are responsible for successful use – better.

The most considerable correlations were found between Controllability and all goal-directed usability criteria as well as Attractiveness, meaning that feeling in control when using computers is associated with better evaluations of system quality. This is a very interesting finding as ‘controllability’ is typically defined as an objective system quality, e.g. in ISO 9241-110 [6]. The results of our study suggest that how users perceive controllability of an interactive system does not only depend on the system design, but also on their own attributional patterns, which can be seen as persistent personality traits (e.g. [1, 17]).

Similarly, there are high correlations regarding the attributional dimension of Stability. That is if users perceive their positive experience of use as temporally stable and persistent – and thus expect to be successful in future interactions again – their system evaluation turns out more positive as well. This could be interpreted as a kind of general confidence, which eases system use and also emanates to a positive rating of the application. Likewise, this applies to Globality – i.e. the extent to which users believe that the cause of their success will take effect in other situations of computer use as well, e.g. when using a different application – albeit correlations are lower here.

It can be noticed that at large attributional patterns which have been identified as beneficial for overall successful computer use – especially high levels of controllability (e.g. [10, 16]) – are associated with more positive system evaluations. This is also true for situations of failure, albeit less and also lower correlations were observed here: Low levels of control – indicating an unfavorable, insecure-resigned attribution style [10] – are associated with more negative system evaluations. In other words: Users with more positive, favorable attribution styles – such as the ‘confident’ or ‘realistic’ types described by [10] – also evaluate applications they use better than users with unfavorable attribution patterns, such as the ‘resigned’ or ‘humble’ types [10]. Of course, it has to be noted that correlational analyses cannot be interpreted in a causal way. However, as attribution styles are seen as persistent personal characteristics (e.g. [1, 17]), it is reasonable to assume that attribution patterns influence system evaluations and not the other way around.

In situation of success, attributional dimensions correlate with goal-directed usability criteria and also overall attractiveness, but not with hedonic system qualities such as Novelty and Stimulation. Quite interesting, in situations of failure, both correlations with goal-directed and hedonic UEQ scales were observed. A possible explanation is that problems using computers – resulting in situations of failure – directly impair user experience as a whole, while success situations might be associated with better usability assessments, but do not automatically improve hedonic aspects of system use.

It should be noted that the relationships between attributions and system evaluations turned out differently for male and female users. For women, we found more significant correlations between the AQ and UEQ scales in situations of success, while in situations of failure this was the case for the male sample, even though there were no general differences between men and women regarding their attributional patterns or system evaluations as such. This indicates that for men, special notice should be paid to possible effects of attribution patterns on system evaluations when a lot of problems occur during the tests. Likewise, for women this effect might be especially strong when they succeed easily in the given tasks. However, this finding needs to be replicated and extended in further research.

To sum up, our study confirms that users’ attributional patterns might have a significant effect on their system evaluations: That means, system evaluations in usability tests probably do not only reflect ‘objective’ system qualities, but also users’ personal characteristics. The same system might be rated significantly different by users with distinct attribution styles. Neglecting this variable in usability tests might lead to misinterpretations and eventually inappropriate design decisions.

Therefore, as a practical implication we recommend including users’ attributional patterns as a variable in usability evaluations. As was shown in this study, attributions can be measured easily by using a short standardized questionnaire. In doing so, the effects of different attribution patterns on usability ratings can be measured and considered (e.g. overly positive or negative evaluations). Likewise, attribution styles could be included in design processes right away, for example using personas with typical attribution patterns (cf. [11]).

Compared to the study by Niels et al. [13] we succeeded in recruiting a larger and more heterogeneous sample, which increases the generalizability of results. Nevertheless, our sample still mainly consists of younger, well-educated, experienced and skilled computer users. Furthermore, the test systems were mainly rated positively. This might provide an explanation for the less pronounced results regarding situations of failures – most test persons obviously did not experience major problems in the test situation.

Another limitation is the somewhat artificial test situation. Using a standardized test procedure and test environment improves the comparability of the tests. As a drawback, however, users interacted with systems they don’t normally use on an everyday basis, so that solving the tasks was probably not overly important to them. We suppose that attributions have an even larger impact in situations of computer use when personal involvement, motivation, and interest for use are high.

Therefore, in future studies, we aim to include even more diverse user groups, especially regarding age and computer skills. Furthermore, the investigation should include test applications that are of more interest to the participants and also include more difficult tasks in order to investigate situations of failures better.