Time related changes of affective dimensions and distinct emotions in the interaction with a tablet PC

2.1 Affective dimensions and discrete emotions

Emotional episodes accompany the human experience. They help us evaluate internal and external stimuli to facilitate an adequate and prompt reaction [20, 21]. Emotional episodes can be described either by basic affective dimensions or distinct emotions [22]. Affect is permanently present in daily life and changes over time. Affective dimensions are an “…integral bled of feelings of valence (ranging from pleasure to displeasure) and arousal (ranging from active to passive), which are considered to reflect the most basic dimensions characterizing emotional feelings.” [23, p. 1]. In this study, we understand an affect as being a simple, non-reflective, but consciously accessible feeling that does not need to be related to a specific object [24].

In contrast, discrete emotions are a set of separate emotional states like anger, joy, fear, or grief [25], [26], [27]. They have unique facial actions that may be universal in different cultures [28], distinct psychophysiological activation patterns [29], and their evolutionary purpose [27]. Often, fundamental discrete emotions are considered primary or basic emotions, implying that they exist naturally, by birth [25]. In the present study, we understand emotional episodes as being adaptive reaction systems that are connected to objects, have a beginning and end, and that find expression in psychophysiological responses, motor (facial) expressions, cognitive appraisal, and action tendencies [21, 30].

The literature is generally spare on findings regarding the decay of induced affect and emotions. For example, a study by Horner et al. [31] investigated the difference between adults suffering from depressive disorder and a control group concerning the effectiveness of inducing positive affect. Their results provide evidence for a short time range of several minutes for the decay of general affective states. For distinct emotions [32] used retrospective evaluation of past experiences to determine the duration of different emotions and found a great range between short-term emotions of several minutes like disgust, shame or fear and, for example, sadness which could last for several days.

2.2 Affective dynamics in human-computer interaction

Researchers of various domains have analyzed participants’ affects for many years. Yet, the notion of dynamic affects is found more in theoretical debates than in empirical research [33]. Affective dynamics are studied based on “[…] global trajectories (the ebb and flow of affect across situations) or distinct emotional episodes (patterns of responses within a given situation).” [34, p. 132]. Verduyn et al. [17] showed that affective activation could result in different dynamic patterns for persons and distinct emotions. Kuppens et al. [35] described two kinds of affective change, pulse and spin. Pulse means that the affective intensity changes from neutral states to extreme ones. Spin subsumes the shift of the affective quality in terms of distinct emotions (e.g., from joy to anger). Waugh, Shing, and Avery [12] reviewed studies that examined the neural bases for affective dynamics. Their core findings revealed that some appraisal processes like valence are fast and directly follow the affective onset. Other appraisal processes like arousal have longer latencies. Complex distinct emotions can last for long periods ranging from seconds to hours or longer.

In HCI research, much of the work on affect and emotions have been subsumed under the term user experience. Forlizzi and Batterbee [36] coined the term “scalability of experiences” (p. 264) to describe that a large number of affective responses that relate to different contexts and actions pile up to larger and more complex user experiences. The evaluation of dynamic changes in these smaller sub-episodes is essential to understand the dynamics of peoples’ experience of interactive technologies. Dynamic changes of peoples’ (affective) experience have been examined on time scales of weeks. For example, Karapanos et al. [14] analyzed participants’ user experience narratives during the first four weeks of ownership of a new mobile phone. They found that the participants’ user experience followed three temporal patterns: orientation, incorporation, and identification. Stimulation, learnability, and usability were important motivators for the transition from orientation to the identification. Positive affective attachment affects the identification with the mobile phone. Sonderegger et al. [16] varied the usability and the design aesthetics of a mobile phone, assessing participants’ emotional responses over two weeks. They found higher valence ratings for the mobile phone with higher usability than the less usable phone. However, this effect disappeared within the first week of usage. Kujala and Miron-Shatz [15] analyzed emotions during phone use over five months. They report that emotions and remembering them impacts the users’ general appraisal of their smartphone. They found a two-fold relationship among emotions and product evaluation: positive emotions showed relationships with positive user experience ratings, whereas negative emotions correlated with low usability ratings [15].

Fewer studies examined the temporal changes of affect within a single human-computer-interaction episode. For example, Backhaus and Brandenburg [37] manipulated affect with people interacting with a gaming console. They found that participants negative affect lasted longer in the following interaction with a tablet PC than their positive affect [37]. These authors concluded that even small changes in the study setting, like the participants’ location or artifact of operation, influence their affective states, which is in line with the results of other research [33, 36, 38, 39]. Also, discrete emotions are highly context dependent. They unfold actions that “[…] serve contextually defined goals.” [34, p. 132]. Context is essential for affective activation over time, and emotions adapt to the context to mediate between environment and person. The current study examines how long induced affective states of different quality like affect and emotions last when peoples affect changes when the study setting is kept constant. Previous work has shown that peoples’ affect changes to a neutral state within 1–2 min when important interaction characteristics like the device or the location of the participant change [40]. This study examines whether affects last longer when the interaction characteristics remain constant. In addition, it firstly assesses temporal changes of affective dimensions and distinct emotions in a single HCI interaction episode, which has not been achieved yet. Discrete emotions may affect peoples’ ratings and their behavior longer than 1–2 min because of physiological latencies and effects on multiple dimensions like cognition and behavior.

2.3 Impact of induced affect and emotions on participants’ HCI performance

Affective states of different quality like affect and emotions do not only lead to subjective feelings, behavioral responses, and changes in the autonomous nervous system [30] but also actively influence information processing like memory and decision making [41]. Also, there is a link between peoples’ emotions and their behavior that varies across contexts and depends on the situational demands [42]. For example, De Longis and Alessandri [43] examined the moderating role of emotional inertia (the extent that emotional states carry on over time) between emotional states and job performance. They found that positive emotions together with low inertia increased performance, while high inertia decreased self-reported job performance. Negative emotions had no significant effect on performance, independent of the analyzed inertia levels [43]. Hockey [44] argued that strain (a combination of anxiety, high negative affect, fatigue, and low positive affect) might even protect performance through the allocation of further resources (at the cost of subjective effort). In general, negative emotions like frustration, anger and sadness help to focus the attentional resources and perception on the objects or thoughts that led to the negative affective states [45] to withdraw or overcome their impacts [20].

In the domain of human-computer-interaction Brandenburg and Backhaus [46] did not find effects of the participants’ affect and emotions on their interaction with a tablet PC. In their study, the authors changed the participants’ location and interaction device (from game console to tablet PC) from affect induction to interaction, which can be understood as contextual change [40]. In another study Brandenburg and Backhaus [13] again found no effects of participants affect and emotions on the participants performance in an interaction with a tablet PC. Now, the authors presented film clips for affect induction on the same tablet PC that was used to assess the participants performance in standard tasks (no change in context). However, in a post hoc analysis, they included covariates like the participants familiarity with tablet PCs. Now they found that participants with a positive affect needed more clicks by tendency for changing the font for the notes app. However, film-induced affect only lasted for about 2–3 min and only impacted the participants’ performance in the first task [13]. Positive affect always led to longer total task times and more clicks than negative affect on a descriptive level only [13].

Positive affective states have other functions. Following the broaden-and-build theory [47], they broaden the range of thoughts and actions. In the long run, thinking outside the box leads to acquiring new skills and scopes of action. Djamasbi and Strong [48] showed that positive affective states foster the intuitive use of technological artifacts and the users’ intentions to use a computerized decision aid. Merritt [49] analyzed the effect of positive and negative affective states on the human-automation interaction. She found that affective states elicited by film clips impacted trust in automation and the liking of interactions with it, even though the affective states were unrelated to the automation. Happiness was positively related to trust in automation and liking of the artifact, mediating the effect of positive affect on automation reliance. Aula and Surakka [50] showed that task completion times in mathematical problem solving on the computer were shorter after positive feedback than negative feedback.

In this work, we assume that participants’ affect and emotions influence their interaction behavior.

2.4 Objectives

To examine the changes of peoples’ affect and emotions and their impact on in their interaction with a tablet PC, we first experimentally manipulate participants’ affective state via positive and negative film clips in the affect induction phase of the study. Concerning this manipulation, positive film clips should lead to higher valence ratings of affect, negative film clips to lower. We expect both conditions to induce similarly increased arousal states, because previous research by Backhaus and Brandenburg [13], that used the same stimuli, provided evidence for an increase in arousal in both conditions. Also, positive film clips should lead to higher ratings of positive emotions and lower ratings of negative emotions. Negative film clips should promote ratings of negative emotions and lower ratings of positive emotions.

Based on this manipulation, we examine two objectives. First, we assess the dynamics of participants’ ratings on affective dimensions (H1a) and distinct emotions (H1b) over time. Participants complete four standard tasks on a tablet PC like changing the visualization of the battery status in the interaction phase of the study. The corresponding hypothesis H1 addresses the different time intervals in which participants’ affective response changes during the interaction with the tablet PC.

H1 (Temporal dynamics of participants’ affective dimensions and distinct emotions):

1. H1a (Affective dimensions): The valence ratings of the participants revert to baseline values within the time they need to finish the first task [13].

2. H1b (Distinct emotions): Participants’ ratings of positive and negative distinct emotions change during processing the four tasks with the tablet PC.

Second, we evaluate whether participants’ ratings on affective dimensions and discrete emotions are related to their task-performance in an interaction episode (H2). Therefore, we assess the participant’s performance (time to first click, TFC; total task time, TTT, and number of entries) when completing the four standard tasks on the tablet PC. The second hypothesis addresses the effects of their affective state on these performance measures.

H2 (Effect of participants’ affective involvement on their performance): The emotional involvement of the participants affects their performance in the interaction phase differently. A negative affective state will lead to better performance like lower TFC and TTT, and a higher number of entries than a positive affective state.

3.1 Participants

Sixty-four people (49% female, and 79% students) participated in the experiment. Their mean age was M = 25.68 years (SD ± 5.77 years) and ranged from 18 to 55. Almost all (96%) owned at least one touch device like smartphones or tablet computers. In addition, 62% stated that they had prior experience with the type of tablet PC of this study. The participants were recruited via direct contact and a web portal for participants. They participated voluntarily and received no incentives. The study was approved by the local ethics committee and conducted under the declaration of Helsinki.

3.2 Questionnaires

We used the Affect Grid to assess changes in participants’ general affective dimensions valence and arousal [51]. A 9-point semantic differential represents each dimension. These axes are arranged perpendicular to one another. The horizontal axis stands for valence increasing from left (extremely unpleasant) to the right (extremely pleasant). The vertical axis represents arousal increasing from extreme sleepiness up to extremely high arousal. Neutral affective states are at the center of the matrix with the coordinates 5, 5. The Affect Grid allows participants to assess both dimensions by putting one single mark in the grid. Figure 1 visualizes the Affect Grid. To prevent participants from developing a response bias, we gave the instruction to base their ratings on their current affective status. Each time, the meaning of the x- and y-axis was explained, and it was said that they can indicate a neutral state with a cross in the middle of the grid.

Figure 1:

The affect grid (adapted from [51]).

We used the modified Differential Emotions Scale (mDES) to assess the ratings of 20 distinct emotions and create independent positive and negative emotion scores [47]. Three affective attributes characterized each of the ten positive and negative emotions. Amusement, for example, is measured by the item “What is the most amused, fun-loving, or silly you felt?”. Participants answered each item on a five-point scale ranging from 0 (not at all) to 4 (extremely) to indicate how they felt during the past time interval. Time intervals were instructed as follows: “Please consider how you felt in the last two hours.” (baseline, T0), “Please consider how you felt during watching the film clips” (manipulation check, T1) and “Pleases consider how you felt during the interaction with the Tablet PC” (end of study, T2).

3.3 Apparatus

We used an iPad^TM (first generation, 3G, factory settings, “9.7” multi touch screen) for showing the film clips and administering the four standard tasks. The tablet PC that laid on a table during the study.

3.4 Affect elicitation via film clips

Researchers have used many affect elicitation procedures to manipulate a participant’s affective state [52]. For example, Bradley, Miccoli, Escrig, and Lang [53] successfully used positive and negative pictures to elicit the corresponding affect. Mayer, Allen, and Beauregard [54] utilized music to induce happiness, anger, fear, and sadness. Lench, Flores, and Bench [55] summarize various affect elicitation methods in their meta-analysis. These authors describe film clips as a reliable method for affect elicitation. Benefits of film clips include their short presentation duration, intuitive understanding, accessibility, and the possibility to standardize them across participants. Rottenberg, Ray, and Gross [39] describe films as capable of eliciting enduring affective reactions that last between 1 and 10 min. Westermann, Spies, Stahl, and Hesse [56] found that film was one of the most effective procedures to induce positive and negative affect. The effect was more extensive when participants were not psychology students, knew the study goals, and participated individually and not in a group. Koelstra and colleagues also used short clips of music videos to successfully manipulate the affect of their participants [57]. In a qualitative review, Siedlecka and Denson [58] found that visual methods like film clips are amongst the most effective induction methods for emotions like anger, sadness, or happiness [58]. Brandenburg and Backhaus [13] updated a standardized set of six film clips that was established by Gross and Levenson [59] and used by Hewig et al. [60]. In their study, Brandenburg and Backhaus [13] added seven new film clips to the existing six film clips and showed this set of 13 clips to 14 participants. Then, the participants rated their valence and arousal while watching each of the film clips with the Affect Grid. A cluster analysis revealed two clusters, each containing the three film clips eliciting the strongest positive (cluster 1) and the strongest negative (cluster 2) affect [13]. The current study uses these film clips of [13] to manipulate participants’ affective states and distinct emotions. Here, participants watched the three film clips of about 3 min length that either induced positive or negative affect. Sections of Bridget Jones II, Pursuit of happiness, and Babies were used to elicit positive affective states. Snippets of the same length from Gladiator, I am Legend, and Wall-E were utilized to foster negative affective states. Since we used validated film clips from well-known movies, we asked participants for each movie whether they have seen it. Their answers to these questions revealed that the proportion of participants that had seen one of the positive movies ranged from 10% (Pursuit of happiness) and 20% (Bridget Jones II) to 58% (Babies). For negative movies, the proportion of participants who have seen one of them ranged from 30% (Wall-E) to 61% (Gladiator and I am Legend). To accommodate for the participants’ variation in familiarness with the movies, we presented either the three positive or the three negative film clips one after another and in balanced order to the participants. Thus, each participant rated his or her affect and emotion only once before and once after him or her seeing all three film clips.

3.5 Procedure

Initially, participants completed the demographic questionnaire and read the participants instruction sheet. Here, the procedure of the study was briefly described, and participants were instructed to answer all questions based on their internal state. There were no right or wrong answers. Then they filled in the Affect Grid and the mDES, with the last two instruments assessing the baseline ratings (T0) of their affective state. They were then randomly assigned to one of the two affect-induction groups (n₁ = 31 in the positive and n₂ = 33 in the negative affective states group) and placed on a seat directly in front of the tablet PC. Each participant consecutively saw the three film clips that should elicit either positive or negative affect [13]. Film clips were order-balanced over participants of each affect group. Taken together, each participant watched about 10 min of either positive or negative film clips. After watching the film clips, participants again filled in the Affect Grid and the mDES. This measurement (T1) served as a manipulation check and closed the affect induction phase of the experiment.

Immediately after the induction phase, participants executed four tasks with the tablet PC in the interaction phase of the experiment: (1) change the font for the notes app, (2) change the visualization for the battery capacity from symbol to percentage, (3) transfer as many contacts from a given list to the address book as possible in 90 s and (4) type in as many activities from a given list in the calendar in 90 s. Participants received a paper list with 13 contacts containing first names and mobile phone numbers for the address book task. We used another list of 13 activities for the calendar task. Each entry briefly described the activity like cooking with friends and a place where the activity takes place. Participants should enter as many contacts (task 3) or activities (task 4) into the tablet PC as possible in 90 s. Each of the four tasks started and finished at the home screen of the tablet PC. Each time, participants said “start” at the beginning and pressed the home button after completion. Tasks were arranged into two blocks of problem-solving tasks (tasks 1 and 2) and performance tasks (tasks 3 and 4). The order of tasks was random within the two blocks, and blocks were balanced across participants. This procedure ensured that possible unwanted effects on the participants’ affect and emotions like time pressure had no systematic impact on their ratings. Their instruction was to complete all tasks as quickly and as correctly as possible. Also, participants should read out loud each task. The experimenter assessed the time with a digital stopwatch the participants needed from giving the verbal start command to the first tap on the screen of the tablet PC (Time to First Click; TFC), the task completion times (Total Task Times, TTT), and the number of entries for the third and the fourth task. After the completion of each task, participants filled in the Affect Grid (T2a–T2d). Participants filled in another mDES after accomplishing the last task (T2d) and filling in the last Affect Grid. Figure 2 illustrates the procedure. The whole experiment lasted for about 45 min.

Figure 2:

Summary of the procedure, AG – affect grid, mDES – modified differential of emotion scale, pos. – positive, neg. – negative.

3.6 Experimental design

First, we manipulated the allocation of participants to the film group (positive or negative) as a between-subjects variable to elicit the congruent affective states (positive or negative affect). Second, we introduced a within-subjects factor with six steps, i.e., points of measurement. Dependent measures included the Affect Grid (valence and arousal) that was assessed six times (T0, T1, T2a–T2d, see Figure 2), the modified Differential Emotion Scale (mDES) that was completed three times (T0, T1, T2d, see Figure 2), and the performance measures time to first click (TFC) in seconds for the four tasks. Total task time (TTT) was assessed for the tasks one and two. The number of entries that participants could make in 90 s was a performance measure for the tasks three and four.

3.7 Analysis

We used dependent t-tests to analyze the effects of watching film clips on participants’ affect and distinct emotions. We report Cohen’s d as a measure of effect size. Effects of 0.20 < d ≤ 0.50 are regarded as small, 0.50 < d ≤ 0.80 medium, and d > 0.80 as large [61]. Mixed (film group x points of measurement) (M)ANOVAs were computed to evaluate the temporal changes of participants’ affect and distinct emotions (hypothesis 1a, b). Finally, we analyzed the effect of participants’ group membership on their performance using multiple ANCOVAs (hypothesis 2). Here, we report η²_part as a measure of effect size. Effects of 0.01 < η²_part ≤ 0.08 are regarded as small, 0.08 < η²_part ≤ 0.14 medium, and η²_part > 0.14 as large.

4.1 Manipulation check: the effect of film clips on participants’ affect and distinct emotions

4.1.1 Affect (Affect Grid)

We computed dependent group t-tests (T1–T0) for the arousal and valence ratings of each group (positive and negative film clips) to analyze the effects of watching short film clips on participants’ affect. We did not obtain any effects of watching the film clips on participants’ arousal, positive group: t(30) = 0.58, p = 0.56, 95% CI of difference −0.39 to 0.72; negative group: t(31) = −0.09, p = 0.92, 95% CI of difference −0.71 to 0.65. However, watching negative film clips lowered participants valence ratings from M = 6.62 (SD = 1.16) to M = 5.15 (SD = 1.87), t(31) = −4.85, p < 0.001, d = 0.89, 95% CI of difference −1.95 to −0.79. Positive film clips, in contrast, did not change participants’ affect, t(30) = 0.12, p = 0.89, 95% CI of difference −0.48 to 0.54. Figure 3 depicts these results for participants’ arousal (Figure 3a) and valence ratings (Figure 3b).

Figure 3:

Affect grid group means for (a) participants’ arousal and (b) participants’ valence ratings, error bars indicate 95% confidence intervals for group means, T0 is before and T1 is after the affect induction. Positive/negative indicates group membership.

4.1.2 Emotions (mDES)

To examine the change of the participants’ emotions, we computed dependent groups t-tests (T1–T0) for the mean of the ten positive and negative mDES ratings of each affect induction group (positive and negative film clips) according to [47]. As expected, watching negative film clips decreased participants ratings of positive distinct emotions from M = 1.85 (SD = 0.62) to M = 1.22 (SD = 0.75), t(31) = 5.84, p < 0.001, d = 1.03, 95% CI of difference 0.40–0.84. Also, negative film clips increased participants ratings of negative distinct emotions from M = 0.41 (SD = 0.30) to M = 0.53 (SD = 0.35), t(31) = −2.08, p = 0.04, d = 0.35, 95% CI of difference −0.22 to 0.002. Positive film clips, in contrast, did not change participants’ ratings of positive (t(30) = −0.31, p = 0.75, d = 0.07, 95% CI of difference −0.26 to 0.19) or negative emotions (t(31) = 0.41, p = 0.68, d = 0.07, 95% CI of difference −0.17 to 0.26). Figure 4 depicts these results for the positive (Figure 4a) and the negative emotions (Figure 4b).

Figure 4:

Change of participants’ ratings of distinct positive and negative emotion scores (mDES), error bars indicate 95% confidence intervals for group means, T0 is before the affect induction, T1 is after the affect induction, positive/negative indicates group membership/film clips.

4.2 The temporal dynamics of participants’ affective dimensions and distinct emotions

4.2.1 Affective dimensions (H1a), affect grid

We analyzed the temporal dynamics of the participants’ affect in the interaction phase of the experiment with a 2 × 5 (film group × point of measurement) ANOVA. Including five points of measurement (T1–T2d) allows us to analyze the changes of participants’ ratings of affective dimensions from the beginning to the end of each of the four tasks. The results revealed a main effect of point of measurement on participants’ arousal ratings, F(2, 248) = 10.92, p < 0.001, η²_part = 0.15, indicating that their arousal level changed over time. Bonferroni corrected post hoc tests showed that participants rated their arousal to be significantly lower (all p ≤ 0.007) after watching the film clips (T1, M = 5.45, SD = 1.68) than their arousal after the execution of each of the tasks (T2a–d), which was M = 6.22 (SD = 1.37) at minimum. There was a tendency for positive film clips to provoke a higher arousal rating (M = 6.41, SD = 1.61) at all points of measurement compared to negative film clips (M = 5.90, SD = 1.56) although this effect was not statistically significant, F(1,62) = 3.02, p = 0.07, η²_part = 0.04.

The ANOVA also revealed an interaction effect of film group and point of measurement for participants’ valence ratings, F(4, 248) = 4.57, p < 0.001, η²_part = 0.06. Bonferroni-adjusted post hoc tests showed that valence ratings differed significantly between film groups at T1, t(1, 62) = −3.47, p < 0.001, d = 0.87, but not at other points of measurement (T2a–T2d, all p > 0.46). Figure 5a visualizes the increase in participants’ arousal ratings over time. Figure 5b shows that their valence ratings reverted to baseline values after the execution of the first task (T2a in Figure 5b), which corresponds to hypothesis 1a.

Figure 5:

Affect grid group means for (a) participants’ arousal ratings and (b) participants’ valence ratings over time; error bars indicate the 95% confidence intervals. T1 is after the affect induction, T2a–T2d are measurements performed after each tablet PC task; positive/negative indicates group membership.

4.3 Participants’ emotional state and its relation to their interaction performance (H2)

We examined the impacts of participants’ emotional state on TFC, TTT, and the number of entries for tasks three and four by computing univariate ANCOVAs, each with group membership as an independent variable, age, sex, and tablet PC experience as covariates, and the respective performance measure per task as the dependent variable. The analysis revealed that members of the positive group were slower (M = 4.54, SD = 5.61) than participants of the negative group (M = 3.12, SD = 3.62) in their initial reaction (TFC) when changing the font type, F(1, 62) = 4.33, p = 0.04, η²_part = 0.07 (H2). We obtained no other meaningful effects, all p > 0.05.

In addition, we found effects of gender, age, and tablet PC experience on TFC and TTT for the problem-solving tasks. Males were significantly faster (M = 2.30, SD = 1.30) than females (M = 5.29, SD = 6.26) in TFC for changing the font of the notes app, F(1, 61) = 13.98, p < 0.001, η²_part = 0.20. They were also faster (M_male = 43.78, SD_male = 31.06; M_female = 61.90, SD_female = 32.84) in TTT for this task, F(1, 61) = 14.49, p < 0.001, η²_part = 0.20. Moreover, males were faster (M = 2.28, SD = 2.61) than females (M = 4.62, SD = 5.54) in TFC for changing the battery options, F(1, 61) = 4.58, p = 0.03, η²_part = 0.07.

Also, age showed positive relationships with participants’ TFC for changing the font of the notes app (r = 0.40, p < 0.001) and for manipulating the battery options (r = 0.37, p = 0.002). Age was negatively related to the number of contacts that participants entered the tablet PC (r = −0.28, p = 0.02). Another negative relationship of age was obtained for the number of activities that participants entered the tablet PC, r = −0.31 (p = 0.01). Finally, tablet PC experience correlated negatively with participants’ TFC when changing the battery options (r = −0.32, p = 0.01). Tablet PC experience was also negatively related to TTT for changing the font of the notes app (r = −0.57, p < 0.001) and for the variation of the battery options, r = −0.36, p = 0.003.

5.1 Temporal dynamics of the participants’ affect and distinct emotions

When testing our hypothesis H1a, we found that valence ratings quickly reverted to baseline values (during the execution of the first task) when changing from watching film clips to interacting with the tablet PC. This finding supports the assumption that affects are remarkably dynamic, volatile, and highly context-dependent [30]. This finding is in line with Brandenburg and Backhaus [13], who found a similar pattern of valence ratings over time. These authors attributed the disappearance of high affect ratings to the variations in location, task, and artifact of operation from affect induction to interaction. The current study maintained the study settings except for the tasks and found the same results. Affective dimensions, therefore, seem to be highly susceptible to characteristics of the study setting [13, 46], the participants’ task in this case. In line with this interpretation, a recent study by McGuckian and Pepping demonstrated that the participants induced positive affective states (via music) remained high for about 40 min during a football training session [61]. In contrast, their induced negative affective states returned to pre-training levels at the beginning of the training session. This shows, that affects may be stable when the task (the football training) further supports the (positive) affect after the affect induction.

The findings of the present study undermine the claims of theoretical models highlighting the interaction characteristics as primary determinants of user experience, including affective experiences [6, 62, 63]. Affect that exist before the interaction with an artifact may not be very important for the affect that results from it. Participants may relate their affect during the interaction to the interaction characteristics. This view does not challenge models that assume emotional priming through the user’s affective state before the interaction. For example, Pohlmeyer and colleagues call pre-interaction positive emotions like hope or negative emotions like fear anticipated user experience that determines the likelihood of using a technical system [64]. Our results do not challenge this claim because Pohlmeyer and colleagues consider product-related emotions. These product-related emotions evolve from the users’ expectations regarding the upcoming interaction experience. The manipulation of affect in our experiment should have been unrelated to the artifact of interaction (the tablet PC). This type of unrelated affective stimulation does not seem to impact the participants’ affect in an interactive situation. This conclusion is essential for advertising campaigns that utilize peoples’ affective states to increase usage intention of technical artifacts. These campaigns often use affective stimuli to promote their products. Positive affect unrelated to the artifact might help to increase the likelihood of buying or using technical systems. However, this affect is not likely to affect the users’ emotional experience during the subsequent interaction.

The findings of the study are different on the level of distinct emotions. Results indicate that participants’ ratings of few emotions changed over time. This finding only partially supports our hypothesis 1b. Watching negative film clips decreased the participants ratings of the positive emotions hope, joy, and serenity, and increased their ratings of sadness only. This result indicates that negative stimuli may lead to changes in general affect, but these changes may be due to decreases in few positive emotions only. During the interaction phase of the study, again positive emotions were rated to be lower than after the induction. In the positive group ratings of amusement, love, and serenity decreased during the interaction. Ratings of sadness decreased as well. This finding may be due to participants entering the experiment in a quite positive affective state. The film clips may have maintained this positive state until the end of the induction phase. Here, executing standard tasks on a tablet PC may have led to interaction characteristics that, in turn, lowered participants ratings of these emotions. More specifically, distinct positive emotions were rated lower throughout the execution of the four tasks in the positive group, possibly because of the tasks’ characteristics. Entering contacts in a contact list, changing the font of an app, and manipulating the battery symbol are tasks that are not associated with positive emotions. Thus, positive emotions in HCI may need constant positive input upholding their level. We did not find similar effects for the negative emotions. For system designers, this underlines the necessity to create interactive experiences supporting a positive attitude towards the artifact, for example by utilizing animations conveying a friendly impression [65]. In addition, the results of the present study concerning the changes of emotion may be informative for researchers in the affective computing domain helping them to focus emotions that are changing during a human computer interaction episode. They could use this information to apply more sophisticated models of multimodal emotion analysis [66].

The non-overlap of temporal changes in participants’ ratings of affective dimensions and distinct emotions may relate to conceptual differences. Russel [24] subsumes hedonic (valence) and arousal values under the term core affect, which denotes a non-reflective and free-floating affective state without object direction. However, when core affects are attributed to an object like the film clips in this study, this attribution process may be the beginning of an emotional episode. The participants’ valence ratings may have reverted to neutral after the induction phase because they might have attributed their changes in unspecific affect to task characteristics. This may have led to an emotional episode, which is sharply defined as having causal connections and temporal order [24]. Thus, the film clips and the tasks may have been emotional objects that may have changed participants’ attributions of affective states. This finding may indicate that HCI research and design should focus more on the assessment of distinct emotions instead of general affective dimensions. Especially positive emotions may reflect changes in peoples’ experiences better than negative emotions and general affect.

5.2 The effect of the emotional activation on participants’ interaction performance

Hypothesis two assumed an effect of positive and negative affective states on participants’ performance in an HCI episode. Results showed that participants from the positive affect group were slower in one task than participants from the negative affect group. This finding only partially supports our second hypothesis. Missing effect of participants’ affect on performance may be related to the affect manipulation in the induction phase of the study. Here, the affect induction did only partially work out as expected. Future studies could implement stronger or more individualized affect induction methods like stories [56].

However, the small effect of the participants’ affect on their performance that was observed in this study is in line with previous research highlighting negative affect to be a facilitator of participants’ performance in standard tasks [13]. Droit-Volet et al. [67] showed that fear distorted participants’ time perception in a computer-based laboratory task. Kensinger and Corkin [68] report that negative affective states speed up long-term memory retrieval. They also report that negative affective states do not have a robust effect on short-term memory retrieval.

However, positive affect may promote participants’ performance in other types of tasks. For example, in study 2 of Fredrickson and Brannigan [47], participants were asked to complete twenty “I would like to …” sentences. A positive affective state led to a higher number of sentence completions compared to a negative affective state. In the first study of Johnson et al. [72]; cited from [47], participants with a higher number of Duchenne smiles (positive affect) showed a better performance in processing global targets compared to local targets (different combinations of the letter “T”). In the second study of (Johnson et al. 2010); more positive participants showed greater attentional flexibility in a Posner attentional orienting task. Taken together, one could conclude that the effect of participants’ affective state on their performance depends on whether it is positive or negative and the type of the task. Positive emotional states foster participants’ performance in tasks that require creativity and a broad focus of attention. Negative emotional states seem to facilitate participants’ performance in tasks demanding concentration on few aspects and a narrow focus [47].

Another group of findings of the present study relates to performance differences based on gender and age. Males were faster than females, and participants became slower with increasing age. We cannot generate explanations for these effects based on other results of the present study. Future studies should investigate this issue.

5.3 Conclusions, limitations and future work

The present study again revealed that film clips can manipulate participants’ ratings of affect and emotions, with their valence ratings returning faster to baseline levels than ratings of emotions. This finding aligns with the results of previous studies [39, 55, 56]. In addition, the study’s results indicate that lowering participants affect ratings via negative film clips may rather be due to the decrease in ratings of positive emotions (hope, joy, serenity in this study) and less to the increase of negative emotions like sadness. This finding is new and should be further investigated. Third, participants’ ratings of negative affect promoted their performance in some standard tasks, at least.

This study also has some limitations that may limit the interpretation of its results. First, we did not assume differences in arousal between conditions (positive and negative group) based on the induction of affective states. Using null-hypothesis significance testing methods, like we did, does not inform us about the absence of an effect in the population [69]. In this study, we refrained from calculating more informative statistics like Bayes factors because we wanted to keep focus on the hypothesis testing and not divert attention to the manipulation check.

Second, we assessed participants’ subjective ratings of distinct emotions at three points in time only. We used the German version of Fredricksons [47] 20-item mDES adapted to German by Brandenburg and Backhaus [46]. This questionnaire is well-validated but too long to be answered after each task. Future experiments could use other emotion assessment methods like single item scales or voice recordings as additional measures of affect and emotions over time. Continuous physiological measurements could also substantiate changes in subjective ratings of affective states. Therefore, a shorter affect induction phase and tasks with a longer duration would be necessary. Concerning the induction of positive affect states, a baseline condition could expose participants to a simple neutral stimulus (e.g., a fixation cross) for a couple of seconds to minimize a positive bias towards the study participation (cf. [56]). Third, we used simple and short standard tasks in the interaction phase of the experiment. Subsequent studies could examine the influence of participants’ emotional state on their interaction performance in longer and more complex tasks like word processing or creativity tasks. Also, we examined the time-related effects of participants’ emotional states on their performance in a laboratory setting, which may have downsized the effects. Participants always knew that they were observed, which could have triggered some emotional control process. Upcoming studies should assess temporal changes in people’s affect and emotions in real-life settings. This study focused the internal validity of the results in a laboratory setting. Changes in people’s affective states may be more pronounced in real-life settings than in the laboratory. However, these studies should then account even more for individual differences. They could do this by assessing their participants performance before the study and use this information to compute difference scores with and without affect manipulation. These difference scores would help to evaluate whether the participants’ performance becomes better or worse compared to their standard performance when experiencing positive or negative affect and emotion.

Finally, the present study has examined the change of participants general affect and specific emotions utilizing a quite young and well-educated sample. A more diverse sample may help to examine the effects of interesting variables like regular usage behavior on the participants performance when executing standard tasks with a tablet PC.

Emotional states influence participants’ experiences and behavior differently before and during the use of artifacts. It is a theoretical and methodological issue that the HCI community should keep on addressing in future studies. Affective states are volatile phenomena, transient and fragile. These properties may lead to an overestimation of their influence on peoples’ experience and behavior. However, emotional (appraisal) processes as constant companions are inevitable because they decide on acceptance, use, and evaluation elements of the user of technical products.