New emotional model environment for navigation in a virtual reality

Marcin Daszuta; Dominik Szajerman; Piotr Napieralski

doi:10.1515/phys-2020-0199

Article Open Access

New emotional model environment for navigation in a virtual reality

Marcin Daszuta , Dominik Szajerman and Piotr Napieralski

Published/Copyright: December 2, 2020

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Open Physics Volume 18 Issue 1

Abstract

Emotions are commonly considered to be the most expressive of everyday human experiences, giving a sense of fullness and reality of life. The ability to recognize human emotions as a manifestation of higher intelligence is desirable feature. There are several models of emotional experience that can become the basis for building a universal emotional recognition system. In this article, we check the correctness of the designed emotional model. We check the evaluation of the system’s operation by human observers.

Keywords: emotions; cognitive modeling; intelligent systems; fuzzy approach

1 Introduction

According to the latest psychological knowledge [1], emotional systems are located in older evolutionary regions of the brain and are mostly homologous in all mammals; also in all mammals, the chemistry of these systems is similar. Emotional systems generate instinctive behavioral responses that are closely related to the primary affections that accompany these responses. In principle, all emotional systems are linked to the system responsible for what we call motivational processes in psychology, i.e., the search system. It stands out from the rest by the fact that it takes part in basically every action of the body that is directed toward the goal [2]. By analogy, it can be assumed that the development of emotion models should contribute to a significant improvement in the search of space by artificial intelligence (AI), above all, to an adequate response and correct decision-making. Computer simulations of psychological phenomena have a half-century history. The following research was aimed at introducing an emotional model that will make the AI movement be interpreted as emotional behavior. This approach will allow to check if the emotional model in AI can be correctly interpreted by humans [3]. The application of the developed model can be used in both computer games [4] and robotics. Developed location models [5] can be enriched with this models. Researchers try model some emotions to a robot or a virtual agent. The respective systems can be called computational systems of emotions. In this article, we concerns the issue of recognizing emotions, in which the latest solutions are able to identify and classify emotions based on agents’ movements.

2 Emotion engines and theories

Emotion engines are adaptations of models describing appraising of events and processing temporal emotional state. These are based on three main theories: PAD space, appraisal theory, and PSI schema. Each of those originate from the evaluation of psychological research regarding emotions. Difference between those mainly focus around interpretation of emotional state processing. The main common aspect of all ideas is a division of elements that have influence on the emotional state. Besides events in outer environment, the emotional state depends on the character of a unit and a current mood. Depending on the considered theory, additional aspects of the appraisal process are included, for example, social aspect or cultural aspect [6,8,10,14,17].

2.1 PAD space

PAD space (based on Mehrabian works) assumes emotion to be point in a three-dimensional space based on three aspects. That is why point or emotion, as it can be understood, may be called composition of space parameters, where the space itself is considered in range between −1 and 1 in each dimension. Dimensions or parameters are pleasure, arousal, and dominance. In this case, the emotional state of a unit is activated or changed by events in surrounding and degradation of activation over time. These changes are based on shifts in the PAD space. Each activation causes deviation in relation to the beginning of PAD space and degradation rapprochement to it. Good example is eating a meal. It is an event that has pleasant characteristic and at the same time needs activation in the direction of focusing on processing of taste and experience itself.

Understanding of the emotional state as a point in the PAD space allows for simple mapping results of emotion engines based on other theories. Model using the PAD space theory is Wasabi. This model describes overall architecture of an engine, which explains dynamics of emotional state processing. Figure 1 shows the structure of an engine based on the Wasabi model [10].

Figure 1

Scheme of Wasabi architecture with distinction of components, where Tick is processors tick and Knowledge is understanding if environment characteristics.

Distinctive part of this engine is parametrization of the emotional state. The main assumption of this architecture is, as mentioned earlier, kept and simulated as point in 3D space. IT depends on three degrees-of-freedom called valence, mood, and boredom. It is additional space correlated and placed between PAD emotional state and event processing. It is used to decide on event’s dynamics and its influence on unit’s state. In the next step to describe the emotional state in an unambiguous way for a user, VMB point is mapped to point in the PAD space. Emotion’s value is decided based on the distance of an emotional state from distinct emotions, since each of particular emotions has defined place in the PAD space.

2.2 Appraisal theory

It is the most popularly used theory in models describing the emotional state interpretation and processing. The basis of this approach is assumption, that, emotion depends on a pattern of individual appraisal regarding event assembled from relation among individual thoughts, intentions, and desires, which is known also as human–environment relation. This relation was formalized in the shape of appraising variables, explaining the meaning of an event for a unit. Hence, in this theory, emotions are the result of direct appraisal of an event done by an individual unit. The most used model that uses this idea is OCC model (from authors names, Ortony, Clore, and Collins)

This model characteristics is event appraisal done based on, event’s attributes, origin of event as well as environment in which it happened. Appraisal itself is built on defined criteria in effect of which emotional state could be received. Attributes of an event are bound with relation of deliberating unit with event itself. What could be understood as what consequences an event has to the unit or how the origin of an event is related to the unit. In other words, unit must check if an event was positive or negative and how an environment reacts. Figure 2 describes schema of the OCC analysis [10,12,13].

Figure 2

Scheme of the OCC model explaining deliberative stage of emotional state creation.

Perfect example of practical use can be FAtiMA engine (FearnotAffective Mind Architecture). Its characteristic point is layered design, where each layer considers other aspect of event’s appraisal. The most vital one is the deliberative component, which in fact ensures engagement of the main appraisal process of the incoming data. The result of this appraisal is certain emotional state, which is then taken under consideration during decision-making about further unit’s actions. Emotional state is presented as floating values in scale between 0 and 1. Knowledge module also plays an important role, defining knowledge about environment and explaining relation to observed events. It is described in Figure 3.

Figure 3

Scheme of FATiMA with additional layer distinction.

2.3 PSI theory

PSI theory, which name comes from Greek letter associated with psychology, is based on assumption, that is, emotional state in relation to event depends on the level of needs realization, according to the basis of Maslow’s needs pyramid. To be exact, actions and reactions of considered unit are dictated by motivations, which emerge from need to fulfill needs. Depending on the priority, agent or unit strive to the homeostasis state trying to fulfill each of the mentioned needs. Starting from primary needs, like security, ending on the most sophisticated like social relations. Therefore, from needs motivations emerge, which cause changes in the level of aims realization. Aims with consideration of the emotional state resulting from event appraisal give possibility to decide about further steps of an agent. This theory is used by a model with the same name. Figure 4 describes an engine architecture created based on this model [9].

Figure 4

Scheme of PSI architecture.

2.4 Hybrid approach

There are also hybrid models getting inspiration from multiple theories. Good example can be SIMPLEX. It uses appraisal theory like in OCC model during event appraisal and PAD space to interpret and store the emotional state. It is described in Figure 5 [10].

Figure 5

Scheme of SIMPLEX architecture.

There is also other idea of assembling PSI and appraisal theories and many other, but differences coming from assembling are not very important to distinct them in comparison to the main theories. In addition, those do not have implementation that could be verified and compared with existing solutions.

3 Emotions in navigation

Emotion engines have use in the area of pathfinding. There are several methods of adaptation into existing heuristic solutions as well as into machine learning modules used for adaptive navigation.

Example can be the particle system used in eCology simulation. It is based on two main aspects. First aspect is treating a moving unit or an agent as a particle owning velocity and mass. Through these attributes, the agent can be attracted or repelled by different objects in virtual environment. Second aspect is the use of engine emotion to define relation with objects in the area, in that way appraising if one should be attractor or repel considered agent and with what force. Depending on events in surrounding, like giving food or shouting, agent might avoid one object or approach other. Figure 6 illustrates the idea [15].

Figure 6

Scheme explaining attractor–repeller system [18].

Similar solution can be used as augmentation of pathfinding algorithms based on heuristics. Using emotional state and the source of emotional change trespassing cost is changed near it, and thus adapting direction of path search. Algorithm will find the way with emotional state reasoning.

Another way to use the emotional state in pathfinding are emotional maps. Similar to the example of earlier solution, this one also manipulates trespassing cost through certain areas. However, aspect which distincs this idea from others is storing data consisiting of connection between position and emotional state, in separate data collection. Thus, forming abstract map of relation reffering to emotion states and its bounding with certain areas. It is treated as abstractive layer, in which data are compared and mapped to real graph of path search. In addition, in emotion maps, there is no necessity to connect emotional state if object in environment. Event in certain area causes emotional state change in emotional map, which is also influenced by degradation over time. For instance, whenever something odd happen in one place in environment, like explosion or scream, it gets registered in emotional map as hazardous area with higher trespassing cost. If in near future nothing happens, this effect will decrease over time. Figure 7 presents visualisation of this effect [7].

Figure 7

Scheme explaining degradation process of tresspassing cost [15,18].

There are approaches to bound emotion engine with machine learning aiming to research influence of emotions on learning process of an agent during maze walking through. Emotion engine is a thruster of awarding and penalizing during tries of getting through a maze. In case of finding dead-end or staying in maze for too long, what changes its arousal level and valence in VMB space, which is further mapped to the emotional state imitating fear and frustration. This results with reinforcing learning effect in the direction of earlier trials. Where in opposite case, relief and satisfaction would be acquired with quick passing through the mentioned maze. This way, positive effect reinforces optimization of path through maze [11].

4 Proposed emotion engine

Based on FATiMA engine, the proposal of emotion engine was introduced. It consists of single internal layer (deliberative layer), allowing to do appraisal of an event according to OCC model rules. Although in contradiction to appraisal theory assumption, it is settled that overall emotional state can be reduced to five basic emotions: anger, fear, sorrow, disgust, and joy. Those were settled as basic by Ekman’s work in this field. According to his theory, each emotion is just composition with different levels of feeling of those basic emotions. This way, assuming scale of feeling range between 0 and 1 for each of basic emotions, and emotional state can be defined. Through this assumption, amount of considered emotions is reduced from 22 to 5. It highly simplifies architecture. Another step taken was translating emotions to PAD space according to the PAD space theory. This allowed comparison of emotional state with other solutions.

Besides storing and processing of the emotional state, there is a need to define methods of translating emotion engines results into reactions. The area of pathfinding used correlation between reaction tendencies and emotions based on the study by Shen and Bigsby. It was noticed that that most of psychological theories connected with emotions agree with certain behaviors to originate from emotions (for example, of the studies by Scherer, Lazarus, Frijida, Dillard, Russell, Johnstone, Schorr, and many others). This aspect is presented in Table 1 [16].

Table 1

Listing of relations between emotions and tendencies [16]

Emotion	Relational theme	Valence	Function	Action tendency
Surprise	Novel stimuli	Open	Orient	Allocate attention
Anger	A demeaning offense against me and mine	Negative	Remove obstacle	Attack/reject
Fear	Facing uncertain, existential threat/danger	Negative	Protection	Revise existing plan/create new plan
Sadness	Having experienced an irrevocable loss	Negative	Learning/recuperation	Review plan/convalesce
Guilt	Having transgressed a moral imperative	Negative	Self-sanction	Strive to attain standard
Disgust	Taking in or being too close to an indigestible object or idea	Negative	Maintain gustatory goal	Reject substance/withdraw
Happiness	Making reasonable progress toward the realization of a goal	Positive	Self-reward	Bask/bond

Treating fear as example, processing of emotion into reaction can be presented. If event recorded by an agent is new, nothing similar happened earlier, and it caused existential danger, then it shall be treated as negative. This causes immediate decrease of positive emotion value. Moreover, that kind of event causes fear of loosing life (assuming typical reaction of human, not considering psychological deviation). Reaction function in this case is chosen as protective aiming elimination or reducing the danger. It assumes revision of the current plan and proposing new action. In other words, this process can be explained on example scenario. Assuming that an agent is moving in the direction of certain goal and approaching a danger, it will try to change the moving direction allowing to keep distance from the source of fear. Velocity of an agent will be also changed to decrease necessary contact with the hazardous area. Depending on the level of danger, goal itself could also be revised and make agent to resign of pursuing it.

To create such correlation, behavioral tree was used, which defines set of actions fitting emotions. Thus, the priority choice of action was stated based on dominating emotion and level of its activation. Level of activation also plays a role in decision-making, causing escalation or de-escalation of actions. This kind of approach allows simple adaptation of engine with many different NPC (none-player character) steering systems. Except of simplicity in adaptation, correlating emotion engine with behavioral tree greatly increases possibilities of the extending proposed solution with additional actions, without losing preferences of emotional state processing.

Moreover, solution includes emotional maps and particle system adaptation to ensure wider influence on pathfinding. Therefore, the system of trespassing costs was introduced with the degradation of effect over time and repeller–attractor attribute. In addition, the area of activation is also changed based on event interpretation. To verify the correctness of approach and proposed solution, test was conducted checking readability of NPC behavior by possible participant or user of simulation.

5 Architecture of testing environment

Conducted experiment was done with the use of UnrealEngine since it has wide choice of tools, which highly simplifies and accelerates implementation. It also has built-in pathfinding system that uses the heuristic algorithm. In addition, it considers cost of trespassing, and blueprints system can be easily used to create behavioral trees. Finally, there is a wide support of graphical aspect for many kinds of tests in virtual environment.

Test was conducted in the form of survey, in which users decided about readability of agents reactions in virtual environment. To reduce any possible interferences and reasons of biased judging, agent and other elements were presented as 2D geometrical figures according to following illustration. Agent, represented as a triangle, has a task of getting through a maze. Green circle represents emotional stimuli. Depending on the trial, different emotions are stimulated, which changes the behavior of the agent. User is supposed to decide which emotion is dominating decisions of triangle.

6 Test

Thirty-four people participated in the test of various age. Exact scenario of test is presented as follows:

Each participant was presented with 6 recordings. One recording for one emotional reaction (joy, anger, disgust, sad, and fear) and additional reference HreHcording without emotion engine influence
1. Recording 1: after seeing stimuli (green circle), triangle moved away in opposite direction (run away) – fear
2. Recording 2: after seeing stimuli (green circle), triangle moved slowly in the direction toward stimuli and stayed in a close range and then continued the movement to its goal – joy
3. Recording 3: after seeing stimuli (green circle), triangle moved around stimuli, keeping distance, in a quick manner – disgust
4. Recording 4: after seeing stimuli (green circle), triangle very quickly approached stimuli and then moved to its goal (attack) – anger
5. Recording 5: after seeing stimuli (green circle), triangle very slowly moved toward its goal, ignoring stimuli – sad
6. Recording 6: after seeing stimuli (green circle), triangle moved with typical velocity to its goal – reference recording
Every participant was asked to decide what emotion is presented, where multiple answer was allowed
After first stage, participants were requested to connect emotions with description of reactions, where multiple answers were also allowed.
Last stage was asking participants to judge reactions again, with changed order of recordings to check if stage 2 had learning aspect

7 Results

Following are the results of test (Figure 8).

Figure 8

Results of the first stage during tests.

Every dominating emotion was correctly recognized. However, participants were not sure of their answers. It is visible on example of recording 2 and 4. Clearly anger and joy were thought to be similar. Probable reason is high level of activation in expressing both emotions. Both are distinct with impulsive and sudden actions, in comparison with other emotions. Clear recognition appeared for disgust, where great majority assumed the origin of behavior. In addition, reference recording also proved to be easily recognizable what seem to proof correctness of testing scenario.

Second stage results are presented in Table 2.

Table 2

Results of correlating behavior descriptions with emotions

*	Attack	Interest	Slow movement	Running away	Omitting
Anger	30	0	2	3	3
Joy	5	34	2	0	0
Sadness	0	1	22	6	9
Fear	2	1	7	25	12
Disgust	2	0	12	6	25

Participants placed descriptions clearly correctly. However, the highest doubts were with slow movement, where a few interviewees seen disgust. Sadness and disgust had the most similar results.

Here are results of the last stage (Figure 9).

Figure 9

Results of last stage during tests.

Judging result in last stage shows much higher certainty in choices what could be understood, as increase in emotion recognition. It would mean that second stage had learning characteristic for participants.

8 Conclusions

During research emotion, engine was created, which connects the appraisal theory and the PAD space theory, allowing to increase adaptation possibilities. In addition, complexity of engine architecture was decreased based on Ekman’s theory with keeping dynamics of emotional state processing. Next behavioral tree was included as augmentation of emotion engine aiming to define quantifiable collection of actions and mechanics of decision-making. The reason was to correlate it with emotion engines steering abilities. Hybrid solution that was created greatly increases the number of possibilities to use it in virtual environment and crowd simulations. It should also increase the potential of user’s immersion during experiments checking human behavior in crowd for different scenarios. Hence, it can lower the possible cost of such experiment substituting the majority of participants with NPC units. Moreover, with use of emotion-tendency correlation, collection of actions was created and defined, which can be assigned to the emotional state, resulting from engine processing, depending on the currently dominating emotion and its level. This way reasonable decision system was acquired with adaptation of predictable behavior, including basis in the research of psychology field. Finally, there was a try to verify the accuracy of chosen approach and chosen actions in the direction of users understanding agent’s acting. It was done on the basis of moving aspect. Results proved to correct user’s understanding of agent’s emotional state observing its behavior. However, some of emotions and actions seemed to have similar impressions. The reason might be similarity between emotions in aspect of the activation level. Moreover, it is highly probable that some of emotions need additional ways of expression, like body expression. Therefore, some additional tests should be conducted with animations included. Field of emotion simulation in virtual environment has the wide use of potential, especially in crowd simulations in which the user is supposed to feel immersion and adapt to groups actions. Further research will focus on this aspect. This work was supported by the NCBiR under Project POIR.01.02.00-00-0133/16.

References

[1] Panksepp J. Affective consciousness: core emotional feelings in animals and humans. Conscious Cognit. 2005;14(1):30–80.10.1016/j.concog.2004.10.004Search in Google Scholar PubMed

[2] Wright J, Panksepp J. An evolutionary framework to understand foraging, wanting, and desire: the neuropsychology of the SEEKING system. Neuropsychoanalysis. 2012;14(1):5–75.10.1080/15294145.2012.10773683Search in Google Scholar

[3] Grycuk R, Wojciechowski A, Wei W, Siwocha A. Detecting visual objects by edge crawling. J Artif Intell Soft Comput Res. 2020;10(3):223–37.10.2478/jaiscr-2020-0015Search in Google Scholar

[4] Andrzejczak J, Osowicz M, Szrajber R. Impression curve as a new tool in the study of visual diversity of computer game levels for individual phases of the design process. In: Computational Science - ICCS 2020. ICCS 2020. Lecture Notes in Computer Science, Krzhizhanovskaya V. et al. editors. vol. 12141 Cham: Springer; 2020.10.1007/978-3-030-50426-7_39Search in Google Scholar

[5] Lipinski P, Leplawy M. WiFi electromagnetic field modelling for indoor localization. Open Phys. 2019;17(1):352–7.10.1515/phys-2019-0039Search in Google Scholar

[6] Daszuta M, Wróbel F, Rynkiewicz F, Szajerman D, Napieralski P. Affective pathfinding in video games. J Appl Comput Sci. 2018;26(2):23–9.Search in Google Scholar

[7] Johansson A, Dell’Acqua P. Pathfinding with emotion maps. In: Plemenos D, Miaoulis G, editors. Intelligent Computer Graphics 2011. Studies in Computational Intelligence, vol. 374. Berlin, Heidelberg: Springer; 2012. 2011.10.1007/978-3-642-22907-7_8Search in Google Scholar

[8] Lim MY, Dias J, Aylett R, Paiva A. Intelligent NPCs for educational role play game. In: Dignum F, Bradshaw J, Silverman B, van Doesburg W, editors. Agents for Games and Simulations. AGS 2009. Lecture Notes in Computer Science. vol. 5920. Berlin, Heidelberg: Springer; 2009.10.1007/978-3-642-11198-3_8Search in Google Scholar

[9] Dörner D. PSI: A computational architecture of cognition, motivation, and emotion. Rev Gen Psychol. 2013;17:297–317. 10.1037/a0032947 Search in Google Scholar

[10] Pena L. Universidad Rey Juan Carlos Departamento de Arquitectura y Tecnolog´ıa de Computadoras. Cienc de la Computaci´e Inteligencia Artificial. Madrid; 2013.Search in Google Scholar

[11] Feldmaier J, Diepold K. Path-fnding using reinforcement learning and affective states, The 23rd IEEE international symposium on robot and human interactive communication, 2014.10.1109/ROMAN.2014.6926309Search in Google Scholar

[12] Li L, Campbell J. Emotion modeling and interaction of NPCS in virtual simulation and games. Int J Virtual Real. 2015;9. 10.20870/IJVR.2010.9.4.2784 Search in Google Scholar

[13] Sales R, Clua E, de Oliveira D, Paes A, Chaimowicz L, Nunes MASN. Evaluation between humans and affective NPC in digital gaming scenario, IEEE 3nd International Conference on Serious Games and Applications for Health (SeGAH) 2014.10.1109/SeGAH.2014.7067101Search in Google Scholar

[14] Veselov N, Chubarov A, Roshchin M, Marder M, Segal S, Samsonovich A. Emotional BICA for non-player characters: new empirical data, postproceedings of the 10th annual international conference on biologically inspired cognitive architectures, 2019.10.1016/j.procs.2020.02.238Search in Google Scholar

[15] Mocholí J, Esteve J, Jaén J, Acosta R, Xech P. An emotional path finding mechanism for augmented reality applications, international conference on entertainment computing, 2006.10.1007/11872320_2Search in Google Scholar

[16] Shen L, Bigsby E. Behavioral activation/inhibition systems and emotions: a test of valence vs. action tendency hypotheses. Commun Monographs. 2010;77:1–26. 10.1080/03637750903514268.Search in Google Scholar

[17] Inzlicht M, Bartholow B, Hirsh J. Emotional foundations of cognitive control. Trends Cogn Sci. 2015:19. 10.1016/j.tics.2015.01.004.Search in Google Scholar PubMed PubMed Central

Received: 2020-10-04

Accepted: 2020-10-17

Published Online: 2020-12-02

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/phys-2020-0199

Keywords for this article

emotions; cognitive modeling; intelligent systems; fuzzy approach

Creative Commons

BY 4.0