Startseite EEG-based emotion recognition with deep convolutional neural networks
Artikel
Lizenziert
Nicht lizenziert Erfordert eine Authentifizierung

EEG-based emotion recognition with deep convolutional neural networks

  • Mehmet Akif Ozdemir ORCID logo EMAIL logo , Murside Degirmenci , Elif Izci und Aydin Akan
Veröffentlicht/Copyright: 25. August 2020
Veröffentlichen auch Sie bei De Gruyter Brill
Manuskript einreichen Informationen für Autor*innen
Biomedical Engineering / Biomedizinische Technik
Aus der Zeitschrift Biomedical Engineering / Biomedizinische Technik Band 66 Heft 1

Abstract

The emotional state of people plays a key role in physiological and behavioral human interaction. Emotional state analysis entails many fields such as neuroscience, cognitive sciences, and biomedical engineering because the parameters of interest contain the complex neuronal activities of the brain. Electroencephalogram (EEG) signals are processed to communicate brain signals with external systems and make predictions over emotional states. This paper proposes a novel method for emotion recognition based on deep convolutional neural networks (CNNs) that are used to classify Valence, Arousal, Dominance, and Liking emotional states. Hence, a novel approach is proposed for emotion recognition with time series of multi-channel EEG signals from a Database for Emotion Analysis and Using Physiological Signals (DEAP). We propose a new approach to emotional state estimation utilizing CNN-based classification of multi-spectral topology images obtained from EEG signals. In contrast to most of the EEG-based approaches that eliminate spatial information of EEG signals, converting EEG signals into a sequence of multi-spectral topology images, temporal, spectral, and spatial information of EEG signals are preserved. The deep recurrent convolutional network is trained to learn important representations from a sequence of three-channel topographical images. We have achieved test accuracy of 90.62% for negative and positive Valence, 86.13% for high and low Arousal, 88.48% for high and low Dominance, and finally 86.23% for like–unlike. The evaluations of this method on emotion recognition problem revealed significant improvements in the classification accuracy when compared with other studies using deep neural networks (DNNs) and one-dimensional CNNs.

Keywords: azimuthal equidistant projection technique; brain mapping; deep learning; EEG images; electroencephalogram; emotion estimation
Corresponding author: Mehmet Akif Ozdemir, Department of Biomedical Engineering, Izmir Katip Celebi University, Izmir, Turkey; and Department of Biomedical Technologies, Izmir Katip Celebi University, Izmir, Turkey, E-mail:

Funding source: Izmir Katip Celebi University

Award Identifier / Grant number: 2019-ONAP–MUMF-0001

Acknowledgments

This study was supported by Izmir Katip Celebi University Scientific Research Projects Coordination Unit: Project numbers: 2019-ONAP–MUMF-0001.

  1. Research funding: This work was funded by Izmir Katip Celebi University Scientific Research Projects Coordination Unit (project number: 2019-ONAP–MUMF-0001). All funding is for equipment for the research project. There is no available funding for publication.

  2. Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Conflict of interest: Authors state no conflict of interest.

  4. Informed consent: Informed consent is not applicable.

  5. Ethical approval: The research related to human use complies with all the relevant national regulations and institutional policies. Human EEG data were obtained from the DEAP dataset which compiles all relevant ethical processes.

References

1. Bălan, O, Moise, G, Moldoveanu, A, Leordeanu, M, Moldoveanu, F. Fear level classification based on emotional dimensions and machine learning techniques. Sensors 2019;19:1738.10.3390/s19071738Suche in Google Scholar PubMed PubMed Central

2. Selvaraj, J, Murugappan, M, Wan, K, Yaacob, S. Frequency study of facial electromyography signals with respect to emotion recognition. Biomed Eng Biomed Tech 2014;59:241–9. https://doi.org/10.1515/bmt-2013-0118.10.1515/bmt-2013-0118Suche in Google Scholar PubMed

3. Li, P, Liu, H, Si, Y, Li, C, Li, F, Zhu, X, et al. EEG based emotion recognition by combining functional connectivity network and local activations. IEEE Trans Biomed Eng 2019;66:2869–81. https://doi.org/10.1109/tbme.2019.2897651.10.1109/TBME.2019.2897651Suche in Google Scholar PubMed

4. Pantic, M, Pentland, A, Nijholt, A, Huang, TS. Human computing and machine understanding of human behavior: a survey. In: Huang, T, Nijholt, A, Pantic, M, Pentland, A, editors. Artificial intelligence for human computing. Berlin: Springer Berlin Heidelberg; 2007. pp. 47–71.10.1145/1180995.1181044Suche in Google Scholar

5. Ekman, P, Friesen, WV, O’Sullivan, M, Chan, A, Diacoyanni-Tarlatzis, I, Heider, K, et al. Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 1987;53:712. https://doi.org/10.1037/0022-3514.53.4.712.10.1037/0022-3514.53.4.712Suche in Google Scholar

6. Ozdemir, MA, Degirmenci, M, Guren, O, Akan, A 2019, EEG based emotional state estimation using 2-D deep learning technique. In: 2019 Medical Technologies Congress (TIPTEKNO), 2019 IEEE: IEEE. pp. 1–4. https://doi.org/10.1109/tiptekno.2019.8895158.10.1109/TIPTEKNO.2019.8895158Suche in Google Scholar

7. Othman, M, Wahab, A, Karim, I, Dzulkifli, MA, Alshaikli, IFT. EEG emotion recognition based on the dimensional models of emotions. Procedia Soc Behav Sci 2013;97:30–7. https://doi.org/10.1016/j.sbspro.2013.10.201.10.1016/j.sbspro.2013.10.201Suche in Google Scholar

8. Yoon, S-a, Son, G, Kwon, S. Fear emotion classification in speech by acoustic and behavioral cues. Multimed Tools Appl 2019;78:2345–66. https://doi.org/10.1007/s11042-018-6329-2.10.1007/s11042-018-6329-2Suche in Google Scholar

9. Mert, A, Akan, A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl 2018;21:81–9. https://doi.org/10.1007/s10044-016-0567-6.10.1007/s10044-016-0567-6Suche in Google Scholar

10. Lan, Z, Sourina, O, Wang, L, Liu, Y. Real-time EEG-based emotion monitoring using stable features. Visual Comput 2016;32:347–58. https://doi.org/10.1007/s00371-015-1183-y.10.1007/s00371-015-1183-ySuche in Google Scholar

11. Khosrowabadi, R, Quek, HC, Wahab, A, Ang, KK 2010, EEG-based emotion recognition using self-organizing map for boundary detection. In: 2010 20th International Conference on Pattern Recognition, 2010 IEEE: IEEE. pp. 4242–5.10.1109/ICPR.2010.1031Suche in Google Scholar

12. Verma, GK, Tiwary, US. Affect representation and recognition in 3d continuous valence–arousal–dominance space. Multimed Tools Appl 2017;76:2159–83. https://doi.org/10.1007/s11042-015-3119-y.10.1007/s11042-015-3119-ySuche in Google Scholar

13. Koelstra, S, Muhl, C, Soleymani, M, Lee, J-S, Yazdani, A, Ebrahimi, T, et al. Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affective Comput 2011;3:18–31.10.1109/T-AFFC.2011.15Suche in Google Scholar

14. Li, J, Qiu, S, Shen, Y-Y, Liu, C-L, He, H. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Trans Cybern 2019:1–13. https://doi.org/10.1109/tcyb.2019.2904052.10.1109/TCYB.2019.2904052Suche in Google Scholar PubMed

15. Shimizu, S, Ito, T, Yin, Y, Arakawa, S, Sawada, O, Aoyagi, I. Driver emotion estimation via convolutional neural network with ECG. Trans Jpn Soc Aeronaut Space Sci 2019;50:505–10.Suche in Google Scholar

16. Kehri, V, Ingle, R, Patil, S, Awale, R. Analysis of facial EMG signal for emotion recognition using wavelet packet transform and SVM. In. Tanveer, M, Pachori, R, editors. Machine intelligence and signal analysis. Singapore: Springer Singapore; 2019. pp. 247–57.10.1007/978-981-13-0923-6_21Suche in Google Scholar

17. García-Faura, Á, Hernández-García, A, Fernández-Martínez, F, Díaz-de-María, F, San-Segundo, R 2019, Emotion and attention: audiovisual models for group-level skin response recognition in short movies. In: Web intelligence, 2019 IOS Press: IOS Press. pp. 29–40.10.3233/WEB-190398Suche in Google Scholar

18. Bazgir, O, Mohammadi, Z, Habibi, SAH 2019, Emotion recognition with machine learning using EEG signals. In: 2018 25th National and 3rd International Iranian Conference on Biomedical Engineering (ICBME), 2019 IEEE: IEEE. pp. 1–5.10.1109/ICBME.2018.8703559Suche in Google Scholar

19. Soroush, MZ, Maghooli, K, Setarehdan, SK, Nasrabadi, AM. A review on EEG signals based emotion recognition. ICNSJ 2017;4:118.Suche in Google Scholar

20. Li, X, Song, D, Zhang, P, Yu, G, Hou, Y, Hu, B 2016, Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2016 IEEE: IEEE. pp. 352–9.10.1109/BIBM.2016.7822545Suche in Google Scholar

21. Gupta, V, Chopda, MD, Pachori, RB. Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sens J 2019;19:2266–74. https://doi.org/10.1109/jsen.2018.2883497.10.1109/JSEN.2018.2883497Suche in Google Scholar

22. Daimi, SN, Saha, G. Classification of emotions induced by music videos and correlation with participants’ rating. Expert Syst Appl 2014;41:6057–65. https://doi.org/10.1016/j.eswa.2014.03.050.10.1016/j.eswa.2014.03.050Suche in Google Scholar

23. Mohammadi, Z, Frounchi, J, Amiri, M. Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl 2017;28:1985–90. https://doi.org/10.1007/s00521-015-2149-8.10.1007/s00521-015-2149-8Suche in Google Scholar

24. Zhang, Y, Ji, X, Zhang, S. An approach to EEG-based emotion recognition using combined feature extraction method. Neurosci Lett 2016;633:152–7. https://doi.org/10.1016/j.neulet.2016.09.037.10.1016/j.neulet.2016.09.037Suche in Google Scholar PubMed

25. Zhuang, N, Zeng, Y, Tong, L, Zhang, C, Zhang, H, Yan, B. Emotion recognition from EEG signals using multidimensional information in EMD domain. BioMed Res Int 2017;2017:8317357. https://doi.org/10.1155/2017/8317357.10.1155/2017/8317357Suche in Google Scholar PubMed PubMed Central

26. Zhang, Q, Chen, X, Zhan, Q, Yang, T, Xia, S. Respiration-based emotion recognition with deep learning. Comput Ind 2017;92:84–90. https://doi.org/10.1016/j.compind.2017.04.005.10.1016/j.compind.2017.04.005Suche in Google Scholar

27. Liu, W, Zheng, W-L, Lu, B-L 2016, Emotion recognition using multimodal deep learning. In: International conference on neural information processing, 2016 Springer: Springer. pp. 521–9.10.1007/978-3-319-46672-9_58Suche in Google Scholar

28. Pandey, P, Seeja, KR 2019, Subject-independent emotion detection from EEG signals using deep neural network. In: International Conference on Innovative Computing and Communications, 2019 Springer: Springer. pp. 41–6.10.1007/978-981-13-2354-6_5Suche in Google Scholar

29. Kuanar, S, Athitsos, V, Pradhan, N, Mishra, A, Rao, KR 2018, Cognitive analysis of working memory load from EEG, by a deep recurrent neural network. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018 IEEE: IEEE. pp. 2576–80.10.1109/ICASSP.2018.8462243Suche in Google Scholar

30. Jirayucharoensak, S, Pan-Ngum, S, Israsena, P. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014;2014:1–10. https://doi.org/10.1155/2014/627892.10.1155/2014/627892Suche in Google Scholar PubMed PubMed Central

31. Zhou, J, Wei, X, Cheng, C, Yang, Q, Li, Q. Multimodal emotion recognition method based on convolutional auto-encoder. Int J Comput Int Sys 2018;12:351–8.10.2991/ijcis.2019.125905651Suche in Google Scholar

32. Tripathi, S, Acharya, S, Sharma, RD, Mittal, S, Bhattacharya, S. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In: AAAI’17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017 AAAI Press: AAAI Press; 2017. pp. 4746–52.10.1609/aaai.v31i2.19105Suche in Google Scholar

33. Snyder, JP. Map projections: a working manual, 1st ed. Washington: US Government Printing Office; 1987.10.3133/pp1395Suche in Google Scholar

34. Morris, JD. Observations: SAM: the self-assessment manikin; an efficient cross-cultural measurement of emotional response. J Advert 1995;35:63–8.Suche in Google Scholar

35. Pereira, ET, Gomes, HM 2016, The role of data balancing for emotion classification using EEG signals. In: 2016 IEEE International Conference on Digital Signal Processing (DSP), 2016 IEEE: IEEE. pp. 555–9.10.1109/ICDSP.2016.7868619Suche in Google Scholar

36. Wichakam, I, Vateekul, P 2014, An evaluation of feature extraction in EEG-based emotion prediction with support vector machines. In: 2014 11th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2014 IEEE: IEEE. pp. 106–10.10.1109/JCSSE.2014.6841851Suche in Google Scholar

37. Li, J, Zhang, Z, He, H. Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput 2018:1–13.10.1007/s12559-017-9533-xSuche in Google Scholar

38. Bashivan, P, Rish, I, Yeasin, M, Codella, N. Learning representations from EEG with deep recurrent-convolutional neural networks. CoRR 2015: 06448. abs/1511.Suche in Google Scholar

39. Panigrahi, N, Mishra, CS. A generic method for azimuthal map projection. Def Sci J 2015;65:390–4. https://doi.org/10.14429/dsj.65.8598.10.14429/dsj.65.8598Suche in Google Scholar

40. Alfeld, P. A trivariate clough—tocher scheme for tetrahedral data. Comput Aided Geom Des 1984;1:169–81. https://doi.org/10.1016/0167-8396(84)90029-3.10.1016/0167-8396(84)90029-3Suche in Google Scholar

41. Savareh, BA, Emami, H, Hajiabadi, M, Azimi, SM, Ghafoori, M. Wavelet-enhanced convolutional neural network: a new idea in a deep learning paradigm. Biomed Eng Biomed Tech 2019;64:195–205. https://doi.org/10.1515/bmt-2017-0178.10.1515/bmt-2017-0178Suche in Google Scholar PubMed

42. Bengio, Y. Practical recommendations for gradient-based training of deep architectures. In: Montavon, G, Orr, G, Müller, K, editors. Neural networks: tricks of the trade. Berlin: Springer; 2012. pp. 437–78.10.1007/978-3-642-35289-8_26Suche in Google Scholar

43. Kiranyaz, S, Ince, T, Gabbouj, M. Real-time patient-specific ECG classification by 1-D convolutional neural networks. IEEE Trans Biomed Eng 2015;63:664–75.10.1109/TBME.2015.2468589Suche in Google Scholar PubMed

44. Kim, B-K, Roh, J, Dong, S-Y, Lee, S-Y. Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User In 2016;10:173–89. https://doi.org/10.1007/s12193-015-0209-0.10.1007/s12193-015-0209-0Suche in Google Scholar

45. Gargeya, R, Leng, T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 2017;124:962–9. https://doi.org/10.1016/j.ophtha.2017.02.008.10.1016/j.ophtha.2017.02.008Suche in Google Scholar PubMed

46. Li, C, Bao, Z, Li, L, Zhao, Z. Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Inf Process Manag 2020;57:102185. https://doi.org/10.1016/j.ipm.2019.102185.10.1016/j.ipm.2019.102185Suche in Google Scholar

47. Al-Nafjan, A, Hosny, M, Al-Wabil, A, Al-Ohali, Y. Classification of human emotions from electroencephalogram (EEG) signal using deep neural network. Int J Adv Comput Sci Appl 2017;8:419–25. https://doi.org/10.14569/ijacsa.2017.080955.10.14569/IJACSA.2017.080955Suche in Google Scholar

48. Yin, Z, Zhao, M, Wang, Y, Yang, J, Zhang, J. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Programs Biomed 2017;140:93–110. https://doi.org/10.1016/j.cmpb.2016.12.005.10.1016/j.cmpb.2016.12.005Suche in Google Scholar PubMed

Received: 2019-11-19
Accepted: 2020-06-16
Published Online: 2020-08-25
Published in Print: 2021-02-23

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Artikel in diesem Heft

  1. Frontmatter
  2. Research Articles
  3. Design of a wearable four-channel near-infrared spectroscopy system for the measurement of brain hemodynamic responses
  4. Age dependency of the diabetes effects on the iris recognition systems performance evaluation results
  5. Controlled differential evolution based detection of neovascularization on optic disc using support vector machine
  6. Workflow and hardware for intraoperative hyperspectral data acquisition in neurosurgery
  7. EEG-based emotion recognition with deep convolutional neural networks
  8. Investigating electroencephalography signals of autism spectrum disorder (ASD) using Higuchi Fractal Dimension
  9. Analysis of autonomic nervous pattern in hypertension based on short-term heart rate variability
  10. Impact of mandibular prognathism on morphology and loadings in temporomandibular joints
  11. A new adaptive XOR, hashing and encryption-based authentication protocol for secure transmission of the medical data in Internet of Things (IoT)
  12. An integrated assessment system for the accreditation of medical laboratories
https://doi.org/10.1515/bmt-2019-0306
Schlagwörter für diesen Artikel
azimuthal equidistant projection technique; brain mapping; deep learning; EEG images; electroencephalogram; emotion estimation

Artikel in diesem Heft

  1. Frontmatter
  2. Research Articles
  3. Design of a wearable four-channel near-infrared spectroscopy system for the measurement of brain hemodynamic responses
  4. Age dependency of the diabetes effects on the iris recognition systems performance evaluation results
  5. Controlled differential evolution based detection of neovascularization on optic disc using support vector machine
  6. Workflow and hardware for intraoperative hyperspectral data acquisition in neurosurgery
  7. EEG-based emotion recognition with deep convolutional neural networks
  8. Investigating electroencephalography signals of autism spectrum disorder (ASD) using Higuchi Fractal Dimension
  9. Analysis of autonomic nervous pattern in hypertension based on short-term heart rate variability
  10. Impact of mandibular prognathism on morphology and loadings in temporomandibular joints
  11. A new adaptive XOR, hashing and encryption-based authentication protocol for secure transmission of the medical data in Internet of Things (IoT)
  12. An integrated assessment system for the accreditation of medical laboratories
Jetzt anmelden und 20% sparen
Institutioneller Zugang
Wie funktioniert Authentifizierung?
Haben Sie eine Idee, wie wir unsere Website verbessern können?
Bitte schreiben Sie uns.
© 2025 De Gruyter Brill
Heruntergeladen am 21.9.2025 von https://www.degruyterbrill.com/document/doi/10.1515/bmt-2019-0306/html
Button zum nach oben scrollen