Study on the influence of biomechanical factors on emotional resonance of participants in red cultural experience activities

1 Introduction

Emotion is a physiological state that integrates all the feelings, thoughts, and activities of the human body, as well as a psychological and physiological response produced by various external stimuli. Red culture is a vital part of Chinese heritage that commemorates the revolutionary history of the Communist Party, promotes collectivism, fosters national pride, and cultivates patriotism. It preserves historical memory through stories, media, and experiential activities, particularly targeting young people. By adapting to modern formats like digital media, red culture remains relevant, inspiring civic responsibility and emotional connection to shared national values. Electroencephalography is considered a physiological measure in which the electrical activity of nerve cells is detected in the human cerebral cortex, while emotion-related brain activity can also be detected through electroencephalography. The proposed method selects the emotions of happy, sad, fear, and neutral for their broad representation of emotional states across the valence-arousal spectrum, aligning with established theories such as Ekman’s and Russell’s models. These emotions show distinct patterns in EEG and eye-tracking data, enhancing classification accuracy. They are also relevant to the red cultural film stimuli used, which are designed to evoke these specific emotional responses. The SEED-IV dataset supports this choice by providing consistent, pre-rated clips for reliable emotional elicitation and labelling. In the study of EEG signals, Thammasan et al. utilised power spectral density to extract information features from the original EEG signal and then applied an emotion classification algorithm to classify continuous music with binary emotions. The classification accuracy reached 72.9% [1]. Wu et al. employed correlation and coherence analysis to estimate EEG connectivity between electrodes, aiming to identify subjects’ emotional levels in arousal and valence dimensions. The results revealed that connectivity features in the gamma band contained information conducive to valence classification [2].

In recent years, eye-tracking technology has also become a crucial research tool in various fields, including psychology and cognitive science. Eye movement signals have become more convenient to capture, and participants’ instinctive cognitive behaviours can be known by interpreting their eye focus. In the study of emotion recognition related to eye movement signals, Oliva and Anikin found that pupil responses are closely tied to the time process of emotion processing and provide rich information sources. The intensity of perceived emotion can only enhance pupil dilation [3]. Moreover, affective feedback significantly affects gaze behaviour, and its performance in arousal is better than that of titer. Lu et al. applied features related to eye movements to identify positive, negative and neutral emotions, with an average classification accuracy of 77.80% [4]. Zheng et al. utilised features such as fixation time, saccade amplitude and duration, and blink frequency to classify the degree of four emotions, achieving an accuracy of 67.82% [5]. Generative AI applications can enhance user engagement in cultural experiences by utilising multimodal emotion recognition, which leverages EEG and eye movement signals. These applications can predict or simulate user emotional responses, create personalised narratives or audio guides, and offer supportive interventions during moments of sadness or fear. A GAN-based synthetic data generator can produce realistic multimodal emotional signals, address data imbalance, and enhance model training. A personalised cultural feedback agent can analyse emotional patterns and generate customised exhibit suggestions, thereby enriching the post-experience journey. These generative AI extensions are scalable and emotionally intelligent.

The proposed multimodal approach, which integrates EEG differential entropy (DE), minimum spanning tree (MST)-based brain connectivity, and eye movement features, enhances the precision of emotion recognition. These insights enable user interfaces to adapt dynamically to users’ emotional states, improving engagement and usability. Generative AI systems can utilise emotional feedback to personalise content in real time, thereby deepening user resonance. In educational settings, recognising emotions can help tailor learning experiences to enhance motivation and retention. Biomechanical cues such as gaze patterns and connectivity metrics inform more intuitive interaction designs. These findings support the development of emotionally aware, human-centred technologies across various application domains.

Regarding the recognition of emotions in multimodal physiological signals, Islam et al. combines EEG signals with facial expressions in a multimodal manner. He finds that the emotion recognition rate of multimodal fusion is 61.88%, while the emotion recognition rates of single modes are 60.2 and 52.43%, respectively [6]. Kwon et al. combined electroencephalogram and electrodermal response and used CNN to merge the EEG spectrogram and electrodermal response features to complete multimodal emotion recognition [7]. Abtahi et al. employed audio/video, electromyography, and EEG to investigate four different modes of emotion [8].

Biomechanical signals, such as EEG and eye movement, provide direct insight into participants’ embodied emotional responses during red cultural experience activities. These responses are not merely physiological but are deeply influenced by the symbolic and historical significance of revolutionary content, which evokes culturally anchored emotions such as patriotism and collective memory. This makes emotion recognition in such a setting uniquely valuable, as it captures how sociocultural meaning shapes sensorimotor expression. The integration of these biomechanical factors thus offers a novel lens for understanding emotion, extending beyond generic stimuli to include culturally embedded affective resonance.

It can be seen from the above collation and induction of relevant literature that in the studies on emotion recognition based on eye movement signals, this article finds that most features, such as pupil response and pupil diameter, are used for classification. Still, there is no clear indication of which features or combinations of features are most beneficial to emotion recognition tasks [9]. Ting et al. propose the ethnokinesiology framework, which links sensorimotor and cultural factors in shaping movement. It highlights motor accents as culturally influenced variations, supporting analysis of biomechanics in Red Cultural activities and their role in emotional engagement [10]. In addition, the number of studies on emotion recognition using eye movement signals is still relatively limited, and single-mode EEG data cannot fully and accurately represent a person’s emotional state. Therefore, utilising the complementarity of multimodal signals to represent emotion states can enhance the accuracy of the emotion recognition model and facilitate valuable exploration in selecting emotion classification features. Neural oscillations and oculomotor behaviour reflect emotional processing by capturing both internal brain activity and external behavioural responses. EEG features, such as DE and connectivity patterns extracted through MST analysis, reveal changes in neural complexity and functional network organisation during emotional states. Simultaneously, eye movement metrics, such as fixation patterns, pupil diameter, and saccades, indicate variations in arousal and attention. When combined, these two modalities enhance the accuracy and depth of emotion recognition by integrating both observable behaviour and physiological signals, thereby offering a more comprehensive understanding of emotional states.

This study aims to investigate how biomechanical factors such as EEG and eye movement influence participants’ emotional resonance in red cultural experience activities. It focuses on developing a multimodal emotion recognition model by combining EEG and eye movement signals to improve classification accuracy. The research examines the effectiveness of features such as DE and MST attributes. It compares unimodal and multimodal approaches across four emotional states: happiness, sadness, fear, and neutrality. The goal is to enhance emotional analysis by leveraging the complementarity of multimodal signals in immersive cultural settings.

To systematically investigate the emotional impact of biomechanical features during red cultural experience activities, this article is structured as follows: Section 2 introduces the methodology, including EEG and eye movement feature extraction, the construction of a multimodal emotion recognition model, and the use of MST analysis. Section 3 presents the experimental setup, dataset, signal preprocessing techniques, and the emotion classification framework, along with detailed performance results. Section 4 concludes the paper by summarising key findings and outlining directions for future research in multimodal emotion recognition within cultural contexts.

2 Feature extraction model based on multiple modes

2.1 EEG feature extraction model

2.1.1 DE features

Given that its probability density distribution function is P(x) for EEG time series X, the complexity of the EEG series can be expressed by DE [9]. The DE feature of the fixed subband in the EEG time series is calculated as follows:

(1) DE = 1 2 ln ( 2 π e σ 2 ) .

In preprocessing, bandpass filtering removes the DC component of the EEG signal, resulting in a mean value of 0. Therefore, the variance formula of the EEG signal is:

(2) σ 2 = 1 N ∑ i = 1 N x i 2 .

The variance of the EEG signal is the mean of its energy. According to Parseval’s theorem that the mean energy is proportional to the energy spectrum, it can be obtained:

(3) ∑ n = 0 N − 1 ∣ x i 2 ∣ = 1 N ∑ k = 0 N − 1 ∣ X k 2 ∣ = P i .

Sequence {x _i } is obtained in the formula by a short-time Fourier transform from sequence {X _k }. Therefore, the DE value of the EEG signal is related to its average energy. In the calculation process, the logarithm of the mean energy is taken to obtain the DE feature [11]. Compared to the frequency band energy, the difference between different sub-bands is much smaller, which can help balance the data gap, reduce errors, and improve algorithm accuracy. In the actual calculation of the DE feature, the logarithmic coefficient between it and the average energy of the EEG signal is ignored, and the formula for calculating the DE feature of each sub-band is obtained:

(4) h i ( X ) ≈ log ( P i ) .

2.1.2 Function connection specifications

In this article, the phase lag index is introduced, which is a measure of the asymmetric phase difference distribution between two signals, which can not only reflect the statistical interdependence and coupling strength between time series but also is expected to be less sensitive to the influence of common source and amplitude effects. The instantaneous phase expression of the original time series x(t) is as follows:

(5) ϕ x ( t ) = arctan x ˜ ( t ) x ( t ) .

The formula x ˜ ( t ) is the Hilbert transform of the original time series x(t). As the original time series, the EEG signal is the projection of the analytic signal on the real number domain. After restoring the analytic signal formula, the phase differences between the two signals can be calculated simultaneously [12]. Therefore, the calculation formula of the phase lag index of the two signals is as follows:

(6) PLI = ∣ 〈 sign ( Δ ϕ ( t ) ) 〉 ∣ = 1 N ∑ n = 1 N sign ( Δ ϕ ( t k ) ) .

In the formula, t _k represents the same time of two signals, and sign is a symbolic function whose phase lag index ranges from [0, 1].

2.1.3 MST

MST is a graph-theoretical approach used to model functional brain connectivity. It simplifies complex brain networks by connecting all nodes (EEG channels) with the minimum total edge weight, helping to quantify information flow and network efficiency during emotional processing. The MST is a valuable method for EEG analysis in emotion recognition, as it provides a threshold-free, sparse, and interpretable representation of brain connectivity. By using the inverse of the Phase Lag Index as edge weights, MST highlights key functional relationships while minimising noise. It enables the extraction of meaningful topological features that reflect changes in emotional state. Compared to traditional connectivity methods, MST ensures consistency across subjects and is less sensitive to noise. When combined with other features, such as DE and eye movement data, MST enhances emotion classification accuracy, providing complementary insights into brain function during emotional processing.

The MST is used to simplify and represent functional brain networks by connecting EEG channels with minimal redundancy while preserving key connections. It utilises phase lag index-based weights to ensure robustness against noise and computes graph metrics, such as degree, diameter, and tree hierarchy, to quantify network properties. MST features enhance emotion recognition when combined with EEG and eye movement data by providing topological and spatial insights into brain activity.

In this article, the MST is obtained using the greedy idea. First, M nodes need to be regarded as M separate spanning trees, and N edges are sorted in ascending order according to the weight value. Next, each of the N edges is extracted in sequence. If both nodes are found to be on two different trees, they are merged into one tree; otherwise, they are discarded, and the process continues traversing [13]. When all edges are traversed and all spanning trees are merged into one tree, the filtered edges and all nodes form the smallest spanning tree in this functional brain network. In this paper, the weight of an edge is defined as the reciprocal of the phase lag index. The constructed MST contains 62 nodes and 61 edges, that is, M = 62 and N = 61.

Six commonly used metrics are calculated to better measure the structure of the MST. They are Degree, Diameter (Dia), Eccentricity (Ecc), Leaf fraction (Lf), Betweenness Centrality (BC), and Tree hierarchy (Th).

The degree is the most basic and widely used metric in network research. It can describe the higher-order topological relationships in brain networks. The diameter is the maximum path length between any two nodes. It measures the global transmission efficiency of the network and can describe the size of the brain network. In small-diameter networks, information can be efficiently processed between distant brain regions [14]. However, it should be noted that the diameter of the MST is related to the number of leaf nodes.

Eccentricity is the maximum path length between a given node and any other node in the network. A higher eccentricity means that the node of the MST is closer to the central node. Average eccentricity E _i represents the average of all nodes deviating from the centre node:

(7) E i = ∑ j = 1 N ( max i ∈ N d i j ) N .

The leaf fraction is the ratio of leaf nodes to the number of nodes in the minimum-spanning tree. Intermediate centrality is an index to judge the importance of nodes in a network.

The tree level quantifies the balanced relationship between the integration capability of the MST and node overload, and its calculation formula is as follows:

(8) Th = L 2 MBC max .

In the formula, the coefficient of molecular position 2 ensures that the value of the tree level is between 0 and 1 [15]. In the MST, if L = M, the tree has a star structure, as shown in Figure 1, when Th = 0.5; if L = 2, the tree has a linear structure, as shown in Figure 2, where Th = 0.

Figure 1

Star MST topology network structure.

Figure 2

Partial MST topology network structure.

2.4 Feature classification

The core idea of support vector machine (SVM) is to find the classification hyperplane that separates samples reasonably and maximizes the margin [19]. The classification hyperplane model, interval maximisation strategy, and convex quadratic programming algorithm are the three key elements of SVM.

For binary classification problems, the model is illustrated in Figure 3, assuming the data is linearly separable.

Figure 3

Linear separable model of SVM.

In this case, the hyperplane data satisfy:

(11) ω T x + b = 1 , y i = 1 ω T + b = − 1 , y i = − 1 .

If the data space is linearly indivisible, the lower-dimensional data can be mapped to a higher-dimensional space by a kernel function to become linearly divisible. In this article, LinearKernel is used as the kernel function of SVM. The formula is as follows:

(12) k ( x , y ) = x t y .

Finally, the objective function of the dual problem is obtained as follows:

(13) L ( ω , b , a ) = 1 2 ∥ ω ∥ 2 + ∑ i = 1 m a i ( 1 − y i ( ω T x i + b ) ) .

3 Research on emotion recognition based on biomechanical factors

3.1 Data set

The data set used in this study was SEED-IV (SJTU Emotion EEG Dataset for Four Emotions), which included 62-channel EEG and eye movement signals from 15 subjects collected using the ESI neural scanning system and SMI eye tracking glasses. SEED-IV dataset uses 168 red culture movie clips as stimulus materials, recruits 44 volunteers to watch movie clips and score them from two dimensions of arousal and titer, and selects 72 movie clips with the highest matching degree and easy-to-induce target emotions for subsequent experiments [20]. Titer refers to the emotional intensity or strength perceived by participants, often in conjunction with arousal, to map emotions on a two-dimensional plane. Each participant conducted three experiments at different times, each approximately a week apart, watching 24 red culture movie clips that contained four emotions: happiness, sadness, fear, and neutral, with six movie clips for each emotion. The emotional experiment paradigm of the SEED-IV dataset is shown in Figure 4.

Figure 4

SEED-IV dataset emotion experiment paradigm.

It can be seen that there is a 5 s prompt time at the beginning of each movie segment. The duration of the movie segment is approximately 2 min, and the PANAS scale self-assessment time is 45 s after the end [21]. Based on the feedback, the data were discarded if the subject failed to trigger the right emotion or if the emotion was not awakened strongly enough. According to the electrode distribution diagram of the 62-channel international 10–20 system shown in Figure 5, the ESI neural scanning system was utilised to collect stable EEG signals. At the same time, SMI eye-tracking glasses were employed to capture eye movement information.

Figure 5

The 10–20 electrode placement system.

4 Conclusion

Emotion is a crucial factor that influences learners’ daily learning and life, and it accompanies learners throughout their entire learning process. However, the lack of emotion can easily cause learners to experience low learning enthusiasm, burnout, and poor emotional engagement, as well as other problems that restrict the long-term development of online learning. In this paper, the influence of biomechanical factors on the emotional resonance of participants in the red cultural experience was investigated. A multimodal physiological signal emotion recognition model based on EEG and eye movement was constructed, and the accuracy of participants’ classification of happy, sad, fearful, and neutral emotions was evaluated.

Overall, this study has achieved certain milestones and met the established research objectives. The research results presented in this paper may serve as a reference for subsequent studies in this field, particularly in terms of feature extraction and fusion strategies. However, since only two EEG and eye movement modes are considered in this paper, the need for emotion recognition in many applications will still be limited. Then, it will not be limited to certain physiological signals. Instead, more emotion-related modal data are used for fusion, such as ECG, myoelectricity, skin electricity, and voice, as well as expression, gesture, and other data to improve emotion recognition.