Emotion recognition and interaction of smart education environment screen based on deep learning networks

2.1 Facial and background action-based dual stream coding network framework

As shown in Figure 1, since the first part of Caps Net, Conv1, is shallowly convolved [11], the extracted features are not sufficient to effectively provide the required semantic information of expressions for the second part, the deep extraction layer, Primary Caps [17]. As a result, the capability of the second part is not fully utilized, which leads to a lower capability of the third part, dynamic routing layer dynamic routing [18], to decide the effective capsule vectors.

Figure 1

Caps Net network structure. Source: Created by the authors.

To solve this problem, this article proposes an image sentiment analysis method based on dual-stream coding of facial and background actions. The MTCNN model offers superior performance in detecting and aligning facial features due to its hierarchical structure that processes tasks at different scales [19]. This capability is crucial for accurately capturing the subtle facial expressions of learners in diverse educational contexts, where variations in lighting, pose, and background can significantly affect recognition accuracy. The overall architecture of the model, which is designed as a two-channel structure [20], including two feature coding streams: facial coding and background coding streams. The facial coding part first performs face cropping on the input frame-level images, after which the processed face images are fed into the facial coding module, which is based on the 3D-CNN structure [21], and two 2D convolutions are used instead of the original 3D convolution to improve the computational efficiency and obtain the face module features. The background coding stream first takes the cropped image as an input to another 3D-CNN, after which an attention coding module is used to find the attentional region of the contextual information in the background image, enabling the background coding stream to focus on the background context that is more helpful for emotion recognition, which allows the network to reduce ambiguity and improve the accuracy and robustness of emotion recognition. Finally, the two parts of the features are fused and fed into a classifier to predict the emotion category.

The advantage of the dual-stream coding method over traditional single-stream approaches lies in its comprehensive analysis capability. While single-stream methods typically focus solely on facial expressions, they often overlook the context, which can provide essential cues about the emotional state. For example, the same facial expression may convey different emotions depending on the context. By integrating both facial and background information, our dual-stream method provides a more nuanced and accurate emotion recognition, particularly in complex and dynamic learning environments where contextual cues play a significant role.

2.2 Human facial feature extraction

The input image data has to be segmented into face parts separately to obtain a face image before it can be analyzed by the face coding network; therefore, the face region in the image is extracted first; in this article, MTCNN is used to extract the face part of the face. This approach not only improves the efficiency and accuracy, but also reduces the performance loss of the traditional approach of using sliding windows and classifiers. MTCNN constructs three different network models, P-Net, R-Net, and O-Net, through the idea of recursive execution, which makes the face facial detection with higher efficiency and speed. First, the image pyramid produced transforms the image input to the input layer at various scales so that it can detect facial images at different scales. Then, the P-Net network produces a large number of region boxes as candidate target boxes, and then, the R-Net network carries out the initial fine selection and border reduction of these candidate region boxes and removes all the negative examples, and then, the O-Net network, which is a more complex network with higher accuracy, carries out the second discrimination and border reduction of all the candidate region boxes. The specific construction of the three networks is shown in Figure 2. In this article, we will utilize the three-layer network of MTCNN for face extraction and cropping and use the face photo as the input of the subsequent face coding system. This method not only improves the efficiency and accuracy of face extraction, but also provides a high-quality data base for subsequent sentiment analysis.

Figure 2

Network architecture of P-Net, R-Net, and O-Net. Source: Created by the authors.

P-Net is an important component in MTCNN, which is a face delineation boundary suggestion network based on CNN. The network accepts raw image features and processes them through three convolutional layers, then uses a face classifier to detect and delimit initially conforming face regions through border region reduction and a face keypoint localizer. This step generates a number of target candidates that may contain faces, and these candidates are fed into R-Net for subsequent detection.

R-Net is the second layer network in MTCNN face detection, which is used to further process the possible face regions output from P-Net and filter out the more credible face regions for subsequent O-Net. In the R-Net network, the input candidate face regions are selected more carefully, most of the wrong face regions are removed, and the border reduction and facial keypoint localizer are used again to restore the border and localize the keypoints of the remaining face candidate regions. Compared with the P-Net network, this layer uses 1 × 1 × 32 features generated by the full convolutional network and introduces a 128-dimensional fully connected layer after the final convolutional layer, which is not only more accurate than the P-Net, but also obtains more original image features.

O-Net is the most complex part of MTCNN face detection, which is richer in features, and a 256-dimensional fully connected layer is added at the end of the structure, which also preserves more image features on the basis of R-Net. O-Net obtains the upper-left and lower-right coordinate values and five feature points of the face region through the process of detecting the face, restoring the region’s edge, and localizing the key points of the face: coordinate values and five feature points. Compared with the first layer of P-Net, O-Net not only has more feature inputs and more complex network structure, but also shows more superior performance. Therefore, the output of the O-Net network is also used as the output of the MTCNN detection algorithm model.

2.3 3D-CNN convolution

In a normal 2D CNN, the 2D convolution operation takes the local nearest neighbor region on the feature maps acquired in the previous layer to acquire the feature, subsequently adds an additional bias (bias), and finally inputs this result to a tanh function[n] (activation function) [22]. The value of the jth feature map of the ith layer at position (x, y) is shown in the following equation:

where tanh (∗) is the activation function, b i j is the bias of this feature map, m denotes the set index of the feature maps of the (i–1)th layer connected to the current feature map, w is the value of the convolution kernel connected to the kth feature map at the position of (p, q) position, and. are the height and width of the convolution kernel. In the downsampling layer, the resolution of the feature maps is reduced by performing pooling operations on the local nearest neighbors of the feature maps in the previous layer. A CNN is a network structure formed by stacking together such alternations of convolution and pooling. The parameters of a CNN, e.g., b i j     and kernel weights w i j m p q , are learned in a supervised or unsupervised manner.

In 2D CNNs, convolution is performed on a flat feature map, and 2D convolution is unable to obtain the corresponding motion coding information from multiple consecutive frames when higher-dimensional aspects of the problem need to be solved. In order to deal with this problem, this article adopts 3D convolution operation, which was motivated by its proficiency in analyzing temporal information, which is vital for understanding the dynamic nature of emotional expressions over time. The method is to construct a 3D convolution kernel and convolve it with multiple consecutive frames, so that the feature maps of the convolutional layers are connected into multiple consecutive frames on the previous layers, and thus, the motion information is captured. Finally, the value at position (x, y, z) of the jth feature map in the ith layer [23] is shown in the following equation:

where R i is the scale of the 3D convolution kernel in the time dimension, and the value of (p, q, r) connected to the mth feature map convolution kernel of the previous layer is w. The difference about 2D convolution and 3D convolution is shown in Figure 3.

Figure 3

Comparison of 2D and 3D convolutions: (a) 2D convolution and (b) 3D convolution. Source: Created by the authors.

Specifically, the face coding network consists of five 3 × 3 × 3 convolutional layers, followed by a BN layer, a ReLU activation function, and five maximum pooling layers, where the first pooling layer has a pooling kernel size of 1 × 2 × 2 in order to avoid premature merging of temporal information. The remaining pooling layers are maximum pooling layers of size 2 × 2 × 2, and the kernel counts of the five convolutional layers are 32, 64, 128, 256, and 256, respectively, and finally, the output features are spatially averaged by an average pooling layer to obtain the final output of the face coding network.

2.4 Background context coding

Compared with the face coding network, the background context coding layer includes a context coding module and an attention inference module. In order to extract contextual information other than facial expressions, specifically, the face part is first masked to obtain the remaining background image, which is fed into the coding network, which has the same design of convolutional and pooling layers as the face coding network, and also employs two convolutions to improve the computation rate to obtain the extracted background contextual features, which are then fed into an attentional inference module, through which the context cataloging can be made to focus on different contexts, extracting the part of the background context that is helpful for sentiment analysis.

The core idea can be expressed by the idea of mathematical level: X = [ x 1 , x 2 , … , x N ] is taken as N inputs, and at the same time, in order to reduce the cost of computing, it is not necessary to perform any processing on these N inputs by neural network, and only some task-related information from these N inputs need to be selected for computing. The soft attention mechanism is a method of selecting input information that, unlike hard attention, instead of selecting only 1 out of N information, is processed by weighted averaging the N inputs and then feeding them into the residual network for computation. The computation of the attention value usually consists of two aspects, i.e., computing the attention distribution of the input data and weighted averaging the input data by the attention distribution obtained from the computation.

Specifically, the background image that has been pooled by 3D convolution is taken as input X c , and since 3D convolution has already feature extracted the input in the temporal dimension, the attention weights are denormalized only in the spatial dimension. In the attention inference module, 128 and 1 feature channels are generated by two convolutional layers with a convolutional kernel size of 3 × 3 × 3 with a normalization layer, and the attention A ∈ R H × W     is obtained from the input X c , where H × W     is the spatial resolution of X _c. Afterward, in order to make the sum of the attention of each pixel to be 1, the output is passed through a spatial Softmax normalization layer. The output is shown in the following equation:

where A ˆ ı denotes the attention to the context at i, j. Afterward, the new background context feature with attention optimized is obtained by combining the weights of the attention part of the output on the input feature X _c, and finally, the output is obtained through a spatially averaged pooling layer [24] as well, which is formulated as follows:

where ⊙ denotes the element-by-element multiplication operator.

The algorithm model proposed in this article takes the cross-entropy of the predicted data and the actual results as the loss function of the model, and optimizes the model parameters by obtaining the minimum value of the loss function, and at the same time, adopts the method of random deactivation (Dropout) and the L 2 regularization operation to avoid the overfitting phenomenon during the training, and the formula of the cross-entropy loss function is expressed as

where y i j denotes the prediction data generated by the ith sentence; y ˆ i j denotes the corresponding actual result; j denotes the specific number of classifications, which is set according to the sentiment category of the dataset for division; λ is the coefficient of the L 2 regularization term; and φ     denotes all the parameters to be learned.

4.1 Experimental data collection

In this article, experiments were conducted using the Emotic dataset [17], which combines annotations for 26 discrete emotion categories and three continuous emotion dimensions. The image sources for the Emotic dataset are divided into two main categories: one part originates from two public datasets – COCO and Ade20k; the other one part is obtained through Google search engine. Moreover, the inclusion of both discrete and continuous emotion dimensions offers a multifaceted view of emotional expressions, essential for developing a more accurate and sensitive emotion recognition model tailored to smart education. As shown in Figure 6, the selected images are characterized by two distinctive features: the wide diversity of backgrounds and the richness of different locations and environments. These features not only ensure the richness and diversity of the Emotic dataset, but also bring additional complexity and challenges to the emotion recognition task. The choice of the Emotic dataset was driven by its comprehensive coverage of diverse emotions and realistic settings, making it highly relevant for the study’s objectives. The Emotic datasets were connected to a computer system equipped with an Intel Core i7 processor, NVIDIA GeForce RTX 2080 Ti GPU, 32GB RAM, running on Ubuntu 18.04 LTS. For data processing and simulation, we utilized PyTorch, a powerful open-source machine learning library, and Python 3.8.

Figure 6

Example of Emotic dataset [17]. (a) Displays an image with multiple people in varying poses, (b) shows an example of an image with a clear background, and (c) includes an image with significant background clutter.

4.2 Loss functions and evaluation indicators

The loss function is a weighted combination of two individual losses, L = λ 1 L 1 + λ 2 L 2 , L 1 and L 2 are the sum of 26 discrete affective losses and the sum of 3 continuous affective losses, respectively, and λ 1 and λ 2 are the weights of the discrete affective losses and the weights of the continuous affective losses, respectively. The discrete affective loss L 1 uses multi-label focal loss (MFL) [25] and the continuous affective loss L 2 uses Huber loss [26], defined as

In equation (5), y ˆ disc is the predicted value of the ith category, y ˆ disc is the true label of the ith category, and a and y are two hyperparameters. In equation (6), y ˆ cont is the predicted value of the kth consecutive sentiment, y cont is the true value of the kth consecutive sentiment, and is an empirical parameter. After many experiments, the recognition effect of the model reaches the best when α = 0.5， γ = 3, and δ = 0.5. The evaluation metrics are Precision (Precision) and mean absolute error (MAE). Furthermore, this article utilized the Adam optimization algorithm with a learning rate of 0.001, selected for its adaptive learning rate properties. The number of epochs was set to 100, to ensure the model adequately learns from the training data without overfitting, with 32 batch size. The choice of these hyperparameters was based on extensive experimentation aimed at balancing model performance and training efficiency.

4.3 Tests results

The sentiment accuracy and sentiment recognition qualitative results obtained by the sentiment recognition model on the Emotic dataset are shown in Figure 7, and the continuous sentiment MAE is shown in Table 1.

Figure 7

Emotional recognition precision results. Source: Created by the authors.

The analysis of Figure 7 and Table 1 reveals the final average accuracy rate achieved by the sentiment recognition model on the Emotic dataset to be 32.517%. Notably, the model exhibits exceptional performance in accurately classifying emotions associated with anticipation, engagement, confidence, excitement, and happiness, with accuracy rates soaring above 70%. This high level of precision underscores the model’s efficacy in discerning these specific emotional states, which are crucial for understanding learner engagement and receptivity in smart educational environments.

4.4 Ablation experiment

Ablation experiments with different branch combinations are performed to ensure that the other parameters of the experiments are consistent. Experiment 1 has character body information, experiment 2 is a combination of character body information and environment semantic information, and experiment 3 is a combination of character body information and depth map information. The ablation experiments with different fusion strategies are carried out to ensure that the other parameters of the experiments are consistent, experiment 1 uses feature-level fusion, experiment 2 uses decision-level fusion, and experiment 3 uses hybrid-level fusion, and the average precision (AP) and mean absolute error (MAE) obtained are shown in Table 2. The AP and mean absolute error (MAE) are shown in Table 2. The experimental results highlight a notable improvement in the performance of the hybrid fusion strategy over its counterparts. Specifically, the hybrid fusion strategy exhibits a 3.1% increase in average accuracy compared to feature-level fusion and a 6.6% increase compared to decision-level fusion. Furthermore, this strategy demonstrates a significant reduction in the mean absolute error, decreasing by 0.0058 in comparison with feature-level fusion and by 0.035 in comparison with decision-level fusion.

Figure 8 presents a comparative analysis of the precision rates achieved through different fusion strategies employed in our emotion recognition model. The strategies include feature-level fusion, decision-level fusion, and hybrid-level fusion. The results indicate that the hybrid-level fusion strategy outperforms the other two, with an AP rate of 33.143% and an mean absolute error (MAE) of 0.0795. In contrast, feature-level fusion and decision-level fusion strategies yielded lower precision rates of 30.042 and 26.547%, respectively, with corresponding increases in MAE. These findings underscore the effectiveness of the hybrid fusion strategy in enhancing emotion recognition performance, offering a robust approach for integrating various data sources, and improving model accuracy.

Figure 8

Comparison of precision of different fusion. Source: Created by the authors.

The proposed emotion recognition model is compared with the models proposed by Kolbadi and Sarvar [21], and the AP and mean absolute error (MAE) obtained are shown in Table 3.