Application of emotion recognition technology in psychological counseling for college students

2.1 Data collection

In this article, the data collection of 2,000 college students in a certain city is implemented through the survey on the status of mental health, with a total of 1,995 questionnaires recovered and 1,977 valid questionnaires. The data from the valid questionnaires are summarized, generalized, and analyzed to provide a more comprehensive understanding of the college students’ psychological status.

Basic information about the survey respondents is displayed in Table 1.

As shown in Table 1, among the survey subjects, the proportion of males is relatively high, at 54.7%, and the proportion of females is 45.3%.

College students are the successors of the great rejuvenation of the Chinese nation and an important group in promoting social development. The mental health status of college students directly affects their personal growth, success, and family happiness, as well as the safety, stability, and harmony of the campus. At the same time, it is closely related to the fate of the nation and the development of society. Adverse events caused by psychological abnormalities among college students are often seen in the media. Most unsafe issues on campus occur due to psychological abnormalities among students, and their mental health is not optimistic, seriously affecting campus safety, family harmony, and social stability [9,10].

Students with psychological issues are depicted in Table 2.

As shown in Table 2, the count of students with psychological issues is 555 with 28.1%; the count of female students with psychological issues is higher with 60.2%; the count of students with psychological issues in the 4th year of the university is the highest out of the 4 years with 36.94%.

The 4 years of university are a crucial period for the growth of college students, as well as an important process for their psychological development. Contemporary college students bear important responsibilities for national construction and development, and healthy psychological qualities and mental state are key factors in pursuing personal development [11,12]. However, the country is in a stage of social transformation, with social problems such as fierce competition and difficult employment emerging. At the same time, the amount of college students is constantly increasing, and the pressure they face in terms of learning, interpersonal relationships, economy, and employment is also increasing. The psychological pressure of college students is also increasing accordingly [13,14]. The conditions caused by psychological disorders are commonly seen in the media. Some students are isolated, introverted, and silent, while others have emotional disorders, anxiety, and depression. Some even seek liberation by ending their own or others’ lives [15,16].

2.2 Data filtering

In addition to focusing on teaching quality, the psychological health of students cannot be ignored in various universities. To provide psychological assistance to students, many schools not only offer psychological courses but also set up psychological counseling institutions and assign professional staff. However, due to widespread discrimination against mental illness, many students with mental health problems do not actively seek help. Every year when new students enter the school, psychological tests are conducted, and some students conceal their answers. Even when there are many questionnaire questions, there is a phenomenon of random answers, which cannot reflect the true mental health status of students [17,18]. Therefore, both counselors and teachers pay close attention to the behavior of students. Counselors promptly report students with mental health issues through the Student Worker Online System, based on feedback from their classmates, parents, teachers, and school hospitals. Experts at the psychological center assess the severity level of students based on reported events and observations of their micro expressions [19,20].

The psychological level includes three levels: mild, moderate, and heavy. The psychological level of students is illustrated in Figure 1 (the horizontal axis indicates mild, moderate, and heavy, and the vertical axis indicates the number of students).

Figure 1

Student’s psychological level.

As shown in Figure 1, the number of people with heavy psychological problems in grades 1, 2, 3, and 4 is 38, 27, 37, and 65, respectively.

College students’ mental health has a key role to play in their own growth and success. Having a healthy psychology is an effective guarantee for college students to pursue education, communicate with others, and work in the future. The down-to-earth progress in learning and the pursuit of truth and pragmatism in life cannot be separated from a healthy and positive psychological state [21,22]. A good psychological state can not only guide college students not to lose heart and not be discouraged when facing setbacks and challenges but also actively adapt to adversity. It can promote the cultivation of various abilities of college students, which is a solid foundation for their growth and success. On the contrary, unhealthy psychology leads to college students neglecting their studies and even going astray, causing serious harm to their personal growth and success.

2.3 Data analysis

The academic pressure of college students is directly proportional to their mental health. When their learning tasks increase, their psychological problems such as irritability and depression worsen, leading to various psychological disorders and serious harm to their mental health [23,24]. For individuals, mental health issues can have negative impacts, weaken their adaptability in society, and even pose a serious threat to their physical health. For families, mental health issues can cause psychological burden and also increase their financial burden.

The situation of students who need psychological counseling is described in Table 3.

As shown in Table 3, it can be seen that the majority of students are in great need of psychological counseling. The number of students in grades 1, 2, 3, and 4 who are in very need of psychological counseling is 51, 43, 50, and 99, respectively.

Overall, college students’ mental health is becoming more serious, which may be related to the increasingly fierce competition and increased pressure on their learning and employment. This also indicates that although universities are currently placing increasing emphasis on college students’ mental health, it is not yet in-depth enough and falls far short of the needs of college students. The focus of the mental health work for college students should not be on how to diagnose and treat mental illnesses, but on preventive and developmental research. The psychological intervention model should be constructed from the perspective of adaptation and development and should be carried out through multiple aspects, levels, channels, and means to truly play a greater role in school mental health education in higher education [25].

3.1 Text emotion recognition

Textual emotion recognition is based on natural language processing, whose main task is to take feature extraction, analysis, induction, and deduction operations on social media comments, especially textual comments containing emotional states [26,27]. Text emotion recognition is shown in Figure 2.

Figure 2

Text emotion recognition.

As illustrated in Figure 2, the primary operation of text emotion recognition is to preprocess the text, then extract features, and finally perform emotion recognition.

The specific ways to preprocess text include word segmentation, part of speech tagging, syntactic and semantic analysis, etc., followed by text feature extraction and fed into a classifier for emotion classification [28,29]. Text feature extraction and representation is the process of quantifying word vectors extracted from text to make it easy for computers to recognize. There are many popular algorithms for extracting statistical features at this stage. For instance, one-hot coding, which is also one bit effective coding, encodes the categorical values as a binary representation. Although it is said to be able to handle discrete values and expand the features for the purpose of handling words well, there are some prominent drawbacks such as not considering the order between words, and so on.

Term frequency-inverse document frequency is a technique used in text analysis and information retrieval to evaluate the importance of a word in a document set. Term frequency-inverse document frequency (TF-IDF) is a simple and fast text representation method that considers word frequency. TF-IDF is essentially a weighted technique for suppressing noise, which calculates the importance of a character or word in the entire corpus by the number of times it appears in the text and the frequency of documents appearing in the entire corpus. The TF-IDF calculation formula is presented in formula (1):

Among them, I DF is the inverse document frequency of the weight parameter and T F is the word frequency.

3.2 Speech emotion recognition

Speech emotion recognition is a technology used to analyze and understand the speaker’s speech signal to determine the emotional state or content it expresses. The information conveyed by speech includes semantic information and acoustic information, and acoustic features contain a lot of key information that can be used as auxiliary semantic information. Different data in the audio modality include different emotional information, which has a certain impact on the findings of the final classification of emotion recognition [30,31]. The Mel spectrum is a feature proposed based on human auditory perception, and the Mel cepstral coefficient is nonlinearly related to frequency. The Mel spectrum in speech recognition is chosen to be borrowed for the purpose of better characterizing the energy distribution laws.

To standardize data, the purpose of standardization is to standardize each dimension of features to a specific interval range, so that the data distribution is not too scattered, and to transform dimensional expressions into dimensionless expressions. Standard deviation normalization is a data preprocessing method used to convert data into a standard normal distribution with a mean of 0 and a standard deviation of 1. Standard deviation standardization is the standardization method, and the standardized data conform to the standard normal distribution. The standardized calculation method is shown in formula (2):

α is the standard deviation of all data samples and μ is the mean of all sample data. The standardized data are more conducive to accelerating speed and improving recognition accuracy.

Power spectral density (PSD) is the distribution of signal power at each frequency point expressed in terms of the concept of density, a measure of the mean square value of a random variable, and the average power measure per unit frequency. The PSD is a real number and nonnegative. The PSD of a signal can be used to describe the energy characteristics of the signal as a function of frequency, while also normalizing data at different frequency resolutions, eliminating the influence of resolution.

If function x t represents the autocorrelation function of the signal, its PSD X can be written as follows:

In the process of collecting speech signals, they are generally stored in a discrete form, so when extracting features from speech signals, only discrete data can be analyzed and processed. Formula (3) is for correlation processing of continuous signals and is not applicable to discrete signals. Therefore, formula (4) is used to process discrete signals:

Among them, x n is a discrete value. Extracting the PSD characteristics of a signal can analyze the patterns and structures of the signal in the frequency domain and thus identify the signal. To solve the problem of some signal features not being obvious in the time domain, PSD can be utilized to extract the frequency domain features of the signal.

3.3 1DCNN-LSTM model

When processing longer signals, CNN cannot recognize longer physiological signals. When the sequence is long, using CNN makes it difficult for the gradients behind the sequence to propagate back to the previous sequence, resulting in the problem of vanishing gradients and sometimes even exploding gradients. To solve this problem, 1DCNN-LSTM emerged. The 1DCNN-LSTM model combines these two types of neural networks, typically on the input of text sequences. 1DCNN is used to extract local features, and LSTM is used to integrate sequence information, allowing the model to simultaneously consider both local features and contextual information in the text, which helps with more accurate emotion recognition.

1DCNN-LSTM is a deep learning model used for emotion recognition, typically used to process text, speech, or time series data. The schematic diagram of 1DCNN-LSTM is illustrated in Figure 3.

Figure 3

Schematic diagram of 1DCNN-LSTM.

As shown in Figure 3, 1DCNN-LSTM includes input layers, convolutional layers, pooling layers, LSTM, output layers, and output data.

LSTM has a memory unit that can store and access previous information, allowing the model to handle tasks that require memory and reference of historical information. LSTM is a special type of neural network primarily designed to address related issues that arise during training and effectively model the context of physiological signals. LSTM can enable the network to automatically learn and adjust the attention to the input of extended contextual features. This method can improve data quality, optimize feature vectors, and effectively highlight key emotional information. By calculating input gates, forget gates, cell states, and output gates, high-level emotional-related features are obtained.

The main inputs in the input gate are x t and h t − 1 . Among them, x t represents the current input of the cell, and h _t−1 represents the previous cell state. These parameters form a weight matrix that determines how much new information is added:

The forgetting gate mainly determines how much old information new cells need to be discarded and unwanted information.

The output gate contains the current input, the previous hidden layer state, and the current cell state. These parameters form a weight matrix that determines which information is output, as shown in formula (6):

LSTM is also unable to correctly learn and process certain very long or continuous time series, which are not prepartitioned into appropriate training subsequences with clearly defined starting and ending points. The problem is that a continuous input stream may ultimately lead to an unlimited increase in the internal values of the unit, even if the repetitive nature of the problem indicates that they should be reset occasionally.

3.4 2DCNN-LSTM model

Different modalities of data, such as text and speech, contain different representation information. Research on multimodal emotion recognition requires the fusion of information from different modalities to achieve the synergistic effect of information from each modality [32,33]. The core idea behind the LSTM architecture is the storage unit that can maintain its state over time, and the nonlinear gating unit can adjust the speed of information flow into and out of the unit. LSTM is now applied to many learning problems, whose scale and properties are completely different from the original problems.

2DCNN-LSTM is a combination of 2DCNN and LSTM. Its core idea is to use two-dimensional CNNs to extract local features of text and speech and use short-term memory learning methods to obtain the temporal and contextual relationships of text and speech.

2DCNN is a deep neural network primarily used for processing high-dimensional data. In the field of emotion recognition, text data are generally represented as vectors. 2DCNN extracts local features from vocabulary information, such as phrases and sentence structures, through convolutional operations. Due to the diverse manifestations of emotions, these local features are crucial for emotion classification, as different emotions are often associated with different expressions. The convolutional layer formula is expressed as follows:

Among them, f I W × k represents the activation function; ( i , j , k ) represents input data; and b represents the bias term. Convolutional layers are utilized to extract features of input signals.

LSTM is suitable for processing time series data. In terms of emotion recognition, LSTM can assist models in better understanding the tense of text, such as the order of words, contextual relationships, etc. This is the key to distinguishing different types of emotions, as the same word has different emotional meanings in different contexts.

2DCNN-LSTM generally includes a complete connected layer, which effectively fuses the features of 2DCNN-LSTM to obtain emotion recognition results based on deep neural networks. Activation functions are often utilized to output a probability distribution for each emotion type. The output layer formula is expressed as follows:

The 2DCNN-LSTM model can be applied to emotional analysis of social media, public opinion monitoring, emotional analysis of customer service, and mental health. It can help businesses understand the level of customer satisfaction or help healthcare workers determine the emotions of patients.

4.1 Single modal recognition accuracy

4.1.1 Comparison of text emotion recognition

The experimental data used data from students with psychological problems mentioned earlier. The actual psychological problems of students are presented in Table 4.

Table 4

Actual psychological problems of students (multiple choice)

Psychological issues and average	Grade 1	Grade 2	Grade 3	Grade 4
Depressed	47	62	50	83
Anxiety	50	47	64	57
Anger	54	55	47	60
Fear	55	75	51	72
Lonely	41	63	67	61
Disappointed	60	55	61	75
Average	51	60	57	68

As displayed in Table 4, the average number of actual psychological problems among students in grades 1, 2, 3, and 4 was 51, 60, 57, and 68, respectively.

Only text emotion recognition was performed on students, and they were asked to write down their emotions. The emotion recognition results of 1DCNN-LSTM and 2DCNN-LSTM are illustrated in Figure 4 (the horizontal axis shows different psychological states, and the vertical axis shows the number of errors).

Figure 4

Emotion recognition performance of 1DCNN-LSTM and 2DCNN-LSTM.

As illustrated in Figure 4, the recognition error of 2DCNN-LSTM for the number of students with different psychological states was within 4, while the recognition error of 1DCNN-LSTM for the number of students with different psychological states was mostly above 4.

The recognition error of 2DCNN-LSTM for the number of students with different psychological states was smaller than that of 1DCNN-LSTM for the number of students with different psychological states. 2DCNN-LSTM can extract more features, better capture text, and improve recognition accuracy.

2DCNN-LSTM can process multiple types of data simultaneously, but 1DCNN-LSTM can only handle local data, so it requires a high amount of data. When the sample size is insufficient, the role of the 1DCNN-LSTM model cannot be demonstrated.

4.1.2 Comparison of speech emotion recognition

The average recognition accuracy of 1DCNN-LSTM and 2DCNN-LSTM for speech emotion recognition of students is illustrated in Figure 5 (where the horizontal axis is different grades and the vertical axis is accuracy).

Figure 5

Accuracy of speech emotion recognition using different methods.

As presented in Figure 5, the left side of Figure 5 shows that the average recognition accuracy of 1DCNN-LSTM for different psychological states was 65.7, 68.8, 66.2, and 64.3%, respectively; the right side of Figure 5 shows that the average recognition accuracy of 2DCNN-LSTM for different psychological states was 85.3, 83.9, 84.6, and 88.0%, respectively.

It can be seen that in speech emotion recognition, the average recognition rate of 2DCNN-LSTM was relatively high, while the average recognition rate of 1DCNN-LSTM was relatively low.

4.2 Multimodal recognition accuracy

1DCNN-LSTM was used to simultaneously recognize students’ text and speech, that is, to perform multimodal recognition. The situation is shown in Table 5.

As presented in Table 5, the average recognition accuracy of 1DCNN-LSTM for first-, second-, third-, and fourth-grade students under multimodal conditions was 76.12, 78.96, 78.12, and 77.89%, respectively.

The recognition accuracy of 2DCNN-LSTM under multimodal conditions is presented in Table 6.

In multimodal conditions, the average recognition accuracy of 1DCNN-LSTM and 2DCNN-LSTM for fourth-year university students was 77.89% (data in Table 5) and 92.48%, respectively. The average recognition accuracy of 2DCNN-LSTM for fourth-year university students was 18.7% 92 .48 − 77 .89 77 .89 = 18 .7% higher than that of 1DCNN-LSTM for the fourth-year university students.

Combining Tables 5 and 6, it can be seen that the recognition accuracy of 2DCNN-LSTM was generally higher than that of 1DCNN-LSTM.

For the emotion recognition task of college students, the 2DCNN-LSTM model can more accurately capture multimodal information compared to the 1DCNN-LSTM model and therefore performs better in processing multimodal data [34,35]. However, it should be noted that in practical applications, it is necessary to choose a suitable model structure based on the characteristics of the scene and data and undergo strict testing and verification to ensure the performance of the model. The psychological counseling model based on 2DCNN-LSTM usually combines multimodal data (such as speech and text) for emotional analysis and counseling support. This model can more comprehensively capture the emotional state of the counselor, thereby providing more accurate counseling support. By analyzing the voice emotions of the counselor, their emotional state can be better understood, and personalized and targeted counseling advice can be provided.

4.3 Psychological counseling effectiveness

Traditional psychological counseling often relies on manual evaluation and is easily influenced by personal cognition and experience. Emotion recognition technology utilizes objective calculation methods to analyze and identify the emotions of consultants, reducing the influence of subjective factors and achieving more objective and less biased emotional evaluations. Emotional recognition can provide real-time analysis of the emotional state of college students and provide information to psychological counseling institutions. Through this approach, psychological counselors can timely understand the psychological changes of students and develop corresponding psychological counseling strategies based on this, providing personalized psychological support for students.

After 2 months of traditional psychological counseling (subjective judgment by psychological counselors) and psychological counseling based on emotion recognition technology, the improvement of students’ psychological problems is presented in Figure 6 (the horizontal axis depicts grade, and the vertical axis depicts the number of students).

Figure 6

Improvement of psychological problems among students under different psychological counseling.

As illustrated in Figure 6, the left side of Figure 6 shows the improvement of psychological problems among students after traditional psychological counseling, which has decreased compared to the number of students who did not receive psychological counseling at the beginning. However, there are still situations where students in their fourth year of university still rank above 30. The right side of Figure 6 shows the improvement of psychological problems among students after psychological counseling based on emotion recognition technology. The number of students with psychological problems is within 20.

It indicates that the improvement effect of psychological counseling based on emotion recognition technology on students’ psychological problems is better than that of traditional psychological counseling.