Speech emotion recognition using long-term average spectrum

2.1 Energy

The energy is obtained as the sum of squares of the signal amplitudes, and it is proportional to Pa 2 s [17,25] because x ( t ) is the amplitude of sound, given in Pa. Then, the energy is defined as follows:

This feature has been used for vowel recognition from speech signals, phoneme speech segmentation [26], and the detection of voice presence in high-quality speech signals.

2.2 Fundamental frequency

The fundamental frequency ( F 0 ) is the vibratory frequency of the vocal cords. The number of air pressure oscillations per second determines the F 0 . The pitch is related to F 0 , as it is how F 0 is perceived by our ears. The F 0 is usually used for tasks such as gender identification and as an important indicator of emotional status.

2.3 Mel spectra

Mel spectra are obtained by passing a speech signal through a triangular filter bank, where the result is given as a vector of spectra. Each spectrum in the vector is obtained by an individual filter, where the size of the vector is the same as the number of filters in the bank (Figure 1).

Figure 1

Extraction of a spectrum through mel filter banks.

The axes of the triangular filters are distributed according to the nonlinear mel scale proposed by Stevens and Volkman [27], where the borders of the filters lie on adjacent frequency axes. These authors proposed that the perception level with respect to a frequency that is heard follows a logarithmic scale expressed by the equation:

This speech codification has been used in various types of speech research, such as speaker recognition and speech segmentation [28].

2.4 Linear predictive coding (LPC)

LPC is a scheme for representing voice production. The LPC models a filter, through which sound source passes. The filter represents a function for obtaining speech parameters such as pitch, formants, and vocal tract [29]. It is helpful in terms of representing signals in a compressed way for net transmissions, thereby removing redundancy. The model is also known as linear prediction (LP), where a speech signal is approximated as a linear combination of its previous values [30].

The MFCC measures the spectral variations of frequencies, thus enabling the treatment of the frequencies in the time domain [31] as the cepstrum is a periodicity measure of frequency. This approach to speech codification has been used in many applications, such as the detection and classification of infants’ cries [32,33].

2.5 MFCCs

MFCC is the most typical method used to extract spectral information from audio data. MFCCs are based on the mel scale in the frequency domain, which follows the distribution of the human ear. This codification scheme for audio has been applied to a wide variety of studies into speech processing due to its robustness in regard to speaker and recording variability [30].

MFCC extraction starts by applying a pre-emphasis filter to emphasise the higher frequencies, as these frequencies are reduced in the recording process; at the same time, some effects from the glottal source are also reduced. The spectral slope is increased by 6 dB/octave over the frequency F as the input parameter. A Hamming window is used to reduce spectral distortions by splitting the continual signal into frames of 20 or 30 ms (typically), overlapping by 10 ms [34].

The MFCC features are derived from the fast Fourier transform (FFT) spectral magnitudes passed through the mel filters bank, and the logarithm of the energy on each filter is computed before applying a cosine discrete transform to produce the MFCC feature vectors.

2.6 Long-term average spectrum (LTAS)

LTAS represents the averaged spectral information on the vocal tract over the frequencies. The process of extracting LTAS from a speech signal involves applying framing with overlapping and computing its spectral power and then obtaining the spectral average. A Hamming window is then applied to each frame to smooth the edges, and an FFT is applied to these frames [35]. The LTAS is obtained using a frequency step (bin width) that ranges from zero to the maximum frequency of the speech signal. At each step, an averaged spectrum is computed. The size of the LTAS vector is the same for all signals with the same sampling frequency. LTAS represents the power spectral density in dB/Hz and is frequently expressed in logarithmic form as follows:

where P ref = 2 ⋅ 1 0 − 5 Pa. Then, this logarithmic power spectral density is the quantity stored in an LTAS [25]. This power spectral density can be calculated from the sound complex spectrum x ( t ) in the time range T = t 2 − t 1 , measured in Pa 2 ∕ Hz :

The average sound power is obtained in the range time ( t 1 , t 2 ) as follows:

LTAS is the average of the spectrum over a long speech signal and is used to describe the resonance characteristics of a speaker; in general, it is useful for recognition of a speaker [35], including gender, age, and diseases [36] and forensic usage [37]. In general, LTAS encodes speech signals with the voice quality [38].

In order to demonstrate the simplicity of LTAS representation, the spectrum and LTAS were extracted from the sentence “Say the word ‘wire”’, spoken by a woman, and both kinds of signals were graphed. The spectrum shows a high level of variability in its values over the frequencies (Figure 2), and a summarised version of the spectrum is therefore used, such as LTAS (Figure 3).

Figure 2

Spectrum for the sentence “Say the word wire.”

Figure 3

LTAS for the sentence “Say the word wire.”

3.1 MFCC, LPC, LPCC, and PLP for emotion recognition

MFCC, LPC, and PLP were described and compared in terms of their individual effectiveness for emotion recognition by Palo and Chandra [15] using three and four classes (boredom, anger, sadness, and surprise). The results indicated that an MFCC-based method could achieve the best accuracy of 80.00–83.20% followed by a method based on PLP speech codification (70–74.3% accuracy), and finally, a framework based on LPC (48.60–56.20% accuracy). The authors concluded that methods based on MFCC gave the best performance.

3.2 Excitation source analysis with MFCC

Pravena and Govind [40] experimented with the IITKGP-SESC and EmoDb emotional databases. The speech codification used was MFCC, and a Gaussian mixture model (GMM) was used as a recognition system. With EmoDb, four classes were considered (anger, happiness, boredom, and neutrality), and 80% of the total utterances were tested. The best performance in terms of emotion recognition was obtained using 256 Gaussian mixtures, where the average accuracy was 73.28% for the classification of these five emotions. In this study, 39 MFCC coefficients were extracted from 20 ms frames with an overlap of 10 ms, and information on the MFCC velocity and acceleration coefficients of each emotion utterance was obtained.

3.3 Mixed time- and frequency-domain features

In work by Karimi and Sedaaghi [16] and Sundarprasad [39], time and frequency-domain features were used. The statistical features of pitch, energy, MFCC, LPC, and PLP gave performance of 82.6% using Bayes classification on the EmoDB database [16], while GMM and KNN gave accuracies of between 79.52 and 75.61%, respectively. The zero-crossing rate, spectral values, MFCC, and chroma coefficients were used by Sundarprasad et al. [39]. The implementation and results of those works will be presented in Section 5.

4.1 Database

The signals considered in our experiments were obtained from the TESS database [41], which was recorded at Toronto University with two speakers (actresses aged 26 and 64 years). The dataset represents seven emotions: anger, disgust, fear, happiness, pleasant surprise, sadness, and neutrality. Both actresses were selected from the Toronto area and were university-educated with musical training. They both recorded the same number of samples, giving a total of 2,800 signals in the set. The database is balanced, and the class distribution is shown in Table 1.

4.2 Speech signal processing software

In this study, the LTAS was extracted from speech signals using PRAAT software version 6.0.39. The classifiers cited earlier, implemented using the Waikato Environment for Knowledge Analysis version 3.7.9, were used for emotion recognition.

4.3 Classification algorithms

The algorithms with the best performance in our experiments, i.e. MLP, KNN, RF, SL, and SMO, will be briefly described in this section.

4.3.1 MLP

MLP is an architecture inspired by a biological neural network using two or more neurons called perceptron, which is organised in the form of layers [42], where each perceptron in a layer is connected to a perceptron in another layer.

Feedforward connections are used between the perceptron layers; the signal flows in only one direction, from the input layer to the output layer, passing through the hidden layers. Each perceptron is fully connected to each perceptron in the next layer, where connections have individual adjustable weights that simulate synapses (Figure 4).

Figure 4

Structure of the MLP.

A perceptron function could be defined by (6):

(6) f = ∑ i = 1 n ⋅ W i n ⋅ X i n + W Θ = W T ⋅ X .

The MLP is trained using the backpropagation algorithm, where the error is propagated from the output to inner nodes, and the weights are then adjusted using a descending gradient, in order to minimise the output error. The error e in the jth neuron, where d is the desired value and y is the output value, is determined by

(7) e j ( n ) = d j ( n ) + y ( n ) .

A variety of neural nets have been used for emotion recognition with successful results [13,14,43]. MLP has been used in this area; many works in the literature survey their structure [15] and have highlighted their relatively high performance. The principal characteristics of MLP are (i) the use of an activation function for each neuron, (ii) a minimum of three layers (input, hidden, and output), and (iii) the learning process.

4.4 Classifier settings

The optimal settings of the principal parameters of the classifiers used in our experiments are detailed here.

The KNN was set based on the Euclidean distance, with no distance weighting, with one and three neighbours, denoted here as KNN and KNN3, respectively. The parameters for the MLP were set to 0.3 for the learning rate, 0.2 for the momentum, and 500 epochs. A single hidden layer was used in the experiments. A criterion of (classes + input size)/2 was used to determine the number of neurons in the hidden layer; for instance, using seven classes and 70 Hz for LTAS extraction gave 175 features and seven classes, where the number of neurons in the hidden layer was 91.

The SMO used multinomial logistic regression where the complexity parameter was set to one, and an epsilon value of 1.0 × 1 0 − 12 was used for the round-off error. The main parameters used for the RF classifier were as follows: bagPercentSize (percentage of the training set size) was set to 100, batchSize (number of instances to be processed) was set to 100, numIterations (number of trees in the forest) was set to 100, numExecutionSlots (number of threads used) was set to one, numFeatures (number of features chosen randomly to build the trees) was set to zero, which apply equation (12) to obtain the number of features (attributes). In experiments TotalAttributes is 175, and then eight features were used. The parameter maxDepth, which denotes the depth of the trees, was set to zero (for unlimited depth).

For the SL classifier, the batchsize was set to 100, and the parameter heuristicStop was set to 50 as the number of iterations if no new minimal error was found. The parameter maxBoostingIterations was set to 500, and its value was directly proportional to the dataset size. The parameter weightTrim was set to zero, meaning that no trim beta was used. The NB and BN classifiers did not require parameter settings for these experiments.

4.5 Framework

The speech emotion signals were read using PRAAT software, and the LTAS Praat function was applied, which required the parameter of the bandwidth (BW). We used BW values of 70, 100, 200, and 300 Hz, respectively (see experiments for details), which was done using a PRAAT script. In order to start the LTAS extraction, the speech signal was split into frames and windowed, where the last was used to minimise spectral leakage. A FFT was then applied to each window. The entire signal was split using a bin width or a frequency step in order to obtain the logarithmic power spectral density from the frequencies, expressed as dB/Hz: 2 × 1 0 − 5 Pa 2 [25]. A set of LTAS values were returned for each speech signal, and the LTAS vector was then built, with labels for the emotions specified in the database. A database with LTAS values was built with a cardinality equal to the number of emotional speech signals in the TESS database; this value was 2799, as one file in the “Fear” class could not be read correctly and was ignored. The LTAS database was then processed in WEKA, using different classifiers with their default parameters. A 10-fold cross validation process was used for the classification. The entire speech emotion recognition framework is illustrated in Figure 5.

Figure 5

Proposed speech emotion recognition framework.

A brief description of the algorithm used in the framework is provided in the following:

1. Begin

2. For each file in Data Set:

   1. Load the audio file and resample to 44 kHz.

   2. Perform frame blocking:

      1. Divide the audio signal into overlapping frames, reducing spectral distortion.

      2. Set frame_size = 20 ms and frame_overlapping = 10 ms.

   3. Apply a Hamming window to each frame to reduce spectral leakage.

   4. Compute the Short-Time Fourier Transform (STFT) to obtain a matrix of spectral values:

      1. STFT[k, n] = FFT(Frame[n]), where k is the frequency bin and n is the frame index.

      2. Compute the magnitude spectrogram STFT[k, n].

   5. Calculate the LTAS:

      1. LTAS[k] = 1 N ∑ n = 1 N ∣ STFT[k,n] ∣ , across all frames.

   6. Generate the LTAS vector (feature vector).

3. End For each

4. With the LTAS[k] vector of each file, train the classifier using ten-fold cross-validation.

5. Predict the emotion of the input audio file using the trained classifier.

6. End

On the other hand, the complexity of the method will be defined by abstracting the previous algorithm as follows:

1. Load N files and resampling, with complexity O ( N × L ) , where:

   1. N is the size of the data set (number of files).

   2. L is the average frame length of the audio.

2. Compute the STFT:

   1. Blocking: Divide the signal into frames without overlap, with complexity O ( N × L ) .

   2. Windowing: Apply a window to each segment (frame), with complexity O ( N × L ) .

   3. STFT computation: Perform the STFT with complexity O ( N × L log ( frame_size ) ) , where:

      –frame_size is the length of the frame in milliseconds.

3. Calculate LTAS: Averaging the spectral data of each division (frame), with complexity O ( B × L frame _ size ) , where:

   1. B is the bin width of frequency.

   2. The complexity of N files is O ( N × B × L frame_size ) .

4. Generate the LTAS vector for N files using B bins, with complexity O ( N × B ) .

5. Perform tenfold cross-validation: assuming C as the complexity of training the classifier, the overall complexity is O ( 10 × C ) = O ( C ) .

6. Predict emotion: the complexity depends on the classifier used.

Summarising the computational complexity analysis in the process:

1. SFFT computing: The computational complexity is given by

   O ( N × L log ( frame _ size ) ) ,

   where

   1. N is the number of files,

   2. L is the average frame length of the audio,

   3. frame_size is the length of each frame.

2. LTAS averaging: The computational complexity is as follows:

   O N × B × L frame _ size ,

   where

   1. B is the bin width of the frequency,

   2. N is the number of files,

   3. L is the average frame length of the audio,

   4. frame_size is the length of each frame.

The method takes advantage of the low computational load of LTAS processing, unlike other schemes such as MFCC, which require additional steps, such as applying the Mel filter bank, logarithmic transformation, and discrete cosine transformation.

5.1 Feature extraction

The feature vectors were created by extracting the LTAS from the audio signals. LTAS extraction requires a BW frequency, as this defines the number of values (descriptors) used to characterise the audio signal. BW values of 70, 100, 200, and 300 Hz were used in this case, giving 175, 122, 62, and 40 LTAS values, respectively. In our experiments, the larger the value used for the LTAS, the better the performance.

5.2 Comparative analysis

Algorithms based on trees, functions, SVM, logistic regression, Bayes, KNN, and NNs were tested, and a tenfold cross-validation method was applied. Table 2 presents the performance of each classifier, using a defined number of descriptors (values) as indicated in the header. In these experiments, the MLP gave the highest performance for each BW frequency in the sampling process. Increasing the LTAS values improved the performance of the SMO and SL classifiers, particularly at BW values of 70 and 100 Hz, where SMO performed just slightly below MLP.

The KNN classifier was robust over different attribute sizes (first row), giving similar performance. The RF among the set of trees algorithms, is outstanding with an accuracy of between 94 and 98%. The Bayesian algorithms gave the lowest performance on emotion classifications using LTAS. Although a set of other algorithms were tested, only the best performing of each type are presented here. For each classifier, the average accuracy for the different BWs was computed to determine the overall performance across the range of BWs. The last column in Table 2 contains the average accuracy, and it can be seen that MLP was the best classifier for emotion recognition using the TESS database, followed by KNN (with a difference of approximately 2%), and then the RF, SMO, and SL classifiers of Figure 6.

Figure 6

Averaged accuracy for each classifier over values of 175, 122, 62, and 40 for the LTAS per signal.

The classifiers based on Bayes gave the lowest performance. In the discussion section, we describe state-of-the-art works where relevant results were obtained using variants of NN, SVM, and KNN. The following works involved experiments with the TESS database, where a subset of features as MFCC, filter bank energies (FBEs), energies, zero-crossing rates, and chroma coefficients were used. A comparison of performance between these works and our framework is presented in Table 3.

Experimental results with a subset (1,369 samples) of the TESS database were reported by Praseetha and Sagil [13], who used 26 FBE coefficients per frame. The total number of values used per signal was not specified, but long frames are typically 20 ms (overlapped by 10 ms), with 2,600 values per second. The best performance was 95.82%, achieved using a GRU, an improved version of an RNN. In our experiments, the best performance was obtained using a common kind of NN called an MLP.

Experiments were also reported by Ugu Bastug [19], who used coefficients such as delta values, acceleration, mean, standard deviation, maximum and minimum of MFCC, and the median, standard deviation, and acceleration of the FBEs. In total, a matrix of values with 49 rows and 13 columns was used, although it was not specified whether this matrix represented the complete audio signal or a frame. Assuming that the matrix represents the complete signal (2 s long), this would give 637 values. In summary, signal processing for the numeric characterisation was based on MFCC and FBE codification, in addition to the statistical measures cited earlier. The KNN classifier was used in these experiments similarly to our experiments, and one of the best results, reported in Table 3, was obtained using KNN.

Experiments with the TESS database, SVM classifier, and MFCC as speech features for speech emotion recognition were carried out by Zafar Iqbal and Farooq Siddiqui [56]. A subset of TESS samples containing four emotions (anger, happiness, disgust, and pleasant surprise) were used for prediction purpose, and 50 and 6 samples were used for each emotion, respectively, for training and testing. These authors argued that with a subset, it was possible to predict the behaviour of the entire database. The MFCC extraction process used rectangular segments of 10 ms (100 per second), and the number of coefficients was 12, giving an estimate of 1,200 values of MFCC per second.

This work was used as a comparison with the results of our method, as the same database was tested (TESS) with different features (our method uses LTAS). When used for emotion recognition, MFCC and LTAS features have the following differences: (i) a method based on LTAS uses 62 to 175 values per signal, as shown in Table 3 (the number of values varies as a function of the BW frequency in LTAS extraction process), whereas the method based on MFCC uses 1,200 values per s; (ii) the accuracy of correct emotion classification in our method ranges from 98.05% using 62 values per signal to 99.24% using 175 values per signal, when the MLP classifier is used, compared with an accuracy of 97% with the method of Zafar Iqbal and Farooq Siddiqui method, which is based on SVM. Our method achieves an accuracy of 98.42% using SMO, a kind of SVM.

Sundarprasad [39] experimented with the entire TESS database, where 34 values of the speech signal were extracted after the application of audio analysis. Some of the features used were MFCC, energy, measures of spectral coefficients, zero crossing rate, and chroma coefficients, with a total of 11 features. Principal components analysis (PCA) was applied to reduce the dimensionality from 34 to 25. The classifier used was SVM, and in a similar way to our experiments, the third best performance was obtained using SMO, an improvement on the SVM classifier, which achieved an accuracy of 98.42% correct classification, using only 175 values for the entire signal.

The work presented in [55] involved the Spanish databases EmoSpanishDB and EmoMatchSpanishDB. MFCCs were extracted from the audio files, with 13 per frame (with frames 20 ms long). The mean, first and second derivative, spectral centroid, and spectral contrast were computed from the MFCC values, and lineal spectral coding (LPC) values, intensity, and fundamental frequency (F0) were used in this method. The best results were obtained from SVM, with an accuracy of 56.04% for recognising the emotions of disgust, happiness, fear, sadness, surprise, anger, and neutrality.

Features are generally high dimensionality, as can be observed from Table 3. There have been research studies in this area [1], where architectures based on DNNs were developed in order to extract, reduce, and homogenise features. Features such as IS10, MFCC, and eGemspd were used and tested using the KNN, logistic regression, RF and SVM classifiers. In our experiments, we also found that these algorithms were the best for the task of emotion recognition. The main difference lay in the order of their performance, as Jiang [1] reported that the SVM achieved the best performance, whereas the MLP was superior in our experiments.

In summary, the experiments carried out in this work yielded similarities with the classifiers applied in state-of-the-art schemes on the TESS database was used. We considered the classifiers NN (GRU), SVM, and KNN. We also compared different speech codifications (MFCC, FBE, chroma, and LTAS) using the TESS database. In Table 3, which shows the results for our method based on LTAS, the number of values extracted for the entire signal is denoted with a + sign, and the number of values used for the other methods for each frame of 20 ms is denoted by a * sign. For the entire signal, the number of values used for emotion recognition, in the state-of-art methods, is higher.