Analysis of the sports action recognition model based on the LSTM recurrent neural network

3.1 Construction of LSTM-based human motion recognition information acquisition and processing system

In order to recognize the sports action of the human body, the feedback of the system has to be considered, and the recognition results have to be timely. Then, the research based on the system will be different from the monitoring system of health, fall, etc. The architecture of other systems will only propose three layers, and the processing system in motion recognition is relatively complete [12]. Moreover, the recognition of the human body’s sports action relatively only needs two layers because the data application in the extraction has been handled better, the missing layer that is the application of the data, as shown in Figure 1.

Figure 1

Motion processing architecture.

In Figure 1, in the data collection layer, the placement of motion sensors is designed to collect human motion data, including acceleration and angular velocity. The collected data can go to the data pre-processing, where the initial cleaning is done to get the formal data. The data computation layer is able to create and store the data in a database, which is the core of the system [13]. Finally, at the computation layer, the data are further processed according to the requirements to output meaningful data of the motion type and re-store the labels. Once the model computing layer produces meaningful data, there will be situations where the human body is outside its typical range of motion, which can lead to significant data fluctuations and the appearance of extreme values. The appearance of extreme values is not only lossy to the instrument, but also yields inaccurate results. The data need to be normalized as in Eq. (1):

After performing the normalization process as in Eq. (1), a set of data compressed to the range of 0–1 can be obtained, and such an operation can also avoid the appearance of outliers σ . However, to obtain the transformed data, feature capture is required. The feature capture includes both the maximum and minimum values as well as the standard deviation, which is calculated as in Eq. (2):

In Eq. (2), “mean” refers to the mean of the data, and the standard deviation “std” is calculated to measure the dispersion of the data from the mean. The skewness and kurtosis of the data can also be found by using Eq. (2), but the operation of the support vector machine (SVM) can only be done using the Euclidean distance. The core of SVM is to obtain a super-shared plane, which can not only solve the longitude catastrophe that often occurs in traditional algorithms but also avoid the local overdose problem [14]. According to the Lagrangian multiplier, it can be transformed while converting the original to the dual, and the dual Lagrangian function is as follows:

where 0.5 ∥ w ∥ is the distance between the support plane w x i + b and the hyperplane is obtained when w x i + b = 0 . This Lagrangian function can be derived, and only partial derivatives can be found. When the derivative function takes the value 0, the intermediate state function is obtained. The intermediate state function can be substituted into Eq. (3) to obtain Eq. (4):

However, Eq. (4) is only applicable to linearly divisible cases. In practice, the existence of linearly indivisible cases should be considered. Then, kernel norm minimization (KNM) can be applied. KNM can restore the observation matrix to the original matrix, as shown in Eq. (5):

where ∥ X ∥ ⁎ is the minimized kernel parametrization, the observation matrix is X , the original matrix of X is Y , and the estimated approximation of X is referred to as Y . The threshold value of X is denoted as λ , and the diagonal elements are solved as in Eq. (6):

where ∑ i i represents the soft threshold function of Y in Eq. (5), which can be applied to the LSTM. The LSTM network structure has output gates, forgetting gates, and cell states. The cell structure can record the time series and iteratively update them by summing. It not only avoids the large impact between data but also preserves the state of the time step [15]. Because of the variety of gates, the gradient does not explode when the weights are updated, as shown in Figure 2.

Figure 2

LSTM flow chart.

From Figure 2, after the input data enter the input gate, the qualified data are outputted after strict screening by the cell to forget the unqualified data. Cell screening is the most important process, the core of which lies in the decoder. The decoder is able to focus on the aggregated historical information based on time by fading and predicting attention [16]. The decoder is based on machine translation, and the end of each decoding is the beginning of the next decoding and the decoding principle is as in Eq. (7):

where the output decoding result is denoted as y 1 ∣ { y 1 , y 2 , … , y t − 1 } , C . From Eq. (7), its form is similar to Eq. (3) to a large extent, and the two can be combined to obtain a more optimal solution.

3.2 Improving LSTM by gradient descent in sports information acquisition system

For LSTM, although it had experienced a low period in the last century, people gradually found that the traditional algorithm needed to go through manual feature extraction, and certain fields could not be involved. With the development of gated recurrent unit (GRU) in computing efficiency, LSTM has been more widely used in image recognition due to its ability to mine time series data. LSTM is like the human nervous system, which consists of many neurons. This research is based on the variant of LSTM, in which the structure of weight processing is added among the structure of each individual neuron, which can avoid the problem of excessive weights in the working process of LSTM so that LSTM can be more fully exploited, as shown in Figure 3.

Figure 3

Improved LSTM process.

From the process in Figure 3, at moment t, the hidden layer accepts the node M_t and the input value X_t, which are input to the weight layer after the calculation of the hidden layer and are processed after merging with the node M_1 and the input value X_1 accepted by the weight layer at moment t ₁. The processed weight information is input to the memory unit, which is able to identify some elements that should be forgotten, some elements that should be output, and the remaining elements that can be recovered by valves. One of the most widespread applications in the memory unit is the Sigmond function, as given in Eq. (8):

where σ ( x ) represents the Sigmond function. When the Sigmond function is used as the activation function, the function image is characterized by a small value-added and degraded value. Since the problem studied is one with a high number of classifications, the loss function used is preferably categorical cross-entropy, as in Eq. (9):

where the actual value is denoted as t , and the predicted values are called p . i , j are used to represent the classification where the actual and predicted data are located, respectively. When using categorical cross-entropy as the loss function, mean squared error (MSE) and cross-entropy (H) are operated as in Eq. (10):

where y i → , y i are given as the predicted and actual values of the model, respectively, and their values are always greater than or equal to 0. The model will be more accurate if it is optimized several times by the GDO method to keep it close to 0. GDO can optimize MSE and H by iteration, as in Eq. (11):

where the step size is represented using θ and J ( θ ) represents the objective function. MSO and H are able to minimize them and make the weights more appropriate after passing the GDO process. In the module of data pre-processing, sports are different from normal working life because of their elegance and beauty. Sports, because of their elegance and beauty, are always accompanied by great risks, so they are optimized for the temporal attention module [17]. The temporal attention module focuses on the SDPA with the following equation:

where the modeling query vector is represented by Q , and the meaning of K , V is to store the vector. From Eq. (12), three eigenmatrices of the input quantity X can be derived, and these three eigenmatrices are subjected to a scaling operation. Then, the normalization should be performed, and the normalization method is as in Eq. (13):

where Y represents the weighted feature matrix of Y , and the meaning of Z is the feature expression of X , Y obtained by residual concatenation. For the aggregation and expression of historical information, it is divided into two parts, which are query and storage. Relating the present moment to some future moment, the two parts of information are fed into the vector for representation, and the two features will no longer be independent. Since the input sequence tends to lose a part of the information, an attention mechanism is introduced, as in Eq. (14):

where the feature mapping is denoted as pos , and the position vector generated by pos is d pos . The data set generated using the attention mechanism is able to preserve the temporal order and also to extract complex action data from sports. This allows us to improve the LSTM (GDO-LSTM) based on the attention mechanism, as shown in Figure 4.

Figure 4

Improved sensor diagram.

In Figure 4, the sensor collects an assortment of data sets during its operation, using a random data set, which is fed into a modified LSTM input gate. After going through the LSTM process once, it will go through a specially designed cell structure. The data that have undergone the cell structure are only expanded for their Lagrangian function, and then a hyperbolic tangent (Tanh) process is performed to judge whether the result should be forgotten or not. If yes, they are temporarily forgotten and Lagrangianized once more; if not, they are transported to the output gate and output as a qualified result.

4.1 GDO-LSTM sports monitoring system development and model parameter determination

This study selected a data set from Hainan University and selected 1,000 data items as the training set to train the model. The performance of the model was judged by the indicators of accuracy and error rate. In the recognition of sports movements, a three-axis magnetometer was used to calibrate sports movements. In the experiment, the specific hardware and parameters are shown in Table 1.

The collected data set required further processing to enable the studied algorithm to gain learning. For the processing of the data set, GDO-LSTM was used for iterative optimization. To verify its accuracy, a comparison with the traditional particle swarm optimization algorithm (PSO) was performed, and two images were obtained: the accuracy-training set image and the error rate-training set image, as shown in Figure 5 [18].

Figure 5

Comparison of (a) accuracy-training set image and (b) error rate-training set image.

From Figure 5, with the increase in the number of iterations, the accuracy of GDO-LSTM was lower than that of PSO until 200 iterations, and the error rate was higher than that of PSO. However, after the number of iterations reached 200 or more, the accuracy of GDO-LSTM was able to reach 1.1–1.5 times that of PSO. Although increasing the number of iterations increased the cost, the role of accuracy was crucial considering the scientific rigor. After the learning of the GDO-LSTM algorithm was completed, it had to be considered for the wearing of the motion sensor during the test. The motion sensor would measure three sets of data during the test: the acceleration of the object in the three axes of motion. The algorithm was applied to build a three-axis magnetometer and calibrate it using sit-to-stand and lie-flat, and the calibration diagram is shown in Figure 6.

Figure 6

Calibration of the three-axis magnetometer. (a) Sitting correction of three-axis magnetometer and (b) flat-lying correction of triaxial magnetometer.

From Figure 6, after the change from sitting to lying position, the acceleration of the X- and Y-axis would decrease due to the change in position, and the acceleration of the Z-axis would increase instead of decrease because of the change in height. After zeroing, the three-axis magnetometer had to be tested for sensitivity, and it was clear that there was a significant difference between it at rest and during weight lifting, as shown in Figure 7 [19].

Figure 7

Three-axis acceleration of static weight lifting. (a) Static acceleration–time and (b) weight lifting acceleration–time.

From Figure 7, the X-, Z-, and Y-axis acceleration were from top to bottom when Figure 7(b) was stationary. The Z-axis acceleration was very close to 0 because no change in height occurred. The Y-axis acceleration hardly changed during the process from rest to the highest point of the lift, which indicated that the tester did not move forward or backward. However, the X- and Z-axis accelerations changed considerably, with the mean value of the Z-axis above the X-axis, due to the fact that the weight lifting changed more for the up and down accelerations than for the left and right. By the moment of 97 s, when the weight lifting reached its highest point, the Z-axis acceleration reached its maximum, so defining such curves attributed to the weight lifting. The data processed by the oblivion gate could be loaded into the three-axis magnetometer to generate a baseline image for reference, as shown in Figure 8.

Figure 8

The base image of the three-axis magnetometer.

4.2 Experimental validation of the LSTM-GDO algorithm for sports action recognition

The calibration and sensitivity testing of a three-axis gyroscope, which works by measuring angular velocity and thus estimating the state of motion, is similar to that of a three-axis magnetometer. To be able to have objective sensor test results for sensor accuracy comparison, this study also explored the comparison of the accuracy of the three-axis magnetometer and gyroscope tests. It was first evaluated for simple motion, as shown in Figure 9.

Figure 9

Comparison between the three-axis magnetometer and gyroscope in skipping rope.

From Figure 9, for the three-axis magnetometer, the changes in the X- and Y-axis were very regular because the jump rope was repeating the same action. The changes in the Z-axis were not obvious because the jumping height was different, which was in line with the jump rope characteristics of the human body. For the three-axis magnetometer, because the angular velocity did not change much when jumping rope, it was reflected in the image that the measurement data did not change obviously, which led to the recognition effect as the action was not very good. To study the motion recognition of three complex motions for the sake of extensive experiments, in Figure 10, the curve was selected with the most obvious change when using the three-axis magnetometer and gyroscope test each time to analyze the accuracy of the two test instruments.

Figure 10

Extensive accuracy test of the three-axis magnetometer and gyroscope.

From the extensive tests of the accuracy of the three-axis magnetometer and gyroscope in Figure 10, in the skating motion recognition, both instruments performed very well. In contrast, among the recognition of swimming and shot put, the test accuracy of the three-axis magnetometer was obviously inferior to that of the three-axis gyroscope because the angular velocity changes in these two movements were not as obvious as the acceleration. To be able to analyze intuitively, the matrix shown in Figure 11 was established to be able to analyze more clearly the effect of the three-axis magnetometer and gyroscope in practical applications.

Figure 11

Three-axis magnetometer and gyroscope test matrix. (a) Extensive test matrix of three-axis dynamometer and (b) extensive test matrix of three-axis gyroscope.

The matrix in Figure 11 shows the recognition ability of the three-axis magnetometer and gyroscope. The test effect of the three-axis gyroscope was not very good, and there was a lot of confusion in the identification of swimming and skating. The three-axis magnetometer was accurate in the identification of four common sports most times, but there was some confusion between rope skipping and swimming, as well as shot put and skating, but it was enough to meet the equipment accuracy requirements.