Basketball action recognition by fusing video recognition techniques with an SSD target detection algorithm

2.1 Cropped frame construction based on SSD TDA

The SSD algorithm works by applying a certain number of predicted edges to the input image. The position of the target object is ascertained, and the output is produced by combining it with feature maps of varying sizes and dimensions. Then, the non-maximum suppression algorithm is utilized to eliminate the predicted frames that do not meet the expectations as well as duplicates, and finally, the true frames that match the original image are obtained. The network framework of SSD is shown in Figure 1 [19].

Figure 1

SSD network framework.

In Figure 1, the network framework of SSD consists of two parts: a truncated visual geometry group (VGG) and a CNN layer with decreasing scale. On feature maps with varying scales, the target is detected simultaneously. The input image is passed through the VGG network and convolutional layer for FE. The edge length of the input image is first converted to 300, and the ReLU function is activated by two initialized convolutional kernel (CK) computations. During the CK computation, the image boundary feature information may be lost as the image matrix data are gradually reduced. To preserve the boundary features, a complementary zero operation is performed on the image boundary, i.e., the image is filled with SAME. This padding ensures the integrity of the image during the computation process, allowing the edge information to be retained to output the final feature map. The result is then detected by extracting the feature map, and the detection process is realized by the computation of CNN. The total CKs required for the feature map are the product of the a priori frames and the classification categories, and the CKs used to determine the location of the bounding box (BBO) are the product of the number of a priori frames and 4. During the prediction process, each a priori frame is predicted to generate a BBO, and the a priori frame function of the effective feature layer is shown in Figure 2.

Figure 2

Feature maps and reference boxes. (a) Image with GT boxes, (b) 16 × 16 feature map, and (c) 8 × 8 feature map.

Figure 2 (a) shows the original images labeled with blue and red true frames. Figure 2 (b) and (c) show the schematic diagrams of the prediction frames corresponding to the two real frames, respectively. The remaining prediction frames are negative samples. Each target in the feature map corresponds to multiple prediction frames. The convolution operation analyzes the features within the prediction region to produce a classification confidence score for the target as well as mapping the boundary coordinates back to the four points of the original map. The degree of overlap between real and predicted frames is performed by calculating intersection over union (IoU). The IoU expression is shown as follows [20]:

In equation (1), A denotes the true frame and B denotes the predicted frame. The derived feature maps are increasingly abstract with an increase of CNN layers. The initial layer feature maps of the CNN contain detailed information about the image data, such as object borders. The neuron model in CNN only extracts features from local regions of the image and constructs higher level feature connections through these local features. SSD enhances the model’s ability to recognize objects of different sizes through the prediction frame. The prediction frame scale calculation formula is shown as follows:

In equation (2), s k denotes the prediction frame of the corresponding feature map. b min and b max denote the minimum and maximum ratio of the predicted BBO to the original image, respectively. k denotes the CNN layers and n denotes the feature maps. The height and width of the default box (DB) are as follows:

In equation (3), a r denotes the prediction box aspect ratio. h k a r denotes the height of the DB. w k a r denotes the width of the DB. Set S, then the extra prediction box is shown as follows:

In equation (4), s k ′ denotes the size of the additional prediction frames, at which point the corresponding position of the feature map contains six prediction frames. The center of the prediction frame is shown as follows:

In equation (5), i and j denote the position of the image in the rows and columns of the matrix, respectively. f k is the size of the k th layer feature image. The feature map can produce six feature boxes of different sizes. To reduce training time and minimize computational complexity, only four a priori frames are used in some convolutional layers, excluding default frames with an aspect ratio of 3. Too many default frames can cause the training process to become complicated. The effective feature layers that retained the feature maps thus avoided the use of redundant prediction frames due to the larger size of the corresponding prediction frames. During the prediction process of SSD, the system generates multiple BBOs from the prediction boxes. To prevent overlapping BBOs in the prediction results, the study uses the non-maximal value suppression technique algorithm to further refine the prediction results. This effectively reduces the redundant prediction boxes and improves the detection accuracy. First, the DBs must be filtered according to the set confidence threshold, and the BBOs with confidence lower than the set value must be filtered out. The study adopts SSD target detection technique to construct the prediction model of the network using the weight files obtained from the training of the VOC dataset to recognize the human body in a single frame image. The VOC dataset covers a total of 21 different categories such as vehicles, animals, and furniture. During the model recognition process, non-human body categories are labeled if they are detected and these detections are ignored to ensure that the model recognizes only the human body. Lastly, the boundary coordinates are taken from the image frames that the SSD model returned using the OpenCV library. This allows us to crop the original image and create the cropped frames needed for the study. The process is shown in Figure 3 [21].

Figure 3

Cropped frame process: (a) original frame, (b) SSD detection, and (c) crop frame.

Figure 3 shows the cropped frame generation process, which utilizes the SSD TDA to remove the invalid parts of the image, leaving only the parts that need to be recognized. To meet the training requirements of 3D-CNN cropped frames, the study tries to select frames with fewer interfering elements when collecting the dataset for basketball technical movement analysis to ensure the clarity and simplicity of the image frames. As a result, the cropped frames are made to contain almost no interfering elements, thus improving the accuracy and efficiency of model training.

2.2 3D-CNN-based video recognition technique

In the process of recognizing basketball technical actions, the input data are video frame sequences. The recognition process not only needs to analyze the spatial representation of the action, but also consider the temporal relationship between the actions in the action sequence. The 3D-CNN video recognition technique adds a temporal dimension to the traditional two-dimensional (2D) convolution, which is able to comprehensively analyze the video data in both temporal and spatial dimensions [22,23]. As a result, 3D-CNN is able to capture action images and process continuous dynamics in the sequence. The difference between 2D convolution and 3D convolution operations in processing video data is shown in Figure 4 [24].

Figure 4

Schematic diagram of (a) 2D and (b) 3D convolution operations.

Figure 4 (a) shows the 2D convolution operation. In Figure 4 (b), by analyzing the image sequence of three consecutive frames, the dynamic changes are successfully captured with the setting of a time span of 3. Similar to the principle of 2D convolution operation, 3D convolution also employs the weight-sharing technique to extract specific features from the video frame stack. To enhance the expression of features, in addition to expanding the CK, the diversity of features can be increased by optimizing the settings of CKs. The training effect of CNN is influenced by multiple aspects, including algorithm structure and hyperparameter configuration. Therefore, in order to enhance model training efficiency and reduce training time, the study employs multi-resolution enhancement based on the 3D-CNN network model. The improved multi-resolution 3D-CNN algorithm architecture is shown in Figure 5.

Figure 5

Dual resolution 3D-CNN algorithm architecture.

In Figure 5, the video clip used as input to the network has a resolution of 112 × 112 and a frame rate of 16 frames. The input process is divided into two parts: one part is the original frame (OF) image after video preprocessing. The other part of the image is the outcome of the SSD TDA processing, which is circled by the red border in the figure. Its size is 64 × 64 pixels. Images are usually synthesized from different proportions of pixel points in the RGB color pattern, which will increase the amount of input data if the original image data is directly fed into the NN [25]. In the recognition task of basketball movements, the practical utility of color information is not high and grayscale images can be used to reduce the amount of computation. However, even in grayscale images, each pixel point has a variation interval, which is sufficient to retain most of the gradient and other key information of the original image. As an example, a 16-frame image frame input is shown in Figure 6 [26].

Figure 6

3D-CNN network model architecture: (a) original frame 3D-CNN and (b) crop frame 3D-CNN.

Figure 6 (a) represents the 3D-CNN network architecture in the OF. Figure 6 (b) shows the network architecture of 3D-CNN in cropped frames. The input image resolution is first scaled to 112 × 112 for 16 consecutive frames and then processed through 5 convolutional layers and 5 pooling layers (PL). The size of the CK is set to be uniformly 3 × 3 × 3 with a step size of 1 × 1 × 1. The first PL uses a window of 2 × 2 × 1 and a step size of 2 × 2 × 1 in order to avoid premature reduction outside the temporal dimension. The pooling window and step size for the remaining layers are 2 × 2 × 2. In Figure 6(b), the image is input with a resolution of 64 × 64 pixels and 16 consecutive frames through 4 convolutional layers and 4 PLs. The CK and step size settings are the same as the OF network, and the window and step size of all PLs are 2 × 2 × 2. After the same two fully connected layer (FCL) processing, the final classification results are output through the softmax layer. To improve the training efficiency, the initialization weight approach is utilized to merge the time series information with the spatial properties of the image frames. The pre-trained 2D convolutional layer weights on ImageNet are used as the initial values of 3D convolutional weights. The mean value is initialized as shown in the following equation [27]:

In equation (6), W 2 D denotes 2D weight parameters. W t 3 D denotes the 3D convolutional weight parameter. T denotes the convolution layer timing information. The generalization of weight initialization can be achieved by the initialization method of proportional scaling. The constraints are

As shown in equation (7), this method is capable of adapting to diverse network architectures and parameter ranges. The method produces diverse results by incorporating randomness and dividing by an uncertain constant. This method allows the use of any combination of constant values as shown in the following equation:

In equation (8), α t represents the initial value of the weights. The initialization of the negative weights is then performed to determine the value of the submatrix by dividing it by a specific constant. In the case where the initial iteration step is 1, the value of this sub-matrix is larger than the other initialized values. The initialization process of negative weights is shown in the following equation:

In equation (9), the OF and cropped frame are processed using 3D-CNN networks of both resolutions to generate weight files as model parameters. The sequence of action frames is predicted in order to extract feature vectors, followed by feature integration for classification. Finally, fusion is performed based on the summing of feature maps to overcome the resolution difference problem. The final feature representation of the input action frames is shown in the following equation:

In equation (10), C denotes the final feature of the input continuous technical action frame. b and X i , j , d b denote the j th feature of a technical action in the i frame of type a and b , respectively. d denotes the feature dimension. Among them, 1 ≤ d ≤ D denotes the dimension of the feature map. After FE and fusion is completed, these output data will be used as input to support vector machine for the final action recognition task [28]. The research fuses an improved 3D-CNN video recognition technique with SSD TDA to propose a 3D-CNN-SSDBAR model. One benefit of this model is that it can effectively recognize basketball moves by utilizing both the quick target detection capabilities of SSD and the spatio-temporal FE capability of 3D-CNN at the same time.

3.1 3D-CNN-SSD model performance test

The study uses the Python 3.7 framework to write the code, and experiments are conducted based on the Ubuntu 16.04 operating system. In all, 32 is the batch size, 40 is the network epoch parameter, and stochastic gradient descent is the optimizer of choice. The 3D-CNN-SSD model is trained using different frame rates of the Kinetics dataset and UCF50 dataset, and the effect of different frame rates on recognition accuracy is compared. Table 1 presents the findings.

The outcomes illustrate that when the input frames consist of 16 continuous video image frames, the algorithm obtains the highest recognition performance of 93.1% in the Kinetics dataset and 92.6% in the UCF50 dataset (Table 1). To test the model performance, the 3D-CNN-SSD model proposed in the study is compared with the OpenPose algorithm and DWT for experiments on both training models, OF and cropped frame. The test outcomes are displayed in Figure 7.

Figure 7

Accuracy of different algorithms on different training models: (a) original frame training results and (b) crop frame training results.

Figure 7 (a) and (b) shows the training accuracy of different algorithms on the OF and cropped frame model. In the OF training results, the 3D-CNN-SSD model has the highest accuracy and reaches the steady state first at the 20th epoch. In the cropped-frame training model, the 3D-CNN-SSD model also has better accuracy than the remaining two recognition models and enters the steady state the fastest at the 7th epoch. The cross-entropy loss function (LF) is utilized for training on both models. The results are shown in Figure 8.

Figure 8

Loss curves of different algorithms in different training models: (a) original frame training and (b) crop frame training.

Figure 8 (a) and (b) show the LF curves for different algorithms trained on the OF model and the cropped frame model, respectively. The loss values on both training models show a decreasing trend, indicating that the models are learning the data features and improving the performance of the models at the same time. In the training results of the OF model, the LF curve of the 3D-CNN-SSD model first reaches a steady state at the 20th epoch, with the fastest convergence and the lowest loss value of 1.1. Similarly, the loss curve of the 3D-CNN-SSD model in cropped frames first reaches a steady state at the 17th epoch and the lowest loss value of 1.3. The results exhibit that the 3D-CNN-SSD model has good generalization ability and a low risk of overfitting. Then, 60 test samples are selected from the OF and cropped frame training models, respectively, to test the response performance of the three recognition methods. The results are shown in Figure 9.

Figure 9

Identify response time test results: (a) original frame training and (b) crop frame training.

Figure 9 (a) and (b) illustrate the variation of response time for different algorithms in the OF and cropped frame training models, respectively. As the test samples increase, the response times of the three recognition methods also change. Among the two training models, the response time of the 3D-CNN-SSD model is always smaller than the other two algorithms. It performs the best among the three algorithms and completes the image recognition with the fastest response time. Continuous loop testing is performed for the three different BAR methods. The test time is set to 150s, and the performance parameters of different algorithms are recorded in the form of curves. Resource overhead indicators are utilized to measure the consumption of resources required during the running process, such as time, memory, processor capacity, energy, etc. Figure 10 displays the outcomes of the continuous loop test.

Figure 10

Continuous cycle test results: (a) OpenPose, (b) DWT, and (c) 3D-CNN-SSD.

By comparing (a), (b), and (c) of Figure 10, it can be found that the three different BAR methods exhibit different performance fluctuation characteristics in the 150-s continuous loop recognition test. In Figure 10(c), the 3D-CNN-SSD model curve amplitude variations are small and regular, and the difference between its maximum and minimum values shows periodic changes. It indicates that the model has a more uniform resource allocation in the recognition task and the computing process is relatively stable. When the changes in resource overhead metrics are compared, it becomes evident that the performance of the OpenPose and DWT methods in Figure 10 (b) and (c) fluctuates more during the computing process than it does with the 3D-CNN-SSD model. Additionally, there is a significant widening of the difference between the maximum and minimum values, as well as an increase in the amplitude and frequency of the fluctuations. The results indicate that the 3D-CNN-SSD model has better stability.

3.2 Effect of 3D-CNN-SSD model application

To verify the application performance of the 3D-CNN-SSD model proposed in the study, the study uses 16 frames of RGB video data as input. Six different basketball movements are selected for recognition, including crossover, Eurostep, feint, Shamgod, spinmove, and shoot, and the results obtained after testing are shown in Figure 11.

Figure 11

Confusion matrix for different training models: (a) original frame and (b) crop frame.

Figure 11 (a) and (b) show the model training confusion matrices constructed from raw frame data and cropped frame data, respectively. In the OF training model, the direction-shifting actions are sometimes misclassified as Eurostep or Shamgod actions, probably due to the commonality of these actions with direction-shifting. Although there are some classification errors in the recognition process, the 3D-CNN-SSD model shows high accuracy in recognizing the action of shoot. The recognition rate (RR) is as high as 96% and its average RR is able to reach 89.5%. In the confusion matrix of the cropped frame model, the 3D-CNN-SSD model removes some interfering elements in the image by target detection, and the average RR is improved to 92.5%. Among them, it is still the highest RR of 98% for shoot action. The recognition accuracy of different algorithms for six basketball actions and the mean average precision (MAP) results are shown in Table 2.

In Table 2, the MAPs of OpenPose, DWT, and 3D-CNN-SSD models for the six actions are 82.95, 86.39, and 91.69%, respectively. Among them, the 3D-CNN-SSD model improves 8.74 and 5.33% compared to OpenPose and DWT, respectively. The 3D-CNN-SSD model has the lowest accuracy of 89.46% in recognizing Eurostep actions compared to other action categories. This is because the motion differences of Eurostep in the video sequences are not significant. Most of the time basketball players maintain similar postures in consecutive frames, resulting in weak frame-to-frame connections. However, for the recognition of the other five motions such as crossover, feint, and shoot, the model achieves an accuracy of more than 90%, with significant improvement in recognition. Therefore, the 3D-CNN-SSD model can fully extract the motion information in video frames for recognition. Table 3 displays the misclassification outcomes of several models during the recognition process.

Table 3 shows the recognized misjudgments for the six basketball actions. Except for case 6 shoot action, all the other actions have been misjudged. Case 1 and Case 5 indicate that crossover and spinmove are misjudged as Shamgod moves, respectively. Situation 2 is where Eurostep is misjudged as a shoot move. Case 3 denotes that feint is misjudged as spinmove. Case 4 denotes that Shamgod action is misjudged as Eurostep. The experimental results show that 3D-CNN-SSD has the lowest misjudgment rate in all five misjudgment cases. For example, in Case 1, the misjudgment rate of OpenPose, DWT, and 3D-CNN-SSD are 12.58, 9.46, and 6.49%, respectively. The misjudgment rate of 3D-CNN-SSD has the lowest misjudgment rate in Case 1. The study uses cluster analysis of the high-dimensional features output from the network’s FCL to assess the model’s performance in the area of basketball motion video recognition. The results are shown in Figure 12.

Figure 12

Visualization of feature clustering: (a) OpenPose, (b) DWT, and (c) 3D-CNN-SSD.

The feature clustering visualization results of OpenPose, DWT, and 3D-CNN-SSD are represented in Figure 12 in (a), (b), and (c), respectively. Different colored points are used to distinguish different action categories, and each color represents a specific kind of action data. In Figure 12(a), when using OpenPose technology for BAR, the feature data distribution of various types of basketball actions is more dispersed, and there is overlapping of data between categories. For example, part of the data in the shoot action is categorized into the Eurostep action, and part of the data in the Eurostep action is categorized into feint and spinmove action. In Figure 12 (b), the DWT algorithm also mixes the feature data of some action classes during recognition, which are lecture and Eurostep, crossover, and spinmove. Other than that the other actions are not overlapped, and the class spacing is more obvious. In Figure 12 (c), the 3D-CNN-SSD model achieves effective differentiation of features in the feature space by enhancing the differentiation between action categories, resulting in closer clustering of similar data. However, there are still some data in the figure with unclear boundaries with other categories, which may lead to misjudgment of actions by the model. This misjudgment may be due to the similarity of the performance of certain actions in the video frames, and the feature differentiation is not obvious, which affects the accuracy of the model action recognition. To evaluate the performance of the proposed 3D-CNN-SSD model in actual basketball games, the study used video datasets from multiple real games for testing. The test data includes various complex situations, such as athlete occlusion, motion blur, rapidly changing backgrounds, different lighting conditions, and crowd interference. And it includes 100 basketball game videos from different matches, with key movements annotated in the videos. By comparing with manually annotated actions, the performance of the confusion matrix analysis algorithm was evaluated, and the accuracy, recall, precision, and F1 score were calculated. The results are shown in Table 4.

According to Table 4, the accuracy of 3D-CNN-SSD is significantly higher than the other two methods, reaching 94.50%, indicating that this model performs the most accurately in BAR. In contrast, OpenPose has relatively low accuracy, possibly because it is mainly used for pose estimation and difficult to effectively handle complex motion blur and occlusion problems. The recall rate of 3D-CNN-SSD is also better than the other two methods, reaching 92.70%. In addition, 3D-CNN-SSD performed the best among all models with an accuracy of 95.30%, which measures the correct proportion of actions recognized by the model. The high accuracy indicates that the model can effectively reduce misidentification and ensure that most of the detected actions are correct. The relatively low accuracy of OpenPose and DWT may be related to their misidentification in certain complex scenarios. In terms of F1 score, 3D-CNN-SSD performed the best at 94.00%, demonstrating the model’s overall performance advantage. Finally, in the real-time results, the real-time performance of 3D-CNN-SSD reached 95 ms/frame, indicating that its processing speed is very suitable for real-time applications. In contrast, OpenPose has the longest processing time at 120 ms/frame, which may affect tasks with high real-time requirements. The real-time performance of DWT is 110 ms/frame, slightly inferior to 3D-CNN-SSD, but still maintained within a lower latency range.