Sports video temporal action detection technology based on an improved MSST algorithm

3.1 Unsupervised video data preprocessing method based on ResNet50

Driven by artificial intelligence technology, the application of video analysis in sports is increasing, especially in sports video temporal action detection technology, which has become one of the current research hotspots. Sports videos contain rich temporal dynamic information. Accurately identifying and analyzing athletes’ movements is of great significance for event analysis, training assistance, automatic refereeing, and audience experience. However, the complexity, variability, and temporal nature of movements in sports videos make motion detection tasks extremely challenging. Traditional video analysis methods rely heavily on manual feature extraction and shallow models, making it difficult to cope with action recognition tasks in complex scenes. Optimizing the preprocessing of sports video data can reduce this redundant information, focus on temporal actions, and improve training efficiency [10]. Therefore, a ResNet-based unsupervised video data preprocessing model is proposed, as displayed in Figure 1.

Figure 1

Unsupervised video data preprocessing model based on ResNet50.

In Figure 1, the detailed network structure has three modules: feature extraction, frame extraction, and recognition. First, in the feature extraction module, ResNet50 is used as the main feature extraction network to perform convolution operations on each frame in the video sequence, extracting preliminary spatiotemporal features. Next, the input frame extraction module clusters the features through hierarchical clustering to generate pseudo labels for unsupervised training of the frame extraction network [11,12]. This process clusters and labels more critical frames in the video to focus on effective information in the future. Finally, it enters the recognition module, which uses the temporal pooling (TP) method to fuse the features of each frame, forming more representative global features. Subsequently, these features are classified through the Softmax layer to recognize temporal actions in the video. In this process, three loss functions are used respectively for the feature extraction network, unsupervised frame extraction network, and final classification result supervision to ensure the comprehensive performance of the model in feature extraction, keyframe extraction, and action recognition. In the feature extraction module, the backbone structure is displayed in Figure 2.

Figure 2

ResNet50 model structure diagram.

In Figure 2, ResNet50 is a deep CNN whose core idea is to solve the gradient vanishing and degradation in deep networks through residual structures [13,14]. ResNet50 has 50 layers, including convolutional layers, pooling layers, fully connected layers, etc. The key structure is the residual module, which is composed of two or three convolutional layers. By introducing “shortcut connections,” the input is directly added to the output to form a residual mapping. This structure allows the network to retain key input information during the training process, avoiding the degradation phenomenon that occurs as the number of layers increases [15,16]. The main structure of ResNet50 includes an initial 7 × 7 convolutional layer and a max pooling layer for basic feature extraction, followed by four sets of residual modules containing 3, 4, 6, and 3 residual units, each consisting of a different number of convolutional layers. Before each residual module, 1 × 1 convolution is used for dimensionality reduction or enhancement to match input and output channel numbers.

In the frame extraction module, the first input is the feature map set output by the feature extraction module. Next, the hierarchical clustering method is taken to process the output features of the two convolutional layers to generate corresponding pseudo labels, which are then passed into the fully connected layer for output calculation. Subsequently, the pseudo labels are treated as real labels, and the classification loss is obtained through the cross entropy loss function based on the output results, thereby updating the frame extraction module. Meanwhile, an iteration threshold is set. After exceeding the threshold of updates, the training process of the small network is stopped [17,18]. Finally, a feature sample is randomly selected from each cluster after classification as the input frame and passed to the recognition module for further processing. The multi-class normalization operation is shown in the following equation:

In Eq. (1), N represents the dimension size of the input feature map. c represents the number of clustering categories. T signifies the dimension size of the input feature map. s represents the output score of the fully connected layer. y ∈ ( 0 , 1 ) as shown in the following equation:

In Eq. (2), L represents the pseudo labels generated by clustering. The recognition module includes a feature fusion module and a classification module. Common feature fusion modules include temporal attention mechanism, recurrent neural network, and TP mechanism. The time attention mechanism can dynamically allocate weights based on the importance of each feature in the time series, thereby more effectively capturing information at critical moments, as shown in the following equation:

In Eq. (3), w d t represents different weight values of the feature. f d t represents the feature extracted at the j -th time step. By multiplying the features at different time steps with corresponding weights and summing them up, this method can more prominently integrate the information of important moments into the final feature representation [19]. In a recurrent neural network, the average output of all long short-term memory units is taken as the fusion feature, as shown in the following equation:

In Eq. (4), o d t represents the output feature value corresponding to the t -th time step. All image-level features are fused into one feature, as expressed in the following equation:

In Eq. (5), n signifies the number of selected frames. By averaging the output features of all time steps, this method can obtain an overall representation of the features in the time dimension, thereby preserving global feature information and reducing noise interference. However, time pooling combines the features of each time step through simple aggregation operations, which can effectively reduce the dimensionality of data and thus reduce computational overhead. It does not compute additional weight matrices like attention mechanisms and avoids the complex recursive operations of time steps in recursive neural networks [20,21]. Therefore, time pooling has higher efficiency in processing long time series. Time pooling performs well in reducing noise and highlighting global features. By averaging or selecting the maximum value of the features at all time steps, time pooling can smooth out the fluctuations of time-series data, avoid excessive attention to individual noise points, and make the fused features more representative. The process is shown in Figure 3.

Figure 3

Time pooling process.

As shown in Figure 3, the feature representations generated at each time step in the network are used as inputs. These features are usually calculated by the previous layers over multiple time steps. The pooling method is determined. Average pooling is used to calculate the average value of each feature dimension over time. Maximum pooling takes the maximum value of all features in each feature dimension at all time steps and outputs the pooled features as the global features of the time series for subsequent analysis or classification tasks [22]. The pooled features can represent the information of the entire time series, and their dimensions are the same as those of individual time step features.

3.2 Sports video temporal action detection model based on the improved MSST algorithm

After completing the video data preprocessing, this study adopted an MSST model based on multi-temporal scale and spatiotemporal encoder for temporal action detection. Unlike traditional temporal action detection methods that prioritize short-term temporal modeling or only focus on spatial relationships, MSST introduces a multi-scale temporal structure that can capture both fine-grained and coarse-grained motion information simultaneously. This enables the model to effectively identify actions of different durations while reducing sensitivity to background clutter. In addition, the FSA module dynamically enhances keyframes to ensure that only relevant motion information is retained for final classification and localization. Its structure is shown in Figure 4.

Figure 4

MSST model structure diagram.

As shown in Figure 4, first, the video input undergoes 3D convolution and spatiotemporal encoder transformation modules to extract preliminary spatiotemporal features. Next, multi-scale features are generated through an FPN to enhance the representation of action features at different time scales. These features then enter two different branches, each performing convolution operations to further extract spatiotemporal information [23,24]. These features are used for similarity calculation and similarity prediction. In the similarity branch, features are selected and aggregated through the feature selection and aggregation module to calculate the similarity matrix between different time steps, forming relevant features for action recognition. In the feature classification branch, feature classification is performed through fully connected layers and Softmax operations to output action categories. Ultimately, the network combines similarity and classification output to achieve accurate temporal action detection. In the MSST model, features at multiple time scales are jointly generated by FPN and STT. In FPN, the final generated feature pyramid contains hierarchical structures of features at different scales, enabling the model to simultaneously focus on both large-scale and small-scale information. FPN is mainly responsible for multi-scale feature extraction. Due to possible differences in the duration and speed of actions in sports videos, FPN preserves fine-grained and coarse-grained action features. It enhances spatial representation and provides hierarchical feature maps at different levels. The STT module focuses on capturing long-term temporal and spatial dependencies in video sequences. Unlike FPN running at the pure feature level, STT explicitly simulates the interaction between frames over time, which is crucial for identifying actions with complex motion patterns and occlusions. FSA aims to dynamically select keyframes, reduce redundant frames, and emphasize important action moments. Unlike FPN that focuses on hierarchical feature extraction and STT that models global dependencies, FSA improves time accuracy by adaptively weighting frames in action sequences. These three modules work together, with FPN extracting spatially rich features at multiple scales, STT simulating long-term dependencies, and FSA refining keyframe selection for final classification and localization. Their combined effect enables MSST to effectively handle various motion actions with different speeds, complex backgrounds, and inconsistent movements. The FPN structure is shown in Figure 5.

Figure 5

FPN.

As shown in Figure 5, the core structure of FPN takes CNN to generate multi-layer feature pyramids. FPN first obtains feature maps at different levels from the backbone network, which contain multi-scale features from high to low layers. Then, feature fusion is carried out through a top-down path, gradually transferring high-level semantic features to low-level features. At the same time, feature maps of the same scale are merged through horizontal connections to achieve information integration [25]. Each level of feature map is processed through convolution to ensure feature dimension matching and enhance representation capability. STT is a deep learning module used for modeling spatial and temporal information, typically applied for analyzing temporal data such as video understanding and temporal action detection. This structure extends the self-attention mechanism in the Transformer structure to the spatiotemporal domain to capture feature relationships at different time steps and spatial positions, as shown in Figure 6.

Figure 6

Structure of spatiotemporal encoder.

In Figure 6, the system obtains sports video frames from multiple time steps with different perspectives, extracts image features of each time step through a feature extraction network, and preserves time step information. Then, the spatiotemporal alignment module aligns these video frame features in both time and space, generating synchronized spatiotemporal features. Next, these aligned features are input into the space-time pyramid transformer and further processed with spatial embedding information to capture multi-level spatiotemporal relationships [26,27]. After being processed by transformer, the model generates action state predictions for future time steps, including information such as the position and action category of athletes in the scene. The final output includes spatiotemporal information of the scene and motion trajectory prediction, providing accurate motion recognition and temporal information for sports video analysis. The feature sequence is shown in the following equation:

In Eq. (6), T represents the quantity of frames in the video. v i represents the feature representation of frame i . First, the features of each frame are layer normalized, and then input into a multi-head self-attention module to capture the internal spatial dependencies of that frame. k s represents the spatial features of each frame after self-attention processing. The expression for generating the final spatiotemporal features is shown in the following equation:

In Eq. (7), L ( ⋅ ) represents a fully connected layer that normalizes the spatial features of all frames and then captures the temporal dependencies between frames through a multi-head self-attention module. Then, a linear transformation is performed through fully connected layers to generate the final spatiotemporal features. The feature sequences obtained by the FPN module are concatenated, and an improved module model based on frame level self-attention is studied. This method is a self-attention mechanism used in video analysis, mainly used to capture the correlation between frames in videos and improve the representation ability of temporal features [28,29]. This method dynamically assigns weights by comparing the features of each frame with other frames, emphasizing the contribution of important frames and suppressing unimportant or redundant frame information. The frame level self-attention mechanism is based on the self-attention structure of transformer. Its core idea is to measure the importance of each frame in the entire video sequence by calculating the “attention weight” of its features in the input sequence. First, the features of each frame in the video sequence are represented as i , and the features of each frame are usually extracted through CNN. Then, the features of each frame are queried and mapped to keys and values [30,31]. These mappings are used to calculate attention scores, as shown in the following equation

In Eq. (8), Q and K are used to calculate inter-frame similarity. V represents the feature value, calculates the similarity between each frame and other frames through the inner product, and obtains the weight distribution through Softmax. The calculated attention weights are used to weigh and sum the features of all frames, resulting in a fused frame-level feature representation. The fused features obtained through the self-attention mechanism can comprehensively consider the correlation between frames, highlight important temporal information, and suppress redundancy and noise [32]. This module uses two one-dimensional convolutions to generate the start and end boundaries, as expressed in the following equation:

In Eq. (9), d n s is obtained by applying 1D convolution to the positional features, representing the distance from position n to the beginning of the time period. d n e represents the distance from position n to the end of the time period. ϕ n c represents the predicted category score for this time period. n represents the position index at different time scales. The structure of the frame-level self-attention module is shown in Figure 7.

Figure 7

Structure of frame level self-attention module.

In Figure 7, first, the input video frame sequence is divided into multiple frame blocks as subsequent processing units. The encoder extracts preliminary spatiotemporal features from each frame block and generates two types of feature representations. Next, these features are input into an FPN for multi-scale feature fusion to capture action features at different time scales. The RFP outputs represent the features used for action localization and classification, respectively. Then, the boundary pooling module performs TP operations on the features to aggregate important temporal information and capture boundary features at the beginning and end positions of actions. Finally, the fused feature output is taken as the final frame-level self-attention feature for subsequent action detection models.

4.1 Performance analysis of unsupervised video data preprocessing model based on ResNet50

The CPU used in the experiment is Intel (R) Xeon^® Gold6226@2.7GHz. The GPU used is NVIDIA GeForce Tesla V100S, with 32 GB of memory. The study uses the THUMOS14 public dataset, which is specialized for time-series action detection and recognition tasks. It contains a large number of labeled action fragments covering various movements, making it one of the classic datasets in the field of time-series action detection. There are 20 action categories in total, all of which are sports, such as gymnastics, diving, weightlifting, volleyball, track and field, etc. The dataset is divided into a training set, a validation set, and a test set. The validation and testing set provides accurate temporal boundary annotations, namely the start and end times of actions in each video clip, as well as corresponding action category labels. The action clips in each video are labeled with a temporal boundary, including the start time and end time of the action. This study introduces TSN and DPC as baseline models for comparative analysis. This study adopts TSN because it is a widely used and efficient temporal action segmentation framework, while DPC represents an advanced time-dependent modeling method. Comparing TSN and MSST helps evaluate improvements in multi-scale spatiotemporal feature extraction while comparing DPC and MSST highlights advances in computational efficiency and adaptability to complex motion. The results are shown in Figure 8.

Figure 8

Accuracy analysis of each model. (a) The relationship between dataset size and accuracy. (b) The relationship between the number of iterations and accuracy.

Figure 8(a) displays the relationship between dataset size and accuracy. Figure 8(b) illustrates the relationship between iteration times and accuracy. As shown in Figure 8(a), as the dataset size increased from 0 to 500, the accuracy of all models gradually improved, indicating that larger amounts of data could provide richer training information. The curve of ResNet50 was consistently higher than other models, with an accuracy of 0.7 when the dataset size was 100, significantly better than TSN’s 0.55 and DPC’s 0.6. When the dataset increased to 500, the accuracy of the three models saturation, but ResNet50 still maintained its best performance, approaching 1.0. This is because ResNet50 provides stronger feature extraction capabilities and can learn richer spatiotemporal features from limited data. As shown in Figure 8(b), from iteration 0 to 30, the accuracy of all three models improved rapidly. In the initial training stage, the model gradually learned effective features from the data. ResNet50 achieved an accuracy of 0.85 in approximately 10 iterations and stabilized at 30 iterations, approaching 1.0. In contrast, DPC had an accuracy of only 0.75 in 10 iterations, while TSN had an accuracy of only 0.7 and a slower convergence speed. This is because ResNet50 has more efficient feature learning capabilities, while TSN and DPC rely on complex time modeling mechanisms, resulting in slower convergence speeds. The unsupervised video data preprocessing based on ResNet50 has a better information retention rate. Each dataset is divided into different sizes, with sizes of 100, 200, 400, and 800 for datasets 1 to 4, as shown in Figure 9.

Figure 9

Comparison of processing time for various models. (a) Time under different training set size. (b) Time under different verification set size.

Figure 9(a) displays the time consumption under various training set sizes, and Figure 9(b) displays the time consumption under different validation set sizes. As shown in Figure 9(a), as the training set size increased from Training set 1 to Training set 4, the time consumption of the three models gradually decreased. The DPC time in Training set 1 was about 320 ms, which decreased to about 100 ms in Training set 4. The TSN also decreased from about 300 ms to about 120 ms, while ResNet50 decreased from about 220 ms to about 80 ms. This indicates that a larger training set improves the training efficiency of the model, and more data helps to quickly optimize model parameters, thereby reducing training time. In Figure 9(b), as the validation set size increased from Validation set 1 to Validation set 4, the time consumption significantly increased, especially for the DPC model, which grew from about 100 ms to nearly 400 ms. The TSN increased from about 150 ms to about 300 ms. ResNet50 increased from approximately 50 ms to approximately 200 ms. The increase in validation set size means that more data needs to be inferred and validated, resulting in a significant increase in computational complexity and time consumption. The experimental results indicate that the proposed unsupervised video data preprocessing based on ResNet50 has higher processing efficiency. The actual effects of each model are analyzed, as displayed in Figure 10.

Figure 10

Analysis of actual processing effects of various modes. (a) Original image. (b) DPC. (c) TSN. (d) ResNet50.

Figure 10(a) shows the original image, while Figure 10(c) and (d) represent the image processed by DPC, TSN, and ResNet50, respectively. From Figure 10, the DPC model had a good effect on motion target detection, framing the movement positions of athletes. However, there is still some blurring in the overall image, especially in the positions of athletes’ hands and feet. This is because the DPC model captures some motion blur on the time series, but the spatial resolution recovery is insufficient, resulting in blurry image details. The TSN model can not only detect athletes, but also restore image motion blur. Compared with DPC, the contours of athletes are clearer. However, there is still ambiguity in the background section. This is because the TSN model focuses more on action recognition and has some optimization for object detection, but its processing effect on the background is relatively weak. ResNet50 has the highest detection accuracy for athletes, especially in details such as arm and leg positions, which are noticeably clearer. The background blur has also been reduced. This is because the ResNet50 model has stronger ability in spatial feature extraction, which can provide higher resolution and clearer image restoration results. The comprehensive performance of each model is analyzed using validation set and test set data, as displayed in Table 1.

According to Table 1, ResNet50 outperformed other models on IoU performance on both datasets, at 91.2 and 89.5%, respectively, indicating its stronger detection capability. TSN closely followed, with IoUs of 85.3, 83.1, and 81.3% on Dataset A, Dataset B, and Dataset C, respectively. The performance was relatively stable. The IoU of the DPC model was relatively low, with 78.5% on Dataset A, 76.2% on Dataset B, and 75.7% on Dataset C. In terms of PSNR, ResNet50 also performed the best, reaching 35.8 dB on Dataset A, 34.6 dB on Dataset B, and 39.3 dB on Dataset C, which was significantly higher than the other two models. This indicates that ResNet50 has good ability in image quality restoration and can perform excellently in image enhancement and denoising processing. The PSNR of the TSN model was 32.5 and 31.8 dB, respectively, with the second highest performance, indicating that its image restoration effect was good. In contrast, the DPC model had lower PSNR on the two datasets, at 28.1 and 27.4 dB, respectively. The ResNet50 model had excellent preprocessing performance.

4.2 Sports video temporal action detection model based on the improved MSST algorithm

To validate the proposed improved MSST algorithm for the sports video temporal action detection model, the basic MSST and MSST + FPN + STT model structures are used as comparative models, named Model 1 and Model 2, and the improved MSST model is named Model 3. Detection accuracy and root mean square error (RMSE) are taken as evaluation indicators, as displayed in Figure 11.

Figure 11

Comparison of accuracy and RMSE values for various models. (a) Accuracy of various models under different training set sizes. (b) RMSE of various models under different training set sizes.

Figure 11(a) shows the accuracy values of each model, and Figure 11(b) shows the RMSE values of each model. According to Figure 11(a), as the iteration increased from 0 to 500, the accuracy of all three models showed an upward trend. At the beginning, Model 1 had the lowest accuracy, at 0.4, while Model 2 and Model 3 had the accuracy of 0.5 and 0.6. After 300 iterations, Model 3 achieved an accuracy of 0.85, ahead of Model 1’s 0.75 and Model 2’s 0.7. At 500 iterations, Model 3 maintained the highest accuracy close to 0.9. According to Figure 11(b), as the iteration increased, the RMSE of all models gradually decreased, indicating a reduction in error. In the initial stage, Model 3 had the highest RMSE, close to 0.6, while Model 1 and Model 2 had RMSE of 0.5 and 0.45, respectively. After 300 iterations, the RMSE of Model 3 decreased to below 0.3, outperforming Model 2 and Model 1. After 500 iterations, Model 3 had the lowest RMSE, approaching 0.2. The proposed MSST model demonstrates excellent performance by integrating deep learning neural networks, surpassing traditional architectures in terms of stability, efficiency, and accuracy. ResNet for feature extraction ensures that the model can maintain high-quality feature representation even in challenging lighting conditions or occluded scenes. The self-attention mechanism in STT significantly reduces computational overhead by selectively focusing on key spatiotemporal interactions, thereby achieving faster and more scalable processing. The FPN, STT, and FSA modules driven by neural networks have improved the accuracy of action classification, with a 12.3% increase in mAP compared with the baseline model. The results indicate that using deep neural networks can not only enhance feature extraction, but also ensure the stability of detecting fast dynamic actions. This confirms that integrating CNN and attention-based Transformers is a promising method for temporal action detection in sports videos. Different types of sports, such as gymnastics, diving, weightlifting, volleyball, athletics, and high jump, are selected and named A–F. The detection results of each type of sport are analyzed, as shown in Figure 12.

Figure 12

Analysis of the recognition effect of different types of sports. (a) Comparison of accuracy of models for different types of movements. (b) Comparison of time of models for different types of movements.

Figure 12(a) displays the accuracy of the model under different types of sports. Figure 12(b) shows the detection time of the model under different types of sports. According to Figure 12(a), Model 3 had the highest accuracy and outperformed Model 1 and Model 2 in all types of sports. In gymnastics, the accuracy of Model 3 was about 95%, significantly higher than Model 2’s 90% and Model 1’s 85%. In diving and athletics, Model 3 achieved an accuracy rate of 98%. For weightlifting and volleyball, although the performance of all three models declined, the accuracy of Model 3 was still close to 95%. This is because Model 3 has stronger feature modeling capabilities, which can better adapt to different types of sports, especially those actions that contain complex temporal and spatial features. According to Figure 12(b), in gymnastics, the average detection time of Model 3 was about 150 ms, while Model 1 and Model 2 were about 300 and 400 ms, respectively. In diving and athletics, the detection time of Model 3 was controlled below 200 ms, while Model 2 was close to 450 ms. For weightlifting and volleyball, the calculation time of Model 3 was about 250 ms, much lower than Model 1 and Model 2. This is because Model 3 has achieved efficient computing power in its structural design, which can significantly reduce computational overhead while maintaining high accuracy. The proposed Model 3 has better performance and performs well in various types of sports. The actual effects of each model are analyzed, as displayed in Figure 13.

Figure 13

Analysis of the actual detection effect of each model. (a) Model 1. (b) Model 2. (c) Model 3.

Figure 13(a)–(c) respectively show the actual detection performance of Models 1–3. As shown in Figure 13(a), Model 1 recognized multiple athletes and their positions, but there were obvious false positives and omissions. In the middle image, player 26 was correctly detected, but some background referees were mistakenly identified as targets. In the image on the right, the border of player 12 was not aligned correctly with the action area. In addition, there were duplicate annotations in multiple detection boxes, resulting in low detection efficiency. As shown in Figure 13(b), compared with Model 1, the alignment effect of the target boundary was improved. In the left image, the detection boxes for player 29 and the opposing player were more accurate, and the false positive rate was reduced. However, in the middle figure, the referee was still mistakenly labeled as a target, indicating that Model 2’s interference with non-moving subjects still existed. In addition, in the image on the right, the boundary of player 26 was incorrectly labeled. Overall, Model 2 could better locate the player’s action area, but its robustness to background noise was still insufficient. According to Figure 13(c), the detection results of Model 3 were significantly better than those of Models 1 and 2. In the left image, the boundary detection between player 29 and the opposing player was accurate and clear, with no false positives or omissions. In the middle figure, the referee was not marked incorrectly, indicating that Model 3 had stronger robustness in distinguishing between background and target. In addition, in the image on the right, the boundary of player 26 was accurate, and all detection boxes were closely attached to the target area, without any duplicate or blurry annotations. The experimental results show that the proposed model has excellent performance. The ablation experiment is conducted on the model, as displayed in Table 2.

From Table 2, the mAP of the basic model MSST was 78.2%, the calculation time was 150 ms, and the performance was low. This indicates that a single module cannot fully capture complex spatiotemporal characteristics. After introducing FPN, the mAP increased to 83.5%, and computation time increased to 170 ms, demonstrating the importance of multi-scale feature extraction for performance improvement. After adding STT, the mAP reached 85.0%, indicating that STT can effectively model spatiotemporal dependencies. After introducing the FSA module, the mAP was 84.2%, and the precision and recall were improved to 83.2 and 85.5%, respectively, demonstrating the enhanced effect of frame-level self-attention on keyframe extraction. The mAP of the complete model reached the highest, at 90.5%, with a calculation time of 220 ms. The recall and precision were 92.0 and 89.5%, respectively, and the F1 score was 90.7%. The research results show that the three-module synergy is the most effective in temporal feature modeling and multi-scale feature fusion, which is higher than the baseline model and can achieve the best balance in performance and precision.