Tai Chi movement segmentation and recognition on the grounds of multi-sensor data fusion and the DBSCAN algorithm

3.1 Vision ISE stereo matching and data fusion

In Tai Chi exercise, due to the presence of a large number of turning movements, relying solely on a single visual sensor for data collection may not be able to obtain complete data in certain situations, such as when the subject is facing away from the sensor or joint points are obstructing each other. To address this issue, the study constructed a multivision sensor sampling system consisting of three Kinects, ensuring that all joint points can be captured by at least one camera at any time. The study utilizes multi-threaded tools to achieve simultaneous sampling of three cameras and real-time recording of the correspondence between each sampling point and system time [13,14]. The relevant structure of the three Kinect sampling is shown in Figure 1.

Figure 1

Schematic diagram of the three Kinect sampling.

In Figure 1, three Kinect sensors form a triangular layout, and no matter how the practitioner moves or turns, at least one Kinect can capture their movements. The data from each Kinect are connected to the central processing unit through a cable, and synchronized sampling is achieved using multi-threaded tools to ensure data consistency and accuracy. Each sampling point is associated with the system time, facilitating time alignment in subsequent processing. Traditional action recognition research often relies on a single visual sensor, which is susceptible to factors such as viewing angle and occlusion during the acquisition process, resulting in data loss. The triangular layout sampling system constructed in this study, consisting of three Kinect cameras, can ensure that at any time, all the joints of the practitioner can be captured by at least one camera, effectively solving the problem of data collection for Tai Chi turning movements. Due to the differences in the coordinates of the joint points collected by ISEs and visual sensors, it is not feasible to directly calculate the linear transformation relationship between the two [15]. Therefore, a new stereo matching method has been proposed in the study. In the inertial visual matching process, the main camera and ISE are first activated, and the left or right wrist is moved closely against a straight line in the environment. The collected data will first undergo Kalman filtering for denoising, and then use the least-squares method to fit these coordinate points onto a straight line. The representation of the line to be fitted is shown in the following equation:

In Eq. (1), x 0 , y 0 , and z 0 represent the mean coordinates after filtering. The equation to be solved can be transformed into the following equation:

In Eq. (2), D represents the direction vector of the corresponding line, and P represents the sampling point. The core idea of the least-squares method is to find suitable parameters that minimize the sum of distances from all points to a given line. According to the basic rules of vector operation, this optimization objective is represented as:

In Eq. (3), V i represents the vector of point P . The purpose of the least-squares method is to find the minimum value of a function f , which requires taking the derivative of f and finding the point where the derivative equals zero. The result of taking the derivative of f is shown in the following equation:

In Eq. (4), I represents the identity matrix. Given that P is deterministic and the modulus of D is 1, the result of multiplying the transposes of D and D is 1. Therefore, f can be expressed as:

The calculation of S in Eq. (5) is shown in the following equation:

Eq. (6) is used to find the direction vector of the straight line in the coordinate systems corresponding to the ISE and the main camera. It assumes that the coordinate system of the ISE is moved to the origin position of the corrected coordinate system of the main camera. For the convenience of subsequent calculations, the study normalized two directional vectors on the same line [16]. These two vectors are connected at one end and can be considered as a common corresponding point. On the grounds of this corresponding point, the rotation matrix can be calculated. It assumes that the rotation matrix is R, the translation matrix is T, and the sample center of a certain line on the ISE is N. In the corrected main camera plane, the analytical expression of this line is shown in the following equation:

After joint analysis, Eq. (8) can be obtained:

It assumes that N pairs of corresponding lines were ultimately collected, and three different pairs of corresponding lines can be selected each time. To ensure the stability of the results, each sampling result should be different [17]. Each set of data can calculate a set of T values. Finally, it takes the average of these 3N sets of calculation results to achieve coordinate alignment between the ISE and the visual sensor. As the sampling period of the ISE is only 0.04 s, the three visual machines are divided into a total of 15 frames per second. In the initial stage, the error of the ISE is small and there is no frame loss problem, so it is chosen as the reference system for the federated Kalman filter. Considering the characteristics of the federated Kalman filter, its update process does not require all sensors to provide data. Using the sampling period of the ISE as the main filter period, even if three visual sensors do not provide data within a certain period, the data fusion process can still proceed in an orderly manner. In data fusion, both ISEs and visual sensors have time-varying measurement errors. To ensure the adaptability of the system, this study calculates the function between the error of the two and time, and utilizes time-varying allocation coefficients. According to the principle of information conservation, the cumulative error covariance matrix of the ISE is normalized. The data fusion algorithm is shown in Figure 2. In Figure 2, IMU represents the ISE, which can detect the acceleration and angular velocity in real time, providing accurate dynamic data for fast actions such as turning. IMU sensors can be attached to joints such as wrists, ankles, or waist to capture subtle motion changes and compensate for blind spots that visual sensors cannot capture at certain angles. Due to the difference in joint coordinates collected by ISE and visual sensors, it is not feasible to directly calculate the linear transformation relationship between the two. The proposed stereo matching method involves collecting data by moving the wrist along a straight line. After denoising through Kalman filtering, the line is fitted using the least-squares method, and the direction vector is calculated and normalized. Based on the corresponding points, the rotation and translation matrices are solved to ultimately achieve coordinate alignment. This data fusion method is innovative and unique in the field of Tai Chi action recognition.

Figure 2

Data fusion algorithm flow.

3.2 Tai Chi movement segmentation and recognition on the grounds of the DBSCAN algorithm

When dealing with the problem of recognizing Tai Chi movements, it is first necessary to preprocess the continuous trajectory of Tai Chi movements. To facilitate subsequent feature extraction and classification, this study rotates all data to be in the same initial position [18]. Then, using the MDL principle, key feature points are accurately extracted from these continuous trajectories. On the grounds of these feature points, it cuts the original trajectory into multiple subtrajectories and further converts these subtrajectories into four-dimensional feature vectors. When solving classification problems, this study chooses the DBSCAN algorithm, which has the advantage of not requiring to know the number of clusters in advance and can effectively discover clusters of any shape. To achieve the best clustering effect, this study optimizes the clustering parameters by combining the statistical characteristics of the samples and two classic clustering evaluation indicators. The trajectory segmentation and clustering process is shown in Figure 3.

Figure 3

Track segmentation and clustering process.

The effectiveness of trajectory segmentation largely depends on the quality of the selected feature points [19]. To ensure optimal segmentation results, feature points should capture the most significant changes in the trajectory as much as possible. It assumes that the trajectory G consists of a series of points p ₁, p ₂, p ₃…p _n , and the selected feature points are p _c1, p _c2, p _c3…p _cpar. By using these feature points, the entire trajectory is divided into multiple subtrajectories. By sequentially connecting these feature points, an approximate representation of the original trajectory is obtained. In the MDL principle, L ( H ) serves as the sum of the lengths of all subtrajectories, while L ( D ∣ H ) serves as the sum of the differences between all subtrajectories and the original trajectory. The calculation of L ( H ) is shown in the following equation:

In Eq. (9), len ( p c j , p c j + 1 ) represents the Euclidean distance. The calculation of L ( D ∣ H ) is shown in the following equation:

In Eq. (10), d ⊥ represents the vertical length of two trajectory segments L i and L j with the same dimension. d θ serves as the angular distance between the two line segments. Logarithmic functions are used to compress larger values into smaller ranges, avoiding computational problems caused by excessively large values. At the same time, logarithmic functions transform data that originally showed exponential growth into a more easily analyzable form, facilitating subsequent processing and analysis of logarithms. Among them, the calculation of d ⊥ is shown in the following equation:

In Eq. (11), L ⊥ 1 serves as the Euclidean distance between the starting point of two line segments and the projection point, and L ⊥ 2 represents the Euclidean distance between the endpoint of two line segments and the projection point. The calculation of d θ is shown in the following equation:

In Eq. (12), ∥ L j ∥ represents the length of L j . The distance between two trajectories is defined as:

In Eq. (13), ω ⊥ , ω ∥ , and ω θ represent the weights of vertical, parallel, and angular distances, respectively, and d ∥ represents the parallel length of two trajectories. A schematic diagram of distance definition is shown in Figure 4.

Figure 4

A schematic representation of the definition of the distance.

After determining the correlation function between L ( H ) and L ( D ∣ H ) , it is then essential for calculating which specific points on a trajectory belong to feature points. This study mainly chooses to search for local optimal solutions to approximate and replace the global optimal solution. By selecting two points p i and p j on the original trajectory, two cost functions can be constructed, as represented by the following equation:

In Eq. (14), MDL par ( p i , p j ) is the encoding length required for the line formed by connecting feature points, while MDL nopar is used to measure the encoding length of the original trajectory. If any point p k between points p i and p j satisfies MDL par ( p i , p k ) < MDL nopar ( p i , p k ) , to obtain a local optimal solution, this study considers extending it appropriately outward. In this way, the global optimal solution can be approximated and the points that meet the conditions can be selected as feature points. The obtained exact and approximate solutions are illustrated in Figure 5.

Figure 5

A schematic representation of the exact and approximate solutions.

After dividing the entire action trajectory into multiple small segments on the grounds of the position of feature points, the points on the original trajectory are arranged in chronological order, and each subtrajectory has its initial coordinates. If clustering subtrajectories directly, it may lead to excessive redundant information, thereby reducing the accuracy of trajectory classification [20]. Therefore, it is necessary to vectorize the subtrajectories to retain their directional information and convert the original trajectory into a series of connected vectors to obtain a point cloud structure distributed around the origin. Meanwhile, it is necessary to retain the quantity of sampling points contained in each subtrajectory and calculate the time corresponding to the subtrajectory on the grounds of the number of sampling points. Traditional clustering methods require a predetermined number of classifications, so they are not applicable. DBSCAN, on the other hand, can automatically determine the final number of classifications from the data. Therefore, a trajectory clustering algorithm on the grounds of an improved spatial-temporal distance-based DBSCAN Algorithm (IST DBSCAN) was adopted in the study. This method does not require presetting the number of categories, which can better adapt to actual situations and improve the accuracy and reliability of clustering. Meanwhile, the algorithm replaces the Euclidean distance with the spatiotemporal distance, which better reflects the true distribution of trajectories. In clustering, it can automatically determine the number of clusters, handle noise, and discover clusters of any shape. By determining the distance between the point and core point, it determines whether it is a core or boundary point, and then performs clustering. The advantage is that it does not require a preset number of clusters and is robust to outliers, improving the accuracy and reliability of clustering. In addition, it considers the limitations of IST-DBSCAN in dealing with environmental changes, including changes in lighting, background noise, and inaccurate sensors. To this end, the study further introduced particle filters, which track trajectories through a set of sample points during data collection and can better adapt to non-linear trajectory data, improving the effectiveness of handling complex situations. During the operation, a set of particles is first initialized and distributed based on prior information or preliminary observations. Next, based on the known state transition model, each particle is predicted to estimate the state of the object at the next moment. Then, the particle filter calculates the weight of each particle based on the observed data, which reflects the degree of matching between the particle and actual observation, and adjusts the distribution of the particles through a resampling step. Through this series of steps, particle filters can effectively respond to nonlinear changes in dynamic environments, improve data accuracy, and provide smoother and more accurate trajectory data for IST-DBSCAN, enhancing the clustering performance of the algorithm in complex environments.

To further enhance the user experience of the Tai Chi action recognition system, a user interface and mobile application for Tai Chi coaches and learners have been developed. This interface has a clear navigation design to meet the needs of different users. For non-technical users, an intuitive layout and actionable features are adopted to reduce the learning curve, allowing users to quickly get started and obtain the required feedback through simple operations. At the same time, the system combines a real-time visual feedback function, which can provide real-time feedback on the accuracy, fluency, and posture correctness of users’ movements during Tai Chi training. When the learner’s action deviates, the system can indicate the incorrect part of the action through a graphical interface and remind the user with color illustrations. In addition, for users of different levels, the system can provide a personalized progress tracking function, allowing learners to view their performance changes in multiple training cycles, thereby improving their skills more targetedly.

4.1 Experimental analysis of MSDF and DBSCAN clustering algorithm

To verify the effectiveness of the Tai Chi movement segmentation and recognition algorithm proposed by the research institute, experimental analysis was conducted. The experiment recruited 100 professional athletes and enthusiasts with different levels of Tai Chi practice as research subjects. During the experiment, three Kinect sensors were used to form a triangular layout. Using multi-threaded tools to achieve synchronous sampling of three cameras and real-time recording of the correspondence between each sampling point and system time. At the same time, ISEs are attached to joints such as the wrists, ankles, or waist of the body to detect acceleration and angular velocity in real time. Pressure sensors are installed on the soles of the feet to detect the contact force between the body and the ground. Through the collaborative work of these sensors, motion data during Tai Chi practice is collected as an experimental dataset. The study first validated the effectiveness of the MSDF algorithm. This experiment took the single whip trajectory of the left wrist as an example to obtain its projection on the x–y plane of the world coordinate system. The experimental environment is shown in Table 1.

The MSDF results of a single whip action are shown in Figure 6. As shown in Figure 6, the fused multi-sensor data can more accurately reflect the projection of the single-whip movement of the left wrist on the x–y plane of the world coordinate system. Compared with single-sensor data, the fused data can better capture the details and changes of actions, thereby improving the accuracy and reliability of action recognition. This proves the effectiveness of the MSDF algorithm in action recognition.

Figure 6

MSDF results for the single-whip action.

To verify the effectiveness of the MDL principle in selecting feature points for Tai Chi movement trajectories, the left wrist joint trajectory of the move Yunshou was studied as the experimental object. The feature point selection results of the left wrist trajectory of the action cloud hand and the representation curve of the feature points are shown in Figure 7. In Figure 7(a), the red dots represent the feature point positions calculated using the MDL principle. The remaining areas represent points that have not utilized the MDL principle. Figure 7(b) shows the trajectory curve corresponding to the original data. Figure 7(c) represents the trajectory curve formed by connecting feature points. Figure 7 shows that the trajectory curve formed by connecting the feature points calculated using the MDL principle is no different from the original trajectory, indicating that the MDL principle can accurately describe the trajectory of Tai Chi movements.

Figure 7

The result of feature points of the left wrist trajectory of the cloud hand and the representation curve of feature points: (a) Feature point selection results, (b) corresponding curve of raw data, and (c) corresponding curve of feature points extracted by MDL principle.

Before evaluating the clustering quality, it is necessary to verify whether there are potential clusterable structures in the original Tai Chi dataset. The research mainly selects Hopkins statistics to analyze the clusterability of the dataset. When the data in the dataset are evenly distributed, the value of the Hopkins statistic is approximately 0.5. If the value is less than 0.5, it can be considered that the dataset cannot be clustered. The closer the value is to 1, the more excellent the clustering effect of the dataset, indicating better separability of the dataset. Taking Tai Chi Cloud Hand as an example, when assigning equal weights to time distance and space distance in the formula for calculating distance, the trajectory of the Hopkins statistic with respect to the number of samples is shown in Figure 8. Figure 8 shows that the value of the Hopkins statistic fluctuates within the range of [0.9979, 0.9996], and its value is very close to 1, which strongly proves the good separability of the sample.

Figure 8

Trajectories of the Hopkins statistics with the number of samples sampled.

After determining the separability of the data, the effectiveness of IST-DBSCAN clustering can be further evaluated. The effectiveness evaluation of clustering on the grounds of internal standards mainly utilizes the inter-class distance and intra-class distance obtained from classification to comprehensively evaluate the clustering effect. Meanwhile, this study compares K-means clustering and means shift clustering, and uses the Silhouette Index (DI) for evaluating the clustering effect. The larger the DI calculation result, the more excellent the clustering effect. The experiment mainly utilizes the trajectories of the left wrist joint and the right wrist joint for action recognition. Therefore, the trajectories of these two are separately clustered. First, it sets the step size of the weight change to 1 and gradually increases it from 1 to 99. The correspondence between the DI results and the spatial distance weight ω _d for the joint points of the left and right wrists is shown in Figure 9. Figure 9(a) shows that when clustering trajectories on the left wrist, IST-DBSCAN can achieve the best clustering effect when the value of ω _d is 68, with a DI value as high as 0.42. The highest DI values of K-means and means shift clustering methods are only 0.28 and 0.26, respectively. As shown in Figure 9(b), when clustering the trajectories of the right wrist, the clustering effect is best when the value of ω _d is 77, with a DI value of 0.53, which is significantly higher than the other two clustering methods. This indicates that the IST-DBSCAN clustering method has significant clustering effects.

Figure 9

Correspondence between the DI results and the spatial distance weights ω _d: (a) Cluster results of left wrist trajectory and (b) right wrist trajectory.

4.2 Experimental analysis of Tai Chi movement recognition

Aiming at verifying the effectiveness of the motion recognition method for Tai Chi on the grounds of the MSDF-DBSCAN algorithm, this study selects ten unique moves as samples for classification, prioritizing the moves that appear more frequently in Wu-style Tai Chi. The final ten action samples determined for classification include A1 Grasping Sparrow Tail, A2 Single Whip, A3 Cross Hand, A4 Cloud Hand, A5 Underwater Needle, A6 Flash Back, A7 Oblique Flying, A8 Overturning Monkey, A9 Tai Chi Qi Shi, and A10 Moving and Blocking Hammer. The study separately extracts and converts each move into Fisher vectors, and divides the sample data into two parts: training samples and testing samples. The training samples are utilized for training and optimizing the classifier, while the testing samples are utilized for evaluating the performance of the classifier. Considering the issue of limited training samples, the study adopted transfer learning methods to increase the sample size. Specifically, the study fine-tunes pretrained deep neural network models to adapt to specific movements and styles of Tai Chi. The fine-tuning process focuses on the unique movement characteristics of each Tai Chi style, optimizing the last few layers of the network to enable the model to recognize different styles of moves while avoiding overfitting and improving training efficiency. The proportion of training samples possesses a direct influence on the accuracy of the classifier. The experiment mainly utilizes the motion trajectory of the left hand for single-joint-based action recognition. The accuracy of the classifier is calculated in binary and multiclassification scenarios when the training samples accounted for 25 and 75%, respectively. The recognition accuracy of binary classification when the training sample accounts for 25% is shown in Table 2. According to Table 2, with 25% of the training samples, the highest accuracy of the classifier is only 94.71%, and the lowest accuracy is only 89.88%.

The recognition accuracy of binary classification when the training sample accounts for 75% is shown in Table 3. According to Table 3, with 75% of the training samples, the classifier achieves the highest accuracy of 99.80% and the lowest accuracy of 97.28%. Relative to the situation where the training sample is 25%, its highest and lowest accuracies have improved by 5.09 and 7.32%, respectively. This indicates that the higher the training sample, the higher the accuracy of the classifier. In addition, it can be seen that there are significant differences in the classification accuracy of certain moves under different training sample ratios. Among them, the accuracy of the A5 underwater needle is lower when the training sample accounts for 25%, but significantly improves when the training sample accounts for 75%. This indicates that the A5 underwater needle has a high similarity in motion trajectory with other moves, which makes it easy to be misclassified when there are few training samples. The A3 cross-hand and A4 cloud-hand movements are relatively similar in their form of action, both involving the rotation and movement of the hands, but the specific trajectory and rhythm of the movements are different. Therefore, the classifier finds it difficult to accurately distinguish these subtle differences when there are few training samples, leading to misclassification. Although there are differences in the specific movement trajectory and direction of the hands between the A2 single whip and A7 oblique flying moves, they exhibit similar movement patterns during the extension or retraction stages of the arms, making it difficult for the classifier to accurately distinguish. Therefore, it is necessary to increase training samples to improve classification accuracy.

This study further validates the effectiveness of the proposed action recognition method and compares it with current advanced action recognition algorithms. This includes comparing the recognition performance of the global adaptive graph convolutional network (GAGCN) and hybrid long short-term memory convolutional neural network model (HLSTM-CNN). The recognition accuracy of each algorithm in binary and multi-classification scenarios was studied and calculated. The results of ten tests are shown in Figure 10. As shown in Figure 10 (a), in the case of binary classification, the mean accuracy of the MSDF-DBSCAN algorithm is as high as 98.52%, which is 5.43 and 2.94% higher than that of GAGCN and HLSTM-CNN algorithms, respectively. As shown in Figure 10(b), in the multiclassification scenario, the mean accuracy of the MSDF-DBSCAN algorithm is as high as 92.33%, which is significantly better than other algorithms. The MSDF-DBSCAN algorithm has a more significant action recognition effect.

Figure 10

The recognition accuracy of each algorithm in binary and multiclassification scenarios: (a) Accuracy in binary classification scenarios and (b) accuracy in multi classification scenarios.

In order to verify the accuracy and universality of the Tai Chi action recognition method based on the MSDF-DBSCAN algorithm in the styles of Chen style, Yang style, and Sun style Tai Chi, an extended experiment was designed for research. On the basis of the original Wu-style Tai Chi dataset, samples of Chen style, Yang style, and Sun-style Tai Chi movements were added. Each style selects representative moves, and Chen style Tai Chi is characterized by changes in strength and energy, including getting rid of energy, golden chicken independence, single-whip style, turning back and changing steps, and white crane flapping wings. Yang style Tai Chi is known for its smoothness, softness, and slowness, including Tai Chi Qi, Single Whip Style, Cloud Hand, Golden Rooster Independent, and White Crane Bright Wings. Sun style Tai Chi emphasizes rapid changes and clever steps, including pushing hands, lifting knees, hanging flower steps, shaking palms, sinking shoulders, and releasing strength. In a multiclass scenario with 75% of the training samples, the recognition accuracy of each algorithm is shown in Figure 11. As shown in Figure 11, in the Chen style Tai Chi style, the motion recognition accuracy of the algorithm proposed by the research institute is as high as 94.21%, which is 20.63% higher than that of the GAGCN algorithm. In the style of Yang style Tai Chi, the action recognition accuracy of the MSDF-DBSCAN algorithm is as high as 92.18%, which is 17.33% higher than that of the HLSM-CNN algorithm. In the style of Sun style Tai Chi, the action recognition accuracy of the MSDF-DBSCAN algorithm is as high as 93.35%, significantly higher than that of the other two models. This confirms the accuracy and universality of the MSDF-DBSCAN algorithm in recognizing different styles of Tai Chi movements.

Figure 11

Accuracy of different algorithms in recognizing movements of Chen style, Yang style, and Sun style Tai Chi.

The study further validated the recognition efficiency of different algorithms in action recognition tasks. By comparing the performance of the MSDF-DBSCAN, GAGCN, and HLSM-CNN algorithms, the latency and frames per second (FPS) of each algorithm in real-time processing were measured. The real-time processing delay and FPS of each algorithm in different styles of Tai Chi are shown in Table 4. According to Table 4, among all styles of Tai Chi, the delay of MSDF-DBSCAN is lower than other algorithms. The delays of Wu style, Chen style, Yang style, and Sun style are only 55, 52, 51, and 58 ms, respectively. This indicates that the algorithm can process input data more quickly and provide real-time feedback. At the same time, the FPS of the algorithm is 18.5 frames per second, demonstrating extremely high processing power, ensuring the smoothness of the video stream and real-time feedback effect.

The study finally evaluated the effectiveness of the Tai Chi gesture recognition system based on the MSDF-DBSCAN algorithm in long-term use. The study recruited 100 Tai Chi learners and randomly divided them into an experimental group and a control group, with 50 students in each group. The experimental group used a Tai Chi action recognition system based on the MSDF-DBSCAN algorithm for Tai Chi teaching. The control group used traditional Tai Chi teaching methods. The teaching frequency is three times a week, with 60 min of training each time. Before the experiment begins, a preliminary test is conducted on the basic movements of Tai Chi for all participants, and the accuracy and fluency of each student’s movements are recorded. At the end of each month, a mid-term evaluation of the students is conducted to record their accuracy and fluency in movements. The accuracy and fluency scores of the students’ movements after 3 months are recorded. The total score for fluency is 50 points, evaluated by 10 experts, and the average is taken as the final score for each learner. The experimental results are shown in Table 5. According to Table 5, in terms of action accuracy, the experimental group had an accuracy rate of 88.5% before the start of the experiment. One month later, the accuracy rate increased to 95.3%, reaching 97.8% after 2 months, and further improving to 98.8% by 3 months. This indicates that the system based on the MSDF-DBSCAN algorithm can provide real-time feedback during long-term training, helping learners correct action errors in a timely manner. Compared to the control group, the improvement in action accuracy was not significant, with only a 2.2% increase. In terms of motor fluency, the experimental group’s fluency score increased from an initial 35.2 points to 48.6 points, which has a long-term effect on improving motor fluency. The fluency score of the control group increased from 35.7 points to 40.2 points, with a relatively small improvement, which is inferior to the teaching method proposed in the study.