Camera-based climbing analysis for a therapeutic training system

Introduction

Bouldering and climbing are increasingly attracting interest across all age groups and have become trend sports all over the world. Various studies demonstrated that climbing improves coordination, flexibility, the cardiovascular system and has positive effects on both physiological and psychical health conditions [2], [8], [16], [17]. This implies that climbing can also be beneficial in view of therapeutic measures. From the very beginning, especially in case of competitive sports, climbing motions were analysed to assess and optimise climbing techniques. In view of therapeutic applications, climbing motion analysis has gained even more importance to avoid movements that are prone to cause injuries and to motivate the climber by means of gamification.

Related work

Previous studies have demonstrated that the centre of mass (COM) is a representative measure for analysing climbing movements [5], [14], [15]. Even though the motion is represented by only one single point, it provides relevant information in case of climbing motions. In that sense, it could be demonstrated that parameters, such as fluency, force and COM distance to the wall can be derived from the COM and that they are relevant for motion evaluation [5], [15]. For therapeutic climbing, such parameters are relevant to draw conclusions about hip positioning with respect to the feet, about motion accelerations and finally about muscle stress that should be analysed to assess the patient’s progress and to avoid overstrain. Existing systems, however, were designed for competitive sports or they were used under laboratory conditions. Strain gauges, force torque sensors and capacitive sensors were employed to analyse the interaction with holds on the climbing wall, but they have the disadvantage that they have to be integrated into the wall setup, see [1], [9], [10], [11], [12], [13]. Other solutions, such as wearables, attached markers or commercial motion capture systems, e.g. employed by [3], [4], [5], [6], [7], require body contact, which is inconvenient for the climber and prone to injuries. Consequently, we propose a solution that works contact-less and can be easily integrated in the climbing setup. Our approach comprises to capture the COM by means of a commercial depth camera and to calculate fluency, force and hip distance to the wall by means of measures derived from the COM, which are entropy, acceleration and the distance of the COM from the calculated plane defining the wall.

Methods

Experimental setup and sensor data

An Intel RealSense D435 RGB-D camera was used to record both RGB (red, green, blue colour channel) and depth images at a frame rate of 30 frames per second (FPS). Because climbing motions in a therapeutic context are normally rather slow compared to other sports analysis applications, such as gait in sprint where expensive high speed cameras are commonly used, 30 FPS are sufficient. This sensor was placed at a distance of approximately 6 m in front of the climbing wall and was attached to a computer that was used for capturing and saving the data, as visualised in Figure 1.

Figure 1:

Experimental set-up: The sensor is placed approximately 6 m in front of the wall.

Examples of a captured RGB and depth frame with the calculated point cloud can be seen in Figure 2.

Figure 2:

Sensor output: RGB and depth image and example point cloud.

Segmentation and COM calculation

The camera was extrinsically calibrated with respect to the wall, which results in a world coordinate system at the top left corner of the wall, (see Figure 3). This calibration step is necessary to make the analysis independent from the position and orientation of the camera with respect to the wall.

Figure 3:

World coordinate system with x, y and z axis, and climber (grey dots) and calculated centre of mass (COM) (magenta dot) extracted from the captured point cloud representing the complete scene (turquoise dots).

Based on the calculated point cloud, the climber is segmented, which means that all the points belonging to the climber are extracted from the whole point cloud of the captured scene. Subsequently, the COM is determined by calculating the mean x, y and z component of all the climber’s points. An example is shown in Figure 3.

The temporal sequence of the COM position is analysed in the next step. Therefore, the measures listed in Table 1 were calculated to represent climbing-relevant parameters. It should be pointed out that, by means of a camera, no direct measurement of parameters such as force is possible. However, as already stated in [15], such measures allow to draw conclusions about the force or the power, for example, that a climber applies during a certain route.

Table 1:

Centre of mass (COM) analysis: Calculated measures to derive climbing-relevant parameters.

Measure	Climbing-relevant parameter
Average acceleration of COM	Force
Entropy of COM trajectory	Fluency
Average z component of COM	Distance to the wall

1. COM, centre of mass

Results and discussion

The algorithm that localises the COM and calculates the above mentioned parameters was successfully evaluated. Therefore, we recorded six probands to climb a pre-defined route and instructed them to realise different levels of the parameters, e.g. to climb at three different levels of fluency or force. An example for the validation of how acceleration correlates to force for every single proband is given in Figure 4. Due to the fact that each climber defines his or her personal level, the results should be separately considered for each climber. In other words, the relation between the three levels should be compared, so that the development of a climber’s style could be monitored to give feedback such as “Your technique improved, so that your climbing requires less force than last week. With reduced force you are less prone to injuries.” To summarise, the algorithm correctly classified the climbers’ levels for all three parameters.

Figure 4:

Relation between force and COM acceleration.

Conclusions

This study demonstrated the feasibility of camera-based climbing analysis. Future work should focus on a more detailed motion capture including joints that are especially important for the analysis, such as hands and feet. Next to therapeutic climbing as a health application, such a system shows potential to become popular in the entertainment industry.