Recognition analysis of spiral and straight-line drawings in tremor assessment

Hungarian drawing database

The drawing database included recordings from healthy and patients with neurological conditions. All subjects were informed in advance about the purpose and details of the research, and they consented to the recording and their use in the research. The recordings were conducted using an external electronic drawing board (Genius MousePen i608X). A clinical neurosurgeon acknowledged the diagnosis, right-handedness, and tremor presence in the patients. Patients with PD and SP were assessed according to the Unified Parkinson’s Disease Rating Scale (UPDRS) motor score [13]. Patients with ET and CD were evaluated based on the Drawing B task (Spiral) from the Fahn-Tolosa-Marin Tremor Rating Scale (FTM) [33]. All patients with PD and SP were assessed using the scoring task for better comparison, as presented in Table 1. PD patients may present features of micrographia and tremors in rare cases during the drawing sessions.

At least one spiral (standard Archimedes’ spiral) and one straight line pattern-based drawing for the right hand were recorded from each person. These drawing patterns were selected from the FTM Tremor Rating Scale. During the recording, the subjects had to follow the drawing patterns placed under the cover of the drawing board in the following way. In the case of the spiral drawing pattern, they started from the center of the spiral and moved between the lines of the spiral pattern to arrive at the end of the pattern (right-directional spiral). In the case of the line, the subjects moved between two bounding lines from a starting point to the endpoint from left to right, resulting in a horizontal line. The diameter for the spiral pattern was 5 cm and included four rounds. For the line drawing, the distance between the starting and end points was 17 cm, and 0.5 cm between the boundaries vertically. In both cases, the palm and the fingers did not touch the drawing cover to prevent compensatory maneuvers.

X and Y coordinates and the pressure value at the pen tip were recorded. The resolution of the drawing board was 14,668 × 11,334, the range of the pressure value was 0–1,023, and the sampling frequency was 110 Hz.

The database included 29 Essential tremor and 13 other Cerebellar Dysfunction patients. These were subsequently treated together as cerebellar symptoms (CS). In addition, 26 patients with Parkinson’s disease and 51 patients with Secondary Parkinsonism were treated together for Parkinson-like symptoms (PS). 24 healthy individuals were also included as a control group (HC). The detailed structure of the database is contained in Table 1. The mean age and the sex distribution of the HC group appear to be significant (using the Mann-Whitney test). However [34], found no statistically significant difference in how men and women drew the spiral under guided drawing conditions.

Image pre-processing

The raw data were centered on unifying drawing images. So, each drawing started from the (0,0) point. The coordinate values were normalized between −1 and 1. Two versions of the images were created: (1) colored images, in which pressure information was added, and (2) monochrome without pressure information. Instead of line width, colorization has been chosen to include pressure because initial experiments have shown that the applied neural network structures could learn more information this way. As a result, four images were created for each person: spiral (colored with pressure and monochrome), straight line (colored with pressure and monochrome).

The resolution of the final images was 224 × 224 with 24-bit depth (3 color channels) to match the neural network input sizes. This pre-processing phase was done in Python 3.6 with numpy (v1.19.5) and matplotlib (v3.3.4) using brg color map. An example of the final images for the colorized version can be seen in Figure 2.

Figure 2:

Example of a spiral (upper images) and a line (lower images) samples from the database. The left side shows a CT case and the right side shows an HC case. In every case, the left images are monochrome and the right ones are coloured with pin-pressure values. Both types of drawing are square images, which is the reason why the vertical component of the line drawing is stressed (rather than a smooth line for the HC example).

Feature extraction and classification

Two approaches were used for feature extraction: (1) pre-trained models and (2) model from scratch. On the one hand, pre-trained networks were assumed to provide robust features due to their training on a large data set. For this purpose, MobileNet [35], ResNet50 [36] and Xception [37] were examined. All three models were trained on the ImageNet database. The weights were frozen for feature extraction, and the top (original classifier) was not included. The resolution of internal images was uniform (224, 224, 3). On the other hand, a simple convolutional model from scratch was also developed (as a baseline). This model contained five convolutional blocks. The number of convolutional blocks was chosen based on the images’ complexity and the database’s size (smaller and bigger models were also probed initially). Such a convolutional block had a Convolution2D layer (filter number is 8, size 5 × 5, same padding and ReLU activation function), a MaxPooling layer (filter size 2 × 2), and a DropOut (neuron drop ratio: 0.1) layer.

The output of the pre-trained models was transformed into feature vectors using the GlobalPooling layer and the Flatten layer for our simple model. These vectors represented input images in a compact, 1D format. Then, there were Dense layers with 64, 8, and 2 output dimensions, respectively, for all four models. DropOut layers were included with a neuron drop ratio of 0.1 between the Dense layers. Finally, a Softmax activation function was added to the output to generate predictions. A schematic diagram of this procedure is shown in Figure 3.

Figure 3:

Schematic of feature extraction and classification.

Evaluation and metrics

The input images and corresponding labels were randomly shuffled before feeding them to the networks. The labels were transformed into categorical variables. The stratified k-fold method was used for model evaluation, where the number of folds was 10. Adam optimizer (with 0.001 learning rate), categorical cross entropy as loss function and categorical accuracy as monitoring metric were chosen for model compiling.

In the case of pre-trained models, only the classification part was trained with 100 epochs. In the case of the baseline model, it was also necessary to train the convolutional part. Thus, 500 epochs were chosen. These parameters were based on the results achieved on a single 10 % validation set in a preliminary phase. During training, the batch size was 32 by default.

Balanced accuracy and macro f1 score were applied to evaluate the performance of the models. The former is the mean of the sensitivity and specificity (Eq. (1)). It also considers the proportion of the classes (imbalanced data) compared to the traditional accuracy metric. The latter is the harmonic mean of precision and sensitivity (Eq. (2)). The “macro” means the unweighted average of the f₁ value calculated for both labels. Notations in Eqs. (1) and (2) are TP as True Positive, TN as True Negative, FP as False Positive, and FN as False Negative samples.

The following binary classification (HC-PS, HC-CS) experiments were conducted using the evaluation mentioned above with all four models:

1. Classification using spiral drawings

   1. without pressure display

   2. with pressure display

2. Classification using straight-line drawings

   1. without pressure display

   2. with pressure display

3. Aggregation of spiral and straight-line predictions using soft voting

   1. without pressure display

   2. with pressure display

Experiment (1) and (2) supplement the idea of which drawing type carry more information (achieve higher classification performance) related to the disorders. Experiments (1) and (2) also examined the relevance of pressure information in Parkinsonism and cerebellar dysfunction for both spiral and straight-line drawings. Finally, experiment (3) determines whether additional information can be revealed using multiple drawing types per individual. Individual model predictions (per drawing type) are fused together for this.

The final prediction is calculated using Eq. (3), where y_final∈{0,1} is the final prediction, y_s∈{0,1} is the prediction from spiral drawing, y_l∈{0,1} is the prediction from line drawing, and Θ is a weighting between the spiral and line predictions. Θ was evaluated from 0 to 1 with 0.1 steps.

The initial predictions are provided by the Softmax function at the end of the models for each drawing task. These predictions assign samples to the preferred class using a decision threshold. However, a weighted prediction was generated by defining a ratio between the drawing tasks subject-wise using Eq. (3).

The decision threshold for y_final, y_s, y_l was 0.5. In other words, the sample that received a prediction score above 0.5 was classified as a positive sample (PS or CS), while below this number, the sample was assigned to the negative class (HC).

Each cross-validation case was executed five times for each model. Metrics were calculated with the resulting predictions. Finally, these metrics were averaged along the executions.

Mann-Whitney non-parametric statistical test was used on macro f1 scores between the groups in interest to test significance. These groups are defined as the experiment points. The null hypothesis states that the macro f1 scores come from the same population. The significance level was set to 0.05 as a generally used limit. In the Results section, the p-value of the test will be compared to the significance level to assess the null hypothesis.

Results on spiral and straight-line drawings

The results obtained on the spiral drawings are shown in Table 2. The top four rows show the results obtained without pressure mapping and the bottom four with pressure mapping. The first five columns belong to the HC-CS classification, and the last five columns to the HC-PS classification. Notations in the table are sens – sensitivity, spec – specificity, b. acc – balanced accuracy, f1 – macro f1 score, std. f1 – standard deviation of macro f1 score along the five executions.

Between the two classification problems (HC-CS and HC-PS), HC-CS recognition performs better than HC-PS. Without pressure information, the baseline model and MobileNet gave the most prominent difference, 11.2 %, and 12.0 %, between the classification problems. In this case, Xception and ResNet50 performed 6.4 % better on average for the HC-CS than HC-PS. When pressure is applied, the greatest difference is a 14.7 % macro f1 score with ResNet50 between HC-CS and HC-PS. On the other hand, our baseline model and the Xception did not result in a detectable difference.

Bearing the pressure presentation on spiral images resulted as follows. HC-CS recognition was more likely to improve without pressure. At the same time, HC-PS classification slightly decreased without pressure data. In concrete terms, Xception and the baseline model had a 2.6 and 6.7 % decrease in the macro f1 score with pressure. Meanwhile, ResNet50 showed a 2.3 % improvement with pressure. In the case of HC-PS classification, Xception, MobileNet, and the Baseline model show a decrease without pressure data (2.3 % macro f1 score on average). Only ResNet50 showed a 6.1 % improvement without pressure.

Applying the Mann-Whitney test, no significant difference can be observed between the macro f1 scores resulting with and without pressure data for both symptom classification problems (HC-PS p-value: 0.701, HC-CS p-value: 0.435).

Table 3 presents the results of the straight-line drawings. The names of the columns and abbreviations of the metrics are the same as in Table 2. Examining straight lines, there is no longer a considerable difference between the HC-CS and HC-PS results. The maximum macro f1 score difference is 4.7 % using MobileNet without pressure data.

Regarding the pressure use, a similar tendency in results was obtained as with the spiral drawings. In the classification of HC-CS, the macro f1 score drop was 1.4 % (with the Xception) when no pressure data was used. The other models achieved an average 1.9 % better macro f1 score without pressure. In the HC-PS classifications, all four models showed better results with pressure except the ResNet50. However, no significant difference can be measured between the macro f1 scores according to the pressure presentation.

Tables 2 and 3 also show differences between spiral and line results. All models performed better for both classification problems using spiral drawings than lines (comparing them model-wise). The minimum improvement is 2.9 % (Baseline model with pressure), and the maximum gain is 21.0 % (ResNet50 with pressure) for HC-CS. That is a minimum of 1.4 and a maximum of 12.4 increase on average in correctly classified samples. Examining HC-PS results, the Baseline model gained similar results for both drawing tasks (0.6–1% macro f1 score differences). For all other models, the macro f1 score of the spiral classifications exceeded the line ones. The maximum gain was 11.9 % (+8 correctly assigned samples) using Xception with pressure data.

Overall, the macro f1 scores of the spiral experiments appeared significantly different from the straight-line experiments’ (HC-PS p-value<0.000, HC-CS p-value<0.000).

Results of prediction aggregation using soft voting

Figure 4 shows the macro f1 scores obtained by aggregating the predictions of the spiral and line drawings for HC-PS. The left figure contains the results of the experiments with pressure, and the right figure the experiments without pressure. The macro f1 score is shown in percentage on the vertical axis, and the weighting factor (Θ) is presented from Eq. (3) on the horizontal axis. If Θ is 0, then purely line results are shown. If Θ is 1, then purely spiral results are displayed. Models are marked with different colors.

Figure 4:

Macro f1 score results with aggregation of images without pressure data (left side) and of images with pressure data (right side) in HC-PS classification.

There is an increasing trend from lines towards the spiral results. A clear peak can be observed between the extreme points for all models. A maximum 3.0 % macro f1 score increase can be acknowledged at the peak compared to the pure spiral results with the Baseline model on monochrome data. This means classifying two additional samples into the appropriate category. The improvement is more conspicuous when peak values are compared to the line’s f1 scores.

With pressure data, a greater difference can be observed. A minimum of 1.6 % (MobileNet) and a maximum of 3.7 % (Baseline) increase can be seen in the macro f1 scores. Respectively, it is an additional one and three samples classified correctly.

Figure 5 similarly shows the macro f1 score obtained by aggregating the line and spiral results for the HC-CS classifications. The notations are the same as in Figure 4. An increasing trend is also observed. However, there is less of a prominent peak between the two extremes (except the Baseline). On the left figure, all models show a macro f1 score improvement of 0.3–4.7 % compared to the spiral results. The maximum gain is achieved with the Baseline model. These represent 0.2–3 additional correctly classified individuals. Similar results can be seen with pressure data. The improvement is between 0.7 and 5 % in the macro f1 score. This means a maximum of 3.4 more correct samples with the Baseline model.

Figure 5:

Macro f1 score results with aggregation of images without pressure data (left side) and of images with pressure (right side) for HC-CS classification.

Significant differences were perceived between the peak and the extreme points’ (pure spiral or line) macro f1 scores regardless of the pressure inclusion (HC-PS p-value: 0.011, HC-CS p-values: 0.032).