Real-time classification of hand movements as a basis for intuitive control of grasp neuroprostheses

Introduction

According to a report from the World Health Organisation [1], each year the population of people with spinal cord injury (SCI) increases by up to 500,000. With grasp neuroprostheses based on functional electrical stimulation (FES), people with tetraplegia are able to perform basic grasping movements [2]. Although most potential users of grasp neuroprosthesis still have residual muscle activities in their upper extremities, established user interfaces do not take these into account. The objective of this work was to develop a method for classifying the user intent in real-time with respect to grasping, based on the electromyogram (EMG) of muscles relevant for grasping. In the future this method is intended to be used for more intuitive control of a grasp neuroprosthesis, helping to reduce the workload of end users.

Material and methods

Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) were chosen as grasp classifiers [3]. To narrow the research focus, we evaluated the performance of Artificial Neural Networks (ANN) with a single convolutional layer in CNN and the long short-term memory (LSTM) layer in RNN models, which helps to overcome the following two problems: vanishing and exploding gradients [5]. The dimension of the input and output vectors corresponds to the number of EMG channels used and the number of prediction classes accordingly. We fed the neural networks with Root Mean Square (RMS) values of the EMG signals.

We used Python with Keras library to train (ANN) models by supervised learning. In order to reduce the chances of model overfitting, the dropout regularization method [4] was applied. Hyperparameters search was performed to find the best configuration by varying LSTM cells in recurrent layer and kernel number in convolutional one. Data from seven healthy subjects (age range 23–41 years; six males and one female) was used. ANN models were trained for each subject individually. The datasets were split into the training (75% of total), the test (25% of total), and the validation (20% of training data) sets. Each model was trained 3 times with the same configuration; given accuracy values are maximum for these trials.

A Field-Programmable Gate Array (FPGA) was selected as an implementation platform for the ANNs, because of its ability to perform mathematical operations in parallel and its form-factor which allows the end product to have wearable size. FPGA board Altera DE2-115 with Cyclone® IV EP4CE115 was used for classifiers implementation. The FPGA chip has 114,480 logic elements with 3,888 Kbits of embedded memory. CNN and RNN scalable architectures were developed on the FPGA with parameterized standalone modules. The trained model was transferred on the FPGA by extracting its weights and biases, converting them into the fixed-point format, and storing them in the embedded memory.

The used FPGA has some hardware restrictions, therefore some of the approximations were implemented as fixed-point (instead of floating-point) data representation and activation functions were substituted by lookup tables (LUT). For evaluation of the consequences of the approximations on classification accuracy, the ANNs were implemented both on Personal Computer (PC) hardware and FPGA.

As stated in [6] to reduce the memory and computation load in CNN implementation without affecting accuracy we removed the fully connected hidden layer and used a smaller number of kernels, which we defined during hyper parameters search.

Results

Mean accuracy for FPGA-based CNN models was 85.23% (maximum 92.25%; minimum 78.37%; standard deviation 4.77%) and 83.30% for RNN (maximum 89.00%; minimum 76.73%; standard deviation 4.36%).

Almost similar prediction accuracies were achieved for floating-point PC and fixed-point (Q10.14) FPGA implementations (Table 1 and Table 2).

The single input calculation time is in the μs range for both FPGA ANN-implementations and therefore negligible in comparison to the feature extraction frame.

Based on the results of the performance evaluation, optimal configurations of 64 kernels and 16 LSTM cells were identified for CNN and RNN, respectively.

Table 3 shows that the CNN resulting model implemented on the FPGA has better accuracy than the RNN but requires more logic elements. Nevertheless, the biggest difference we can see in the memory requirements, due to RNN’s activation function implementation with LUT.

Conclusion

Most research on the use of ANNs for EMG parameter classification is focused on artificial hand prostheses [7], [8], [9]. In this work, we investigated CNN and RNN as potential classification method of grasp patterns and phases for future control of grasp neuroprostheses. The neural networks were trained to recognize three different states of two grasp patterns and the open hand pattern.

The difference in accuracy between PC and FPGA implementations is small. Moreover, it takes in the worst case 112 µs to process single data input, which is much faster than data transfer in some of the communication links, e.g. radiofrequency, and electromechanical delay of a neuroprosthesis [10]. This allows using FPGA implementations for real-time grasp classification and computer implementation for prototyping. On the other hand, achieved accuracy is not enough for medical use and, therefore, improvements have to be made. Among the future steps for enhancing accuracy are incorporating different features, besides RMS, as input to ANNs and potentially more complex ANNs architectures or other ANN types. Possible candidates are U-Net [11] for CNN and the Legendre Memory Unit [12] for RNN.

We used only three different phases of two grasp patterns, which increases the misclassification rate of the ANN’s and results in a relatively high amount of false-positives. As a future extension, we would like to include data from additional grasp patterns such as wrist flexion, wrist extension, pronation, and supination and evaluate the effect on ANN’s robustness and accuracy.