▷ 2019 InnovateFPGA: Artificial Neural Network based Accelerometer and Gyroscope recognition for gesture communication

• ➡️ #DigitalSystems #DigitalElectronic #DigitalCircuits #HDL #VHDL #FPGA
• ⭐ https://github.com/vasanza/MSI-VHDL

Description

It is estimated that over 466 million people suffer from a hearing disease, which can cause partial or total hearing loss. The main impact of hearing loss on individuals, is on the ability to communicate with others. In developing countries, most children with hearing loss, do not have access to education.

Over the last decade, there have been a great improvement on artificial neural networks, which allows developing sophisticated systems for pattern recognition, and classification of data from different types and sources. Our proposal is the implementation of a wearable embedded system of pattern recognition and classification, focused on sign language, using the DE10-nano. This project is focused on improving deaf individuals’ communication with those who do not know sign language.

Authors

• Galo Sanchez Granja, estudiante ESPOL (galosanc@fiec.espol.edu.ec) 
• Juan Carlos Solano, estudiante ESPOL (jcsolano@espol.edu.ec)

Demo Video

Project Proposal

1. High-level Project Description

Machine learning algorithms are divided into two subclasses; supervised learning and unsupervised learning. The use of supervised learning algorithms require a large data base of input features and output labels, to adjust and optimize the parameters of the neural network, this process is known as training. A set of 8 electrodes around the arm are used in order to obtain the necessary data. It is also contemplated the use of accelerometer and gyroscope for more complex gesture which involve not only muscular activation but upper limbs movement.

There are several ways to characterize EMG signals, wrong feature selection can lead to poor performance of the neural network. Root mean square voltage is the only feature used to train our neural network, it has shown to be the most representative feature for this application.

Once the Neural Network is trained it is ready to perform real time inferences, of single letter input or whole complex gesture that represent words or actions. For single letter inputs the user has the option to concatenate several letters to form a word.  

2. Block Diagram

3. Intel FPGA virtues in Your Project

•  Optimization on signal preprocessing, feature extraction, and classification, trough custom hardware design.
• Flexibility on the prototyping process allowing comparison between different designs’ performance alternatives.
• Real time inference performance boost due to the compatibility between FPGA and the parellel computing nature of artificial neural networks' topology

4. Design Introduction

The American sign language is composed by many signs that involve upper limb movement, it also includes signs to communicate single letters in order to create words that are unique like names, or just acronyms. For both types of signs, different designs are proposed.

 Accelerometer based classification

For signs that involve upper limb movement, the best way to capture and represent them is using the adxl345 digital accelerometer. The three-axis information its use by the system to identify which sign is the user performing.

EMG based classification

For single letters, the user perform pattern using his hands, these gestures are best represented by EMG signals, the electrical activity of the forearm is by the system to determine which pattern is the user doing to communicate a single letter, and then present as an image.

Machine Learning Algorithm

Machine learning algorithm are used every time there is the need to generalize inputs of the same characteristics and classify each of them whit the most accuracy possible. The algorithm chose for this design is the feed forward neural network, as it is the best suited for pattern recognition and classification.

5. Function Description

To create the neural network on the cyclone v FPGA, the same topology of the nprtool from the neural network toolbox of MATLAB was followed. The training load is carried by a host computer, once all parameters are tune and the classification model is completely optimized, these parameters are exported and included in the FPGA Neural Network model, so it is ready to classify and performance inference on the edge.

The feed forward neural network receives an input of several features, these features most be selected to represent a sample, in a single value. Wrong feature selection can lead to poor classification accuracy. The neural network model working with EMG uses the root mean square voltage to characterize every signal on each electrode, also the placement of every electrode is a feature itself.

Several features were used to design the neural network model to work with the adxl345 accelerometer, each feature was tested individually, and collectively. It was found that the combination of all the features proposed gave the most precise model. The features used are mean value, variance, maximum and minimum value of each axis. Therefore the Neural Network input is made of 12 values that represent one sample.

6. Performance Parameters

Three axis accelerometer Neural Network

 To train the model a dataset of five basic words was made, and for each word 200 samples were taken, every sample was made of 4000 values of the x, y, and z axis during a 2.66 seconds time period. All samples were characterized and the final product was a matrix of input features, and another matrix for the output labels, both matrix were used to train the model, and test how many neurons on the hidden layer were needed to get the best possible accuracy.

With 9 neurons on the hidden layer an accuracy of a 100% on the dataset classification was achieve.  

Every time the nn_adxl program is executed, and the user makes a sing, the systems gives back an array and each value represent how certain the system is that the sign just capture is Hello, Deaf, Me, You, or Sign respectively.

 The EMG Neural Network model still on development 

7. Design Architecture

The DE10-NANO development kit provides with lots of resources to the designer, build around the cyclone v soc FPGA, we took advantage of the analog to digital capabilities, and the adxl345 three axis accelerometer to create an embedded system that combines both software and custom hardware.

8. Reference

• Asanza, V., Peláez, E., Loayza, F., Mesa, I., Díaz, J., & Valarezo, E. (2018, October). EMG Signal Processing with Clustering Algorithms for motor gesture Tasks. In 2018 IEEE Third Ecuador Technical Chapters Meeting (ETCM) (pp. 1-6). IEEE.
• C. Cedeño Z., J. Cordova-Garcia, V. Asanza A., R. Ponguillo and L. Muñoz M., "k-NN-Based EMG Recognition for Gestures Communication with Limited Hardware Resources," 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Leicester, United Kingdom, 2019, pp. 812-817.

9. Read More

• Innovate FPGA
• EMG Signal Processing