Skeleton-based human activity recognition with wifi CSI using a hybrid approach combining convolutional neural network and long short term memory

Pose estimation based on visual images is evolving, but it is also limited by environmental factors such as occlusion and darkness. Due to its non-intrusive and ubiquitous characters, WiFi Channel State Information (CSI)-based human activity recognition attracts immense attention. In this paper, a CSI-based passive sensing system is proposed to predict joint points of human skeleton for activity recognition. The system leverages a pair of ESP32 based CSI sensors with bidirectional link that can prevent unidirectional link from missing important activity information to collect the amplitude of CSI signals. The Kinect 2.0 is employed to obtain skeleton data as ground truth label synchronously. A hybrid deep neural network composed of Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) is utilized to extract features of CSI signal and map to corresponding human skeleton. K-means clustering algorithm is incorporated to cull the outliers. Experimental results demonstrate that the proposed system achieves satisfactory results with 3.489% average error.