Motion feature estimation using bi-directional GRU for skeleton-based dynamic hand gesture recognition

Dynamic hand gesture recognition continues to be an interesting field in computer vision applications. Occlusion and background clutter make dynamic hand gesture recognition challenging. In this study, we proposed two parallel pipelines. The first pipeline uses skeleton data to generate a skeleton point trajectory video where the fingertips are tracked across the frame and a trajectory video is created. The use of skeleton data overcomes the challenges of occlusion and complex background. Similarly, in the second pipeline optical flow videos are calculated from RGB/Depth data that capture the motion information of the moving hand. Creation of an optical flow video filters out irrelevant data and concentrates on the gesturing hand that helps in extracting spatio-temporal information. Then, features are extracted parallelly from both pipelines using pre-trained Xception-Net. The created feature vector is passed to the Bi-GRU unit for sequence-to-sequence learning. At the feature level, features of both Bi-GRU networks are averagely fused and flattened at the FC layer and the Softmax classifier is used to classify the gesture. We tested our proposed model on two benchmark datasets, namely NWUHG dataset and the DHG-14/28 dataset. The proposed model achieved \(99.2\%\) $99.2\%$ accuracy on NWUHG, \(98.1\%\) $98.1\%$ on DHG-14, and \(94.2\%\) $94.2\%$ on DHG-28, i.e., comparable with the state-of-the-art methods.