Efficient Sensing Network and Decoupled Neural Model for Hand Pose Estimation

Hand pose estimation (HPE) is a key technology for various Internet of Things (IoT) applications, such as sign language recognition, smart healthcare, and augmented reality/virtual reality experiences. Existing HPE methods based on visual or wearable sensing have limitations in terms of accuracy, robustness, privacy, and user-friendliness. Addressing these limitations, this paper proposes a novel HPE approach employing an efficient sensing network. This network comprises an inertial measurement unit (IMU) and five tensile sensors integrated within a glove prototype. Based on the kinematic analysis of human hand, this network is designed to capture the essential features of hand movements. To confront the scarcity of publicly available datasets, we devise a technique for synthesizing IMU and tensile sensor data from video sources. A decoupled neural model leveraging transformer and autoencoder is developed to map the sparse sensor data to a complete hand pose. Our methodology not only demonstrates superior accuracy and robustness but also respects user privacy and improves usability, offering a holistic wearable sensor-based solution for HPE and broadening the horizons for IoT implementations.