Hand gesture reconstruction plays a crucial role in enabling a range of interactive technologies such as human-computer interaction, sign language interpretation, virtual reality simulation, and more. Current methodologies predominantly rely on wearable devices like gloves or wristbands to capture hand gestures, which often involves high deployment costs and can result in a disruptive user experience. Alternatively, some techniques leverage vision-based systems, but these can suffer from challenges of varying lighting conditions and potential privacy concerns. In this chapter, we introduce mmHand, a novel system for 3D hand pose reconstruction based on mmWave radar signals. This approach enables the generation of 3D hand skeletons and the reconstruction of 3D hand meshes. First, mmHand uses mmWave signals to detect the hand and process the acquired mmWave data. Then, it applies a custom-designed attention-driven hourglass network, mmSpaceNet, to capture spatial features and employs LSTM networks to extract temporal characteristics. With the extracted features, mmHand then performs a regression to predict hand joint locations in 3D space, generating accurate 3D hand skeletons. Lastly, 3D hand meshes that provide a detailed and continuous representation of hand poses are reconstructed through a hand Model with Articulated and Non-rigid defOrmations (MANO). Extensive experimental evaluations show that mmHand achieves a mean per-joint position error of 18.3 mm and 95.1% accuracy, demonstrating its effectiveness in hand pose estimation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

mmWave-based Hand Gesture Reconstruction

  • Hao Kong,
  • Jiadi Yu,
  • Xuemin Sherman Shen

摘要

Hand gesture reconstruction plays a crucial role in enabling a range of interactive technologies such as human-computer interaction, sign language interpretation, virtual reality simulation, and more. Current methodologies predominantly rely on wearable devices like gloves or wristbands to capture hand gestures, which often involves high deployment costs and can result in a disruptive user experience. Alternatively, some techniques leverage vision-based systems, but these can suffer from challenges of varying lighting conditions and potential privacy concerns. In this chapter, we introduce mmHand, a novel system for 3D hand pose reconstruction based on mmWave radar signals. This approach enables the generation of 3D hand skeletons and the reconstruction of 3D hand meshes. First, mmHand uses mmWave signals to detect the hand and process the acquired mmWave data. Then, it applies a custom-designed attention-driven hourglass network, mmSpaceNet, to capture spatial features and employs LSTM networks to extract temporal characteristics. With the extracted features, mmHand then performs a regression to predict hand joint locations in 3D space, generating accurate 3D hand skeletons. Lastly, 3D hand meshes that provide a detailed and continuous representation of hand poses are reconstructed through a hand Model with Articulated and Non-rigid defOrmations (MANO). Extensive experimental evaluations show that mmHand achieves a mean per-joint position error of 18.3 mm and 95.1% accuracy, demonstrating its effectiveness in hand pose estimation.