<p>Facial expression reconstruction technology offers considerable potential in areas like human-computer interaction, affective computing, and virtual reality. To tackle the privacy challenges and environmental constraints inherent in traditional camera-based systems, researchers have recently introduced ear-worn devices as a viable solution. Nevertheless, these methods still demand enhancements, particularly in aspects such as design appeal and energy efficiency, to achieve broader applicability and practical use. This paper introduces a system called IMUFace. It uses inertial measurement units (IMUs) embedded in wireless earphones to detect subtle ear movements caused by facial muscle activities, allowing for covert and low-power facial reconstruction. A user study involving 12 participants was conducted, and a deep learning model named IMUTwinTrans was proposed. The results show that IMUFace can accurately predict users’ facial landmarks with a precision of 2.21 mm, using only five minutes of training data. The predicted landmarks can be utilized to reconstruct a three-dimensional facial model. IMUFace operates at a sampling rate of 30 Hz with a relatively low power consumption of 58 mW. The findings validate the feasibility of IMUFace and highlight potential directions for further research towards its practical adoption in mobile environments. </p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

IMUFace: towards always-on 3D facial reconstruction via earphone inertial sensing

  • Xianrong Yao,
  • Lingde Hu,
  • Dong She,
  • Yincheng Jin,
  • Yang Gao,
  • Zhanpeng Jin

摘要

Facial expression reconstruction technology offers considerable potential in areas like human-computer interaction, affective computing, and virtual reality. To tackle the privacy challenges and environmental constraints inherent in traditional camera-based systems, researchers have recently introduced ear-worn devices as a viable solution. Nevertheless, these methods still demand enhancements, particularly in aspects such as design appeal and energy efficiency, to achieve broader applicability and practical use. This paper introduces a system called IMUFace. It uses inertial measurement units (IMUs) embedded in wireless earphones to detect subtle ear movements caused by facial muscle activities, allowing for covert and low-power facial reconstruction. A user study involving 12 participants was conducted, and a deep learning model named IMUTwinTrans was proposed. The results show that IMUFace can accurately predict users’ facial landmarks with a precision of 2.21 mm, using only five minutes of training data. The predicted landmarks can be utilized to reconstruct a three-dimensional facial model. IMUFace operates at a sampling rate of 30 Hz with a relatively low power consumption of 58 mW. The findings validate the feasibility of IMUFace and highlight potential directions for further research towards its practical adoption in mobile environments.