<p>The remote, non-contact assessment of human psychological states is a significant challenge in affective computing, as current video-based methods often focus on single, specific physiological channels. While remote photoplethysmography (rPPG) captures cardiac activity from color changes and motion magnification reveals subtle movements, both suffer from limitations in dynamic scenarios and lack physiological specificity, respectively. This paper proposes a synergistic approach, demonstrating that cardiac-related color and vestibular-related motion can be extracted and fused from a single video stream to create a richer psychological representation. We introduce dual-mag, a unified framework that jointly amplifies and fuses these complementary channels. Our core contributions include: 1) The first supervised learning paradigm for video-based head micro-tremor (HMT) amplification using high-fidelity inertial measurement unit (IMU) data as ground truth, establishing HMT as a quantifiable biomarker; and 2) a novel dual-branch architecture with a physio-dynamic magnification block (PDM-block) and a cross-attention fusion module to learn the deep interplay between signals. Extensive experiments on the large-scale ReMAP dataset show that dual-mag significantly advances the state-of-the-art (SOTA), improving perception accuracy over leading unimodal methods by 7.2% for discrete emotions, 5.6% for continuous emotions, and 8.3% for personality traits. Our work establishes that deep, intra-sensor fusion can unlock unprecedented performance from ubiquitous cameras for remote psychological sensing.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unveiling Hidden Psychological States: Synergistic Fusion of Amplified Color and Motion from Video

  • Yiwei Ru,
  • Qi Li,
  • Yongji Liu,
  • Huijia Wu,
  • Zhaofeng He,
  • Zhenan Sun

摘要

The remote, non-contact assessment of human psychological states is a significant challenge in affective computing, as current video-based methods often focus on single, specific physiological channels. While remote photoplethysmography (rPPG) captures cardiac activity from color changes and motion magnification reveals subtle movements, both suffer from limitations in dynamic scenarios and lack physiological specificity, respectively. This paper proposes a synergistic approach, demonstrating that cardiac-related color and vestibular-related motion can be extracted and fused from a single video stream to create a richer psychological representation. We introduce dual-mag, a unified framework that jointly amplifies and fuses these complementary channels. Our core contributions include: 1) The first supervised learning paradigm for video-based head micro-tremor (HMT) amplification using high-fidelity inertial measurement unit (IMU) data as ground truth, establishing HMT as a quantifiable biomarker; and 2) a novel dual-branch architecture with a physio-dynamic magnification block (PDM-block) and a cross-attention fusion module to learn the deep interplay between signals. Extensive experiments on the large-scale ReMAP dataset show that dual-mag significantly advances the state-of-the-art (SOTA), improving perception accuracy over leading unimodal methods by 7.2% for discrete emotions, 5.6% for continuous emotions, and 8.3% for personality traits. Our work establishes that deep, intra-sensor fusion can unlock unprecedented performance from ubiquitous cameras for remote psychological sensing.