Scholarlink

<p>We address the challenging task of text-driven 3D human-object interaction (HOI) motion generation. Existing methods primarily rely on a direct text-to-HOI mapping, which suffers from three key limitations due to the significant cross-modality gap: (Q1) sub-optimal human motion, (Q2) unnatural object motion, and (Q3) weak interaction between humans and objects. To address these challenges, we propose MP-HOI, a novel framework grounded in four core insights. (1) Multimodal data priors: We leverage multimodal data (text, image, pose/object) from large multimodal models as priors to guide HOI generation, which tackles (Q1) and (Q2) in data modeling. (2) Enhanced object representation: We improve existing object representations by incorporating geometric keypoints, contact features, and dynamic properties, enabling expressive object representations, which tackle (Q2) in data representation. (3) Multimodal-aware mixture-of-experts (MoE) model: We propose a modality-aware MoE model for an effective multimodal feature fusion paradigm, which tackles (Q1) and (Q2) in feature fusion. (4) Cascaded diffusion with interaction supervision: We design a cascaded diffusion framework that progressively refines human-object interaction features under dedicated supervision, which tackles (Q3) in interaction refinement. Comprehensive experiments demonstrate that MP-HOI outperforms existing approaches in generating high-fidelity and fine-grained HOI motions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal prior-augmented text-driven 3D human-object interaction generation

  • 期刊论文

  • 发表日期：

    2026年4月16日

查看全文
  • Yin Wang,
  • Ziyao Zhang,
  • Zhiying Leng,
  • Haitian Liu,
  • Frederick W. B. Li,
  • Mu Li,
  • Xiaohui Liang

摘要

We address the challenging task of text-driven 3D human-object interaction (HOI) motion generation. Existing methods primarily rely on a direct text-to-HOI mapping, which suffers from three key limitations due to the significant cross-modality gap: (Q1) sub-optimal human motion, (Q2) unnatural object motion, and (Q3) weak interaction between humans and objects. To address these challenges, we propose MP-HOI, a novel framework grounded in four core insights. (1) Multimodal data priors: We leverage multimodal data (text, image, pose/object) from large multimodal models as priors to guide HOI generation, which tackles (Q1) and (Q2) in data modeling. (2) Enhanced object representation: We improve existing object representations by incorporating geometric keypoints, contact features, and dynamic properties, enabling expressive object representations, which tackle (Q2) in data representation. (3) Multimodal-aware mixture-of-experts (MoE) model: We propose a modality-aware MoE model for an effective multimodal feature fusion paradigm, which tackles (Q1) and (Q2) in feature fusion. (4) Cascaded diffusion with interaction supervision: We design a cascaded diffusion framework that progressively refines human-object interaction features under dedicated supervision, which tackles (Q3) in interaction refinement. Comprehensive experiments demonstrate that MP-HOI outperforms existing approaches in generating high-fidelity and fine-grained HOI motions.