Diffusion models have shown great promise in generating high-fidelity images from random noise, with advancements in controllable image generation incorporating modalities like subject, style, pose, and edge. However, controlling interaction relationships in generated content remains underexplored due to the control conditions often focusing on the visual aspects, while interaction is a high-level abstract concept involving the mutual influence between entities. To tackle this, we propose a pluggable model, called EIU-IC, comprised of two primary modules for understanding interaction relationships and controlling interaction positions. The Interaction Comprehension module employs positive and negative prompts to guide the model in understanding interactions, coupled with a timestep sampling strategy that shifts focus toward high-level interactions rather than low-level details. The Position Controlling module identifies Objects, Subjects, and Interactions, determines their bounding boxes, and encodes this information into visual tokens. Our model enhances existing text-to-image (T2I) diffusion models by better understanding interaction relationships and precisely controlling interaction positions. Extensive quantitative and qualitative experiments demonstrate the superiority of our approach over existing methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EIU-IC: Enhancing Interaction Understanding in Text-to-Image Generation Models with Interaction Control

  • Chengyang Zhang,
  • Yonghua Zhu,
  • Wenjing Gao

摘要

Diffusion models have shown great promise in generating high-fidelity images from random noise, with advancements in controllable image generation incorporating modalities like subject, style, pose, and edge. However, controlling interaction relationships in generated content remains underexplored due to the control conditions often focusing on the visual aspects, while interaction is a high-level abstract concept involving the mutual influence between entities. To tackle this, we propose a pluggable model, called EIU-IC, comprised of two primary modules for understanding interaction relationships and controlling interaction positions. The Interaction Comprehension module employs positive and negative prompts to guide the model in understanding interactions, coupled with a timestep sampling strategy that shifts focus toward high-level interactions rather than low-level details. The Position Controlling module identifies Objects, Subjects, and Interactions, determines their bounding boxes, and encodes this information into visual tokens. Our model enhances existing text-to-image (T2I) diffusion models by better understanding interaction relationships and precisely controlling interaction positions. Extensive quantitative and qualitative experiments demonstrate the superiority of our approach over existing methods.