Decoupling Control in Text-to-Image Diffusion Models

Large text-to-image models allow for high-quality and diverse synthesis of images from a given text prompt. However, many scenarios require that the content creation be controllable. Recent methods add image-level controls, e.g., edge and depth maps, to manipulate the generation process together with text prompts to obtain desired images. In this work, we propose a decoupling control to disentangle one or multiple objects and individual objects’ shapes and appearances in a given reference set while synthesizing novel renditions and rearranging them in different contexts. Given a set of images as input, we establish mapping relationships between the target’s appearance and different “circles” through fine-tuning a pretrained text-to-image model. We achieve control over the local position of different “circles” by designing a novel local feature loss to decouple multi-targets. Extensive experiments demonstrate that our model can disentangle individual objects and allow for their translation within a scene, as well as arbitrary control over the combination of multiple targets while maintaining appearance consistency among the targets.