Multimodal Face Generation and Manipulation with Collaborative Diffusion Models
摘要
Diffusion models arise as a powerful generative tool recently. Despite the great progress, existing diffusion models mainly focus on unimodal control, i.e., the diffusion process is driven by only one modality of condition. To further unleash the users’ creativity, it is desirable for the model to be controllable by multiple modalities simultaneously, e.g., generating and editing faces by describing the age (text driven) while drawing the face shape (mask driven). In this chapter, we discuss Collaborative Diffusion, where pre-trained unimodal diffusion models collaborate to achieve multimodal face generation and editing without retraining. The key insight is that diffusion models driven by different modalities are inherently complementary regarding the latent denoising steps, where bilateral connections can be established upon. Specifically, we discuss dynamic diffuser, a meta-network that adaptively hallucinates multimodal denoising steps by predicting the spatial-temporal influence functionsInfluence function for each pre-trained unimodal model. Collaborative diffusion not only collaborates generation capabilities from unimodal diffusion models but also integrates multiple unimodal manipulations to perform multimodal editing. Extensive qualitative and quantitative experiments demonstrate the superiority of this framework in both image quality and condition consistency.