Retention-Consistent Concept Unlearning for Diffusion Models

Generative models, such as text-to-image diffusion models, have shown exceptional ability in generating high-quality images but pose a risk of generating inappropriate content (e.g., harmful or copyrighted material). Concept unlearning is one of the most prominent methods for ensuring safety and compliance of the diffusion models. However, existing methods overly focus on unlearning itself while neglecting its impact on the retained distribution. Previous studies have shown that the key challenge in unlearning research lies not only in achieving unlearning but also in minimizing the impact on the retained distribution, i.e., maintaining consistency before and after unlearning. The most severe manifestation of this challenge is the unintended degradation of unrelated concepts, which is a common flaw in many concept unlearning methods. These methods cannot sustain unlearning training because excessive unlearning updates lead to severe degradation in model performance on the retained set. Our analysis shows that this inconsistency is derived from a severe imbalance between the unlearning loss and the retention loss in the unlearning process, which leads to destructive updates of the feature space. To address the above challenges, we propose an unlearning framework named Consistent Unlearning, which prioritizes the maintenance of retention consistency. This framework leverages knowledge distillation to rebalance the unlearning and retention losses from multiple perspectives and introduces a feature space optimization objective to ensure the unlearning process is stable and controllable. Our method enables accurate and effective unlearning, making the behavior of the unlearned model on the retained dataset nearly indistinguishable from the original model. Extensive experiments on object, style, celebrity, and NSFW unlearning tasks demonstrate that our method effectively removes target concepts while achieving state-of-the-art consistency on the retained set.