TAD: A Plug-and-Play Task Arithmetic Approach for Augmenting Diffusion Models
摘要
Artificial Intelligence Generated Content (AIGC) is quickly becoming popular and widely used. Many projects now use a single pre-trained model, like Stable Diffusion (SD), as a base and then adjust it for specific tasks. Usually, modifying these models requires a lot of computing resources and time. In our paper, we propose a new paradigm of the utilization of SD models by task arithmetic achieving plug-and-play without additional training process. To be specific, we find SD models have an extraordinary capacity to digest other task-specified SD models that have been fine-tuned on specific datasets. This allows a single model to encompass the capabilities of multiple models, addressing issues in multi-task scenarios. We conduct model manipulation in the following paradigms: 1) Enhancement through the principle of double negation, which uses task arithmetic to enhance features by leveraging models originally designed to weaken them. 2) Multi-tasks are achieved through the addition of task vectors. Since different SD models for specific tasks possess unique groups of parameters, we combine those parameters together in just one SD model. To verify the effectiveness, we conduct experiments to apply our method in text2image generation tasks on three conditionally generated categories: 1) object (e.g. Snoopy), 2) style (e.g. Monet), and 3) conditional control (e.g. depth map). Our results prove that the aforementioned different categories can be combined into one single model, without training according to the proposed paradigm. We also evaluate the similarity score of the outcomes from different task vectors and reveal that the integrated model not only conserves storage conserves computing resources and reduces time expenditure, but also improves performance, without extra training or fine-tuning.