LeakGuard: Detecting and Mitigating Attribute Leakage in Fine-Tuned Diffusion Models
摘要
The widespread use of text-to-image diffusion models like Stable Diffusion raises concerns about inadvertent leakage of confidential information through generated images. While prior studies mainly focused on pretraining data leakage using captions provided during training, they often overlooked realistic scenarios. This paper experimentally demonstrates that fine-tuning Stable Diffusion with a small set of confidential images can lead to unintended and intentional Data extraction, even when general prompts without explicit object keywords are used. We design a multi-modal detection framework that analyzes input prompts, latent representations, and output images to identify potential data leaks. Furthermore, we propose an output-stage Leakguard system that filters confidential images in real-time, reducing the leakage rate by over 71.5%. Through systematic analysis, we reveal that visual attributes of confidential data can persist within the model’s latent space and influence image generation beyond text conditions. Our findings emphasize the necessity of defense strategies to mitigate privacy risks in diffusion models.