Enhancing Image Generation of Diffusion Models with Structural Image Guidance
摘要
ControlNet incorporates structural control images into diffusion models, enhancing controllability in text-to-image generation. However, it requires training on millions of paired samples over days or even months, consuming substantial resources, which poses challenges for ordinary users to develop new model types. Moreover, ControlNet exhibits limited generalization across different control types. Based on this, this paper proposes StrControl, which fine-tunes a Scribble model pre-trained on a large-scale dataset. Channel attention SECA is introduced into the middle layer to enhance the perception of structural information, while zero-convolution layers in the original ControlNet are replaced with cross-normalization to improve the robustness of shallow feature extraction, thereby stabilizing the structural enhancement effect of SECA. Building on these structural optimizations, StrControl further integrates LoRA into the linear layers of the attention modules, enabling lightweight adaptation and enhancing flexibility in handling complex control images. Experimental results demonstrate that the proposed method maintains high image quality under limited-sample conditions while effectively highlighting the subject and aligning with semantic content.