VT Prompt Tuning: Improving Portrait Generation with Combined Visual Prompt and Text Prompt in Stable Diffusion Model
摘要
The generative capabilities of large-scale text-to-image diffusion models have achieved remarkable success in producing high-fidelity images. However, generating specific images using only text prompts often requires intricate prompt engineering. An alternative approach is to use image prompts, as images can convey rich and detailed information that text alone may not capture. Existing methods that rely on directly fine-tuning pretrained models are effective but demand significant computational resources and often lack flexibility with other base models, text prompts, and structural controls. We explore a method to improve the quality of person generation by combining visual prompts and text prompts as input to the Stable Diffusion model. Specifically, we propose a strategy to extract visual features (visual prompt) from a specific person’s portrait image and combine them with other text prompts (such as scene prompt, posture prompt, and so on) to generate an image of a very similar person, but with arbitrary scene or postures. Experimental results show that our approach significantly enhances the similarity between the generated image and the target person. Our method achieves an average score of 4 out of 5 in human evaluations, outperforming baseline models.