Recent advancements in text-to-image synthesis (TIS) models, such as DALL-E 3 and stable diffusion XL, have significantly improved image quality. However, these models typically require experts with artistic and photographic expertise to create model-preferred prompts for high-quality image generation, a requirement that challenges non-experts in real-world applications. This study proposes PROMPTIST, an automated framework that optimizes user-original prompts into model-preferred prompts, thereby enabling TIS models to produce more aesthetically pleasing images. We employ the retrieval augmented generation (RAG) technique with Large Language Models (LLMs) to automatically refine prompts, ensuring that the proposed framework is broadly applicable and extends beyond the specific models’ styles like stable diffusion or DALL-E. Furthermore, to ensure the LLMs generate visually appealing images, we fine-tuned a quality assessment model to analyze prompt quality and provide necessary feedback to the LLMs. The proposed prompt engineering method for LLMs can significantly enhance the quality of both prompts and images. Extensive experimental results consistently demonstrate PROMPTIST’s superiority over strong baselines.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PROMPTIST: Automated Prompt Optimization for Text-to-Image Synthesis

  • WeiJie Li,
  • Jin Wang,
  • Xuejie Zhang

摘要

Recent advancements in text-to-image synthesis (TIS) models, such as DALL-E 3 and stable diffusion XL, have significantly improved image quality. However, these models typically require experts with artistic and photographic expertise to create model-preferred prompts for high-quality image generation, a requirement that challenges non-experts in real-world applications. This study proposes PROMPTIST, an automated framework that optimizes user-original prompts into model-preferred prompts, thereby enabling TIS models to produce more aesthetically pleasing images. We employ the retrieval augmented generation (RAG) technique with Large Language Models (LLMs) to automatically refine prompts, ensuring that the proposed framework is broadly applicable and extends beyond the specific models’ styles like stable diffusion or DALL-E. Furthermore, to ensure the LLMs generate visually appealing images, we fine-tuned a quality assessment model to analyze prompt quality and provide necessary feedback to the LLMs. The proposed prompt engineering method for LLMs can significantly enhance the quality of both prompts and images. Extensive experimental results consistently demonstrate PROMPTIST’s superiority over strong baselines.