Optimizing low-resource language encoders for text-to-image generation: a case study on Thai
摘要
Diffusion models have made remarkable progress in generating high-quality images based on user input prompts. However, most open-source models are designed to support only English text prompts, creating challenges for users of non-English languages. Commonly used translation methods fail to address cultural nuances in image generation, and training models from scratch on language-specific data is prohibitively costly. This study introduces a lightweight solution for training low-resource text encoders, using a cross-lingual teacher-student learning approach to guide the image generation process in Stable Diffusion effectively. Extensive experiments on both general and culturally relevant benchmarks were conducted to assess image quality, alignment with human preferences, and visual appeal. In the Thai cultural benchmark, our method outperforms GlueGen as reported by Qin et al (Proceedings of the IEEE/CVF International Conference on Computer Vision, 23085–23096, 2023.), the current state-of-the-art PEFT technique, achieving 100% winning rate on CLIPScore, ImageReward, and PickScore, with 81% on VQAScore and 85% on Aesthetic Score. Importantly, the integration of enriched synthetic data significantly improves both performance and efficiency in cross-lingual learning across all benchmarks. These advancements offer a pathway for broader adoption of text-to-image models for low-resource languages.