Integrating Text-to-Image and Vision Language Models for Synergistic Dataset Generation: The Creation of Synergy-General-Multimodal Pairs

This study presents the creation of the Synergy-General-Multimodal Pairs dataset through an innovative integration of vision language models (VLMs) and text-to-image (T2I) technologies. The code and dataset used in this research are publicly available for replication and further research. The code can be accessed at GitHub Repository and the dataset at Dataset Link . We developed a cyclical generation process that begins with generating initial narratives using either VLMs or large language models (LLMs), which are then visualized by a T2I model. This initiates a feedback loop where each generated image inspires a new narrative, creating a rich sequence of text-image pairs. This iterative approach enhances the diversity and complexity of the dataset, fostering advancements in multimodal research by providing a voluminous and varied resource. Key experimental results show significant improvements: the mean BERTScore increased by 15% (from 0.54 to 0.625), BLEU score by 20% (from 0.026 to 0.032), and ROUGE-L score by 18% (from 0.20 to 0.235). These results demonstrate substantial enhancements in the multimodal model’s performance. The dataset is specifically designed to support the development and fine-tuning of models for enhanced performance and generalization in tasks requiring deep multimodal understanding and generation.