Text2Omni is an innovative framework using only text modality to generate high-quality synthetic multimodal data, which aims to eliminate the need of large-scale annotated real multimodal data for training Multimodal Large Language Models (MLLMs). The framework leverages the geometric structure of multimodal contrastive representations to generate diverse, high-quality datasets that facilitate pretraining and instruction-tuning for multimodal models. The process involves a three-stage pipeline: (1) Diverse Caption Data Synthesis; (2) Instruction-Tuning Data Generation; and (3) Modality Representation Transfer. The resulting datasets, Text2Omni-1.8M for pretraining and Text2Omni-540K-Instruction for instruction-tuning, significantly reduce training costs while supporting the development of small- to medium-scale multimodal models. The paper also introduces a two-phase multimodal training paradigm to enhance multimodal understanding and reasoning capabilities of MLLMs efficiently. Experimental results across image-to-text and audio-to-text tasks demonstrate that the Text2Omni framework improves the performance of existing models on a variety of benchmarks, establishing its potential as an effective tool for advancing multimodal learning without requiring large-scale real-world data.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Text2Omni: A Text-Only Training Strategy for MLLMs

  • Junxin Li,
  • Yifu Guo,
  • Zishan Xu,
  • Fengyu Yang,
  • Siyue Chen,
  • Siyan Wu,
  • Lihua Cai

摘要

Text2Omni is an innovative framework using only text modality to generate high-quality synthetic multimodal data, which aims to eliminate the need of large-scale annotated real multimodal data for training Multimodal Large Language Models (MLLMs). The framework leverages the geometric structure of multimodal contrastive representations to generate diverse, high-quality datasets that facilitate pretraining and instruction-tuning for multimodal models. The process involves a three-stage pipeline: (1) Diverse Caption Data Synthesis; (2) Instruction-Tuning Data Generation; and (3) Modality Representation Transfer. The resulting datasets, Text2Omni-1.8M for pretraining and Text2Omni-540K-Instruction for instruction-tuning, significantly reduce training costs while supporting the development of small- to medium-scale multimodal models. The paper also introduces a two-phase multimodal training paradigm to enhance multimodal understanding and reasoning capabilities of MLLMs efficiently. Experimental results across image-to-text and audio-to-text tasks demonstrate that the Text2Omni framework improves the performance of existing models on a variety of benchmarks, establishing its potential as an effective tool for advancing multimodal learning without requiring large-scale real-world data.