Scholarlink

Large language models (LLMs), as exemplified by the GPT series, have demonstrated exceptional performance in NLP tasks. In this work, we attempt to augment these LLMs with speech data training to achieve end-to-end comprehension and generation capabilities for the speech modality, thereby developing models with inherent dual-modal capabilities in both speech and text. To incorporate speech data into language model training, we discretize the speech data and expand the LLM’s vocabulary to include these representations. Compared to speech interaction methods that utilize text as an intermediary auxiliary modality, this architecture allows the model to comprehend the abundant paralinguistic information inherent in speech. Based on this, we utilized Llama as the initial model and experimented with various training strategies. We evaluated the resulting models on semantic understanding tasks, such as Blimp and StoryCloze, and compared their performance with current open-source general speech models. The results indicate that the composition of pre-training data has a significant impact on the model’s performance.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Investigation on Training Strategy for Cross-Modal Large Language Models with Speech and Text

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Hang Zheng,
  • Yuncong Liu,
  • Hankun Wang,
  • Kai Yu

摘要

Large language models (LLMs), as exemplified by the GPT series, have demonstrated exceptional performance in NLP tasks. In this work, we attempt to augment these LLMs with speech data training to achieve end-to-end comprehension and generation capabilities for the speech modality, thereby developing models with inherent dual-modal capabilities in both speech and text. To incorporate speech data into language model training, we discretize the speech data and expand the LLM’s vocabulary to include these representations. Compared to speech interaction methods that utilize text as an intermediary auxiliary modality, this architecture allows the model to comprehend the abundant paralinguistic information inherent in speech. Based on this, we utilized Llama as the initial model and experimented with various training strategies. We evaluated the resulting models on semantic understanding tasks, such as Blimp and StoryCloze, and compared their performance with current open-source general speech models. The results indicate that the composition of pre-training data has a significant impact on the model’s performance.