Scholarlink

<p>Direct speech-to-speech translation (S2ST) systems have emerged as a promising approach for real-time cross-lingual communication. However, these systems face significant challenges in balancing translation quality with decoding efficiency. In this paper, we present DESpeech, a novel direct S2ST model that effectively addresses this challenge through a dual-pass encoder architecture. Our architecture decomposes translation into two specialized stages: acoustic feature extraction via a speech encoder and semantic understanding via a text encoder. This modular design enables optimal resource allocation while maintaining cross-modal information flow. To enhance performance, DESpeech employs discrete units as intermediate representations and adopts a multi-task learning framework that integrates automatic speech recognition and speech-to-text translation as auxiliary tasks. The dual-pass architecture allows for efficient pre-training integration and provides a natural framework for balancing computational efficiency with translation accuracy. Experiments on the CVSS-C and GigaS2S datasets show that DESpeech consistently outperforms or matches existing methods in terms of translation quality while achieving clear improvements in inference speed, indicating a promising approach for efficient S2ST with minimal quality degradation.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

DESpeech: a dual-pass encoder approach for efficient speech-to-speech translation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月9日

查看全文
  • Zichao Li,
  • Yanyan Xu,
  • Dengfeng Ke,
  • Kaile Su

摘要

Direct speech-to-speech translation (S2ST) systems have emerged as a promising approach for real-time cross-lingual communication. However, these systems face significant challenges in balancing translation quality with decoding efficiency. In this paper, we present DESpeech, a novel direct S2ST model that effectively addresses this challenge through a dual-pass encoder architecture. Our architecture decomposes translation into two specialized stages: acoustic feature extraction via a speech encoder and semantic understanding via a text encoder. This modular design enables optimal resource allocation while maintaining cross-modal information flow. To enhance performance, DESpeech employs discrete units as intermediate representations and adopts a multi-task learning framework that integrates automatic speech recognition and speech-to-text translation as auxiliary tasks. The dual-pass architecture allows for efficient pre-training integration and provides a natural framework for balancing computational efficiency with translation accuracy. Experiments on the CVSS-C and GigaS2S datasets show that DESpeech consistently outperforms or matches existing methods in terms of translation quality while achieving clear improvements in inference speed, indicating a promising approach for efficient S2ST with minimal quality degradation.