The cocktail party problem remains a fundamental challenge in single-channel speech separation. Models trained on English datasets (WSJ0-2Mix, LibriMix) perform well in-domain but often fail to generalize to Mandarin. We address this gap with Aishell1Mix, the first open-source Mandarin separation dataset, constructed by mixing AISHELL-1 utterances and optionally adding WHAM!-style noise. In this work, we define robust separation as generalization across (i) tonal characteristics of Mandarin, (ii) diverse speakers and speaking styles, and (iii) realistic noisy acoustic environments. We further define scalable as (a) a reproducible, extensible mixture-generation pipeline (two/three speakers, min/max modes, LUFS normalization, SNR control) and (b) efficient adaptation of large audio language models (ALMs) via parameter-efficient tuning (LoRA). Trained on Aishell1Mix, MossFormer2 attains 14.97 dB SI-SNRi (two-speaker) and 13.55 dB (three-speaker), substantially outperforming English-trained counterparts. In addition, fine-tuning qwen2_audio_7b_instruct with LoRA markedly improves speaker-count detection and multi-speaker transcription on overlapping Mandarin speech. Our dataset and recipes provide strong baselines and a scalable path toward robust Mandarin speech separation and audio-language modeling.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Aishell1Mix: Towards Robust Mandarin Speech Separation with Scalable Audio Language Models

  • Zijian Huang,
  • Cem Subakan

摘要

The cocktail party problem remains a fundamental challenge in single-channel speech separation. Models trained on English datasets (WSJ0-2Mix, LibriMix) perform well in-domain but often fail to generalize to Mandarin. We address this gap with Aishell1Mix, the first open-source Mandarin separation dataset, constructed by mixing AISHELL-1 utterances and optionally adding WHAM!-style noise. In this work, we define robust separation as generalization across (i) tonal characteristics of Mandarin, (ii) diverse speakers and speaking styles, and (iii) realistic noisy acoustic environments. We further define scalable as (a) a reproducible, extensible mixture-generation pipeline (two/three speakers, min/max modes, LUFS normalization, SNR control) and (b) efficient adaptation of large audio language models (ALMs) via parameter-efficient tuning (LoRA). Trained on Aishell1Mix, MossFormer2 attains 14.97 dB SI-SNRi (two-speaker) and 13.55 dB (three-speaker), substantially outperforming English-trained counterparts. In addition, fine-tuning qwen2_audio_7b_instruct with LoRA markedly improves speaker-count detection and multi-speaker transcription on overlapping Mandarin speech. Our dataset and recipes provide strong baselines and a scalable path toward robust Mandarin speech separation and audio-language modeling.