The scalability of ambient sound generators is hindered by data scarcity, insufficient caption quality, and limited scalability in model architecture. This work addresses these challenges by advancing both data and model scaling. First, we propose an efficient and scalable dataset collection pipeline tailored for ambient audio generation, resulting in AutoReCap-XL, the largest ambient audio-text dataset with over 47 million clips. To provide high-quality textual annotations, we propose AutoCap, a high-quality automatic audio captioning model. By adopting a Q-Former module and leveraging audio metadata, AutoCap  substantially enhances caption quality, reaching a CIDEr score of 83.2, a <InlineEquation ID="IEq1"> <EquationSource Format="TEX">$3.2\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>3.2</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> improvement over previous captioning models. Finally, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. We demonstrate its benefits from data scaling with synthetic captions as well as model size scaling. When compared to baseline audio generators trained at similar size and data scale, GenAu  obtains significant improvements of <InlineEquation ID="IEq2"> <EquationSource Format="TEX">$4.7\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>4.7</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> in FAD score, <InlineEquation ID="IEq3"> <EquationSource Format="TEX">$22.65\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>22.65</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> in IS, and <InlineEquation ID="IEq4"> <EquationSource Format="TEX">$13.5\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>13.5</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> in CLAP score. Our code, model checkpoints, and dataset are publicly available.

Taming Data and Transformers for Audio Generation

期刊论文
出版模式：

开放获取
发表日期：

2026年1月28日

查看全文

Moayed Haji-Ali,
Willi Menapace,
Aliaksandr Siarohin,
Guha Balakrishnan,
Vicente Ordonez

摘要

The scalability of ambient sound generators is hindered by data scarcity, insufficient caption quality, and limited scalability in model architecture. This work addresses these challenges by advancing both data and model scaling. First, we propose an efficient and scalable dataset collection pipeline tailored for ambient audio generation, resulting in AutoReCap-XL, the largest ambient audio-text dataset with over 47 million clips. To provide high-quality textual annotations, we propose AutoCap, a high-quality automatic audio captioning model. By adopting a Q-Former module and leveraging audio metadata, AutoCap substantially enhances caption quality, reaching a CIDEr score of 83.2, a $3.2\%$ $3.2 %$ improvement over previous captioning models. Finally, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. We demonstrate its benefits from data scaling with synthetic captions as well as model size scaling. When compared to baseline audio generators trained at similar size and data scale, GenAu obtains significant improvements of $4.7\%$ $4.7 %$ in FAD score, $22.65\%$ $22.65 %$ in IS, and $13.5\%$ $13.5 %$ in CLAP score. Our code, model checkpoints, and dataset are publicly available.