Scholarlink

<p>Video-conditioned background music generation under licensed-data constraints requires models to use sparse visual dynamics, dense textual intent, and acoustic context without destabilizing a pretrained audio prior. We present Adaptive Multimodal Composition (AMC), a parameter-efficient visual–text–audio adaptation framework that freezes the AudioLDM acoustic backbone and trains lightweight Temporal Alignment (TAM), Adaptive Multimodal Fusion (AMF), Cross-Modal Transformer (CMT), and Temporal Semantic Fusion (TSF) components. TAM places visual and acoustic features on a compatible one-second temporal grid; AMF estimates block-wise condition-specific usefulness; CMT models non-local structure in the aligned fused sequence; and TSF supplies a training-only, one-way multimodal-to-text semantic anchor. Under a controlled 100-h licensed-data protocol, AMC obtains FAD/KL scores of 1.66/0.19 on the complete in-domain evaluation while updating 4.8% of system parameters. On MusicCaps, where no paired video exists, masked visual–text–audio inference obtains 2.01/0.21 and is interpreted only as out-of-domain semantic transfer. Learned reliability weights correlate significantly with empirical marginal utility, with Spearman correlations of 0.52, 0.60, and 0.46 for the visual, text, and audio-context modalities, respectively (all <i>p</i> &lt; 0.001), and 30/60-s evaluations provide evidence that CMT improves tempo stability, harmonic continuity, and recurrence organization. Code, configuration files, split metadata, evaluation scripts, and representative generated examples are available at: <a href="https://github.com/Music-aitech/AMC-main.git">https://github.com/Music-aitech/AMC-main.git</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Low-resource video-conditioned music generation via reliability-aware visual–text–audio fusion

  • 期刊论文

  • 发表日期：

    2026年7月9日

查看全文
  • Yanwei Zhang,
  • Yiyu Yang

摘要

Video-conditioned background music generation under licensed-data constraints requires models to use sparse visual dynamics, dense textual intent, and acoustic context without destabilizing a pretrained audio prior. We present Adaptive Multimodal Composition (AMC), a parameter-efficient visual–text–audio adaptation framework that freezes the AudioLDM acoustic backbone and trains lightweight Temporal Alignment (TAM), Adaptive Multimodal Fusion (AMF), Cross-Modal Transformer (CMT), and Temporal Semantic Fusion (TSF) components. TAM places visual and acoustic features on a compatible one-second temporal grid; AMF estimates block-wise condition-specific usefulness; CMT models non-local structure in the aligned fused sequence; and TSF supplies a training-only, one-way multimodal-to-text semantic anchor. Under a controlled 100-h licensed-data protocol, AMC obtains FAD/KL scores of 1.66/0.19 on the complete in-domain evaluation while updating 4.8% of system parameters. On MusicCaps, where no paired video exists, masked visual–text–audio inference obtains 2.01/0.21 and is interpreted only as out-of-domain semantic transfer. Learned reliability weights correlate significantly with empirical marginal utility, with Spearman correlations of 0.52, 0.60, and 0.46 for the visual, text, and audio-context modalities, respectively (all p < 0.001), and 30/60-s evaluations provide evidence that CMT improves tempo stability, harmonic continuity, and recurrence organization. Code, configuration files, split metadata, evaluation scripts, and representative generated examples are available at: https://github.com/Music-aitech/AMC-main.git.