Scholarlink

<p>With the explosion of social networks, automatically generating music has attracted the attention of researchers. This study proposes a video background music generation framework, consisting of four components: a multi-modal feature extractor, a chord sequence generator, a music analyzer, and a post-processor. The feature extractor has a music feature extractor for processing note density, loudness, chords, keys, and instruments, and a video extractor for extracting emotions, semantics, scene offsets, and motion. To address the out-of-chord issue in chord generation, we introduce a k-NN imputation method using a pre-trained word2vec chord embedding model to increase chord diversity and help chord prediction closely match musician prediction. The core of our framework is a novel Hybrid Shared Mixture of Expert Multi-modal Transformer model, named Hybrid SharedMoE-MT, to generate chord sequences. This Transformer-based model adheres to an asymmetric architecture with SwiGLU-based FFN and integrates rotary positional embedding to enhance continual learning, improve contextual awareness, and capture better knowledge. Inspired by DeepSeekMoE, Hybrid SharedMoE-MT leverages a Mixture of Expert specialization strategy to enhance both computational efficiency and model performance. In addition, we introduce a Bi-Mamba+-based music analyzer to estimate loudness and note density and to predict instruments. The post-processor adjusts and aligns the music file with the video. The experimental results on the MuVi-sync dataset demonstrate that our proposed model outperforms existing approaches.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Video background music generation using hybrid shared mixture-of-experts multimodal Transformer

  • 期刊论文

  • 发表日期：

    2026年7月7日

查看全文
  • Khang Nhut Lam,
  • Thuan Trung Phan,
  • Khang Minh Duong,
  • Thuy Phuong Nguyen,
  • Tai Tran-Anh Pham,
  • Van Lam Le,
  • Jugal Kalita

摘要

With the explosion of social networks, automatically generating music has attracted the attention of researchers. This study proposes a video background music generation framework, consisting of four components: a multi-modal feature extractor, a chord sequence generator, a music analyzer, and a post-processor. The feature extractor has a music feature extractor for processing note density, loudness, chords, keys, and instruments, and a video extractor for extracting emotions, semantics, scene offsets, and motion. To address the out-of-chord issue in chord generation, we introduce a k-NN imputation method using a pre-trained word2vec chord embedding model to increase chord diversity and help chord prediction closely match musician prediction. The core of our framework is a novel Hybrid Shared Mixture of Expert Multi-modal Transformer model, named Hybrid SharedMoE-MT, to generate chord sequences. This Transformer-based model adheres to an asymmetric architecture with SwiGLU-based FFN and integrates rotary positional embedding to enhance continual learning, improve contextual awareness, and capture better knowledge. Inspired by DeepSeekMoE, Hybrid SharedMoE-MT leverages a Mixture of Expert specialization strategy to enhance both computational efficiency and model performance. In addition, we introduce a Bi-Mamba+-based music analyzer to estimate loudness and note density and to predict instruments. The post-processor adjusts and aligns the music file with the video. The experimental results on the MuVi-sync dataset demonstrate that our proposed model outperforms existing approaches.