A hierarchical multi-modal injection architecture for synergistic music understanding and generation
摘要
Artificial intelligence in music has long struggled with the dichotomy between “understanding” and “generation”. Moreover, existing multimodal systems frequently fail to balance instruction efficiency with deep feature fusion, resulting in outputs that misalign with complex human intent. To bridge this gap, we propose H-M2UG, a unified hierarchical architecture built upon LLaMA-7B that synergizes perception and creation. Within an encode-inject-decode pipeline, the encode stage extracts music, image, and video semantics through dedicated pretrained encoders and modality-specific adapters; the inject stage first prunes visually irrelevant prompt tokens using a CLIP-assisted static pruning module and then progressively injects video, image, and music cues into middle-to-deep LLM layers via gated cross-attention; the decode stage uses a unified prediction head to generate either text tokens for understanding or acoustic code indices that are reconstructed into waveforms by a frozen neural audio decoder. By leveraging a task-adaptive unified prediction head alongside the decoder, H-M2UG flexibly performs both textual reasoning and waveform reconstruction within a shared semantic space. Experiments on MUEdit, MUCaps, and MusicQA, together with additional sensitivity, robustness, and cross-dataset analyses, demonstrate that our method achieves superior semantic consistency, audio quality, and multimodal reasoning. Furthermore, blind subjective listening tests validate improvements in harmony and expressiveness, establishing H-M2UG as a robust foundation for future general-purpose multimodal music agents.