Modality-experts coordinated adaptation for large multimodal models

Driven by the expansion of foundation models and the increasing variety of downstream tasks, parameter-efficient fine-tuning (PEFT) methods have exhibited remarkable efficacy in the unimodal domain, effectively mitigating the consumption of computational resources. Although recent research has shifted attention to the multimodal domain and achieved efficient parametric adaptation of large multimodal models (LMMs) for downstream tasks, they still encounter two limitations: (1) low performance; (2) poor compatibility. This work proposes a modality-experts coordinated adaptation (ModeX) method for the multimodal domain, offering an effective, plug-and-play, and lightweight adaptation architecture for diverse LMMs. Specifically, ModeX adaptively coordinates different modality experts in terms of the types of network structure and input data. Besides, an effective coordinator equipped with a routing algorithm is developed for generating corresponding weights, which centers on leveraging the synergy among multimodal data. Extensive experiments on 15 multimodal downstream benchmarks and five LMMs demonstrate that ModeX is capable of seamlessly adapting to diverse LMMs, outperforms the state-of-the-art PEFT methods and even exhibits superior performance compared with full fine-tuning methods. Notably, on NLVR² task, ModeX achieves 84.06% accuracy with only 12.0M trainable parameters, outperforming the full fine-tuning by 1.63%. Moreover, our ModeX method demonstrates superior stability and offers higher training efficiency, both in terms of training parameters and training duration. Our source code has been released at https://github.com/zhangy0822/ModeX.