Mixture-of-Adapters with Routed Distillation: Unsupervised Expert Routing for Efficient Multi-task LoRA

Low-Rank Adaptation (LoRA) is widely used for fine-tuning large language models (LLMs) in a parameter-efficient manner. However, it struggles when applied to multiple tasks simultaneously due to the well-known issue of catastrophic forgetting. To address this, several Mixture of LoRAs approaches have been proposed. However, these methods still face challenges, including task interference, limited dynamic control, and scalability issues. To overcome these limitations, we introduce a novel Mixture-of-Adapters with Routed Distillation (MoA-RD) based unsupervised routing strategy that effectively selects the most relevant experts from a pool of LoRA adapters. Our approach addresses two key challenges: selecting the appropriate expert for a given input and minimising resource usage during inference. We design an unsupervised router based on semantic sentence embeddings and Liquid Neural Networks (LNNs), and we further compress the expert pool by merging similar experts through expert distillation. We evaluate our method on six public datasets covering a diverse set of tasks, including instruction following, open-domain question answering, fact questions, math problems, and headline writing. Experimental results demonstrate that our method selects experts more accurately, maintains or improves output quality, and reduces both memory and compute costs. Furthermore, the router distributes tasks more evenly across experts, enhancing efficiency and robustness.