Improving Abstractive Summarization with Unsupervised Dynamic LoRA Mixtures

The recent jump in hardware requirements for state-of-the-art language models have increased training and inference costs considerably. While methods such as Mixture-of-Experts (MoE) address the network throughput decline they retain high memory consumption. To alleviate the issue researches explored the possibility of replicating MoE results with layer adapters such as LoRA using various ensembling schemes. However, the utilization of the pre-trained LoRA adapters in the current approaches is limited as they are only considered in static ensembles while the dynamic counterparts are designed to be trained from scratch. We argue that dynamic LoRA ensembles can be obtained without additional fine-tuning by leveraging the network inference statistics. We propose Unsupervised Dynamic LoRA Mixtures (UDLM), a dynamic adapter ensembling method that translates pre-trained LoRA adapter sets to equivalent MoE networks by accumulating adapter spatial statistics. Evaluation on 6 abstractive summarization datasets in knowledge transfer setting demonstrates that UDLM substantially outperforms best static ensembles having a 30 \(\%\) summarization quality improvement while introducing only 35 \(\%\) increased latency.