Adapter-Based Contextualized Meta Embeddings

This paper introduces MetaLoRA and MetaUniPELT, two meta-embedding approaches that extends Low Rank Adaptation (LoRA) and adapters for fine-tuning and combining multiple pretrained models. We find that both models improve performance across a range of monolingual and multilingual tasks, outperforming baselines such as fully fine-tuned single models, simple concatenation of pretrained embeddings with classification layer fine-tuning and soft-voting ensembles. On the XGLUE benchmark, we find a 1.7 test score increase over the best fully-fine tuned model and a 0.24 increase over the best fully-fine tuned ensemble on sentence classification tasks. Our results underscore the potential of parameter-efficient fine-tuning of ensembles as efficient and effective alternatives to full fine-tuning and standard ensemble methods.