Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation
摘要
In this paper, we argue that the same rank setting of LoRA will inhibit its potential. In response, we proposed two novel PEFT strategies to improve the capability of LoRA in single-task and multi-task scenarios. We also conducted extensive experiments to show the effectiveness of our proposed method.