LoRA and its variants are among the most widely used parameter-efficient fine-tuning (PEFT) methods, as they greatly cut trainable parameters yet preserve strong performance. Determining the rank of LoRA remains an open problem. In this paper, we propose COMLoRA—a novel fine-tuning approach that integrates a chain-based training framework with the Mixture of Experts (MoE). Rooted in the residual learning paradigm and COLA, COMLoRA iteratively trains multiple sub-LoRA modules to progressively approximate optimal weights, thereby substantially enhancing the performance of LoRA. In our implementation, each sub-module is treated as an independent expert, and we propose an innovative shared A matrix strategy to capture domain-general knowledge. Additionally, we employ hierarchical learning rate scheduling and incorporate orthogonal regularization to mitigate catastrophic forgetting, in line with continual learning principles. Experimental results demonstrate that COMLoRA outperforms existing methods on GLUE benchmark evaluations, with all sub-modules seamlessly integrating into pre-trained models without increasing inference latency. The implementation is publicly available on GitHub at https://github.com/Ledger-cs/COMLoRA .