The rise of language models has achieved state-of-the-art performances across a wide range of downstream tasks. However, traditional full-parameter fine-tuning is computationally expensive and impractical for scaling to increasingly larger models. To mitigate these challenges, parameter-efficient fine-tuning (PEFT) methods have been introduced, with Low-Rank Adaptation (LoRA) emerging as a prominent approach. LoRA significantly reduces fine-tuning costs by modifying only a small set of trainable low-rank weight matrices, while keeping the majority of the pretrained model weights frozen. This survey provides a mathematically rigorous exploration of LoRA-based variations, focusing on their algorithmic structures and working mechanisms. By conducting a systematic analysis, we offer deeper insights into the mathematical characteristics and adaptation dynamics of LoRA-based methods. This survey aims to equip the PEFT research community with a solid mathematical framework for designing more effective, scalable, and computationally efficient fine-tuning algorithms, further advancing the adaptability of language models in resource-constrained settings.