In recent years, large language models (LLMs) have garnered considerable attention across a multitude of fields, and fine-tuning is the process of further training them for specific tasks or domains. However, in medical applications, this approach presents unique challenges. The diversity of tasks often leads to imbalanced datasets, and the substantial number of parameters in LLMs requires considerable computational resources and time for fine-tuning. To address these issues, we propose a novel parameter-efficient fine-tuning framework, termed RDLoRA. This framework improves the efficiency of fine-tuning and effectively handles data imbalances, offering a practical solution for fine-tuning LLMs in medical scenarios.