LoRA (Low-Rank Adaptation) has emerged as an efficient fine-tuning technique for large language models, allowing for significant model customization with minimal computational overhead. However, the original LoRA formulation has room for improvement in terms of efficiency and performance. In this study, we present Learnable Group Low-Rank Adaptation (LG-LoRA), an innovative technique that introduces a learnable grouping mechanism to dynamically cluster model parameters based on task relevance. LG-LoRA enables fine-grained control over adaptation by assigning parameters to groups and applying separate low-rank updates to each group. We propose a differentiable group assignment process that integrates into end-to-end optimization, allowing the network to learn optimal parameter grouping. Our experiments on LLaMA and CLIP models demonstrate LG-LoRA's effectiveness in enhancing model performance and efficiency across various tasks and modalities. Notably, LG-LoRA outperformed LoRA and DoRA on LLaMA 7B models, achieving 77.7% accuracy across tasks. For vision-language models (VLMs), LG-LoRA achieved 76.5% accuracy, showcasing its promise for versatile and efficient model adaptation in various applications.