In this study, we explore a novel method to enhance the performance of LLM models when applied to multilingual understanding. Our approach involves dynamic rank adjustment in Low-Rank Adaptation (D-LoRA) during training, along with quantization prior integrating with the TensorRT-LLM framework and for this study we utilize a 2B Gemma model, a series of lightweight and state-of-the-art open models. The novel adaptation of D-LoRA dynamically adjusts the rank of matrices, depending on how complicated the language input is, specifically considering the complexities of code-mixing and the advanced sentence structures in Hinglish. This technique enables a more customized fine-tuning process, allowing the model to be well-suited for picking up the nuances of mixed-language environments and we utilize the Nemo framework to carry out the fine-tuning. Then we decrease the precision of the model by quantization, to achieve a balance between speed and accuracy. Finally, we take this optimized model and pass this through TensorRT-LLM framework, where we find a positive significant difference in both inference speed in terms of tokens per second and resource consumption compared to standard approaches. Our findings suggests that this dynamic strategy for LoRA has the potential to become a new benchmark for fine-tuning large language models for multilingual or codemixed settings, which is a great trade-off between computational efficiency and performance, particularly for applications involving multilingualism.