Resource-Efficient Vietnamese Text Summarization: Enhancing Vistral 7B Performance Through Data Filtering, QDoRA’s Low-Memory Footprint, and DeepSpeed’s Training Optimization

This paper introduces a solution based on combining fine-tuning the large language model Vistral 7B [8] with the QDoRA [15] technique to improve the quality of Vietnamese text summarization. We conduct a thorough data filtering strategy to refine the training dataset, ensuring that the resulting summaries are of the highest quality. To address memory capacity limitations, we apply QDoRA, an advanced quantization technique capable of significantly reducing the model’s memory size without compromising performance. Additionally, we utilize advanced model training optimization techniques from DeepSpeed [1], enabling efficient and scalable training of large language models (LLMs). The Vistral 7B [8] model developed by C. V. Nguyen et al. (2024) is based on the original Mistral 7B [13] model but has been further trained on carefully selected and curated Vietnamese datasets, ensuring the data’s relevance and diversity. We compared the Vistral 7B model combined with QDoRA to the ViT5 [21] model during the research. ViT5 is a model developed in the field of Vietnamese text summarization. Experimental results show that Vistral 7B combined with QDoRA outperforms ViT5, primarily when evaluated using ROUGE metrics. Specifically, on the preprocessed dataset, our method achieved a ROUGE-1 score of 76.42, a ROUGE-2 score of 57.88, and a ROUGE-L score of 59.00, compared to ViT5 \(_{base}\) ’s scores of 71.85, 51.95, and 53.40 respectively.