Large language models (LLMs) with billions of parameters achieve impressive results on domain-specific tasks but are often too resource-intensive for practical deployment. Knowledge distillation (KD) is an effective technique for compressing large models. It transfers knowledge from a teacher model to a smaller student model while maintaining performance. In this work, we focus on distilling SeaLLMs, which are large language models specifically designed for Southeast Asian languages (including Vietnamese). We transfer knowledge from a 7-billion-parameter teacher model to a smaller 1.5-billion-parameter student model. Our experiments on a Vietnamese domain-specific knowledge base show that KD from the fine-tuned QLoRA 7B teacher achieves a BERTScore-F1 of 0.7217, outperforming distillation from the base 7B teacher (0.7065) and fine-tuned student models without distillation. Moreover, the distillation process reduces the model size by approximately 78%, enabling more efficient deployment of domain-specialized LLMs. These findings demonstrate that teacher model quality critically impacts KD effectiveness and provide practical guidance for building compact, high-performance LLMs for specialized Vietnamese knowledge domains.