The advancement of large language models (LLMs) has revolutionized artificial intelligence applications across multiple fields, including law. In Vietnam, legal documents are characterized by complex structures, specialized terminology, and intricate content relationships, creating significant challenges for efficient information processing. This study presents the Graph RAG system, an innovative knowledge graph based approach designed to improve the retrieval and synthesis of information from Vietnamese legal texts. By ensuring high accuracy and thorough analysis in legal decision making, this work pushes forward the use of natural language processing (NLP) in the legal domain. The Graph RAG system was built using six Vietnamese legal codes as its core dataset. We utilized the Neo4j graph database to organize and visualize legal information, effectively capturing entities and their relationships. A key feature of our approach was the development of custom prompts tailored to legal contexts, enabling precise identification of concepts and their connections. This was refined through iterative visualization and validation in Neo4j to ensure the knowledge graph’s accuracy and usefulness. We evaluated Graph RAG by comparing it to traditional Retrieval Augmented Generation (RAG) methods using two datasets: one with multiple choice questions and another with open-ended queries. Performance was measured through metrics such as Normalized Discounted Cumulative Gain (NDCG) for retrieval relevance, Answer Relevancy Score (ARS) for response quality, and accuracy for multiple-choice answers. The results show that Graph RAG outperforms conventional RAG, with a 14.05% increase in retrieval relevance (NDCG), a 4.34% boost in accuracy, and enhanced ability to handle complex legal information. This research contributes to the exploration and future implementation of RAG systems in addressing administrative and legal tasks in Vietnam.