Government budget auditing is critical for ensuring the legality and rationality of resource allocation, but its hierarchical structure and extensive numerical data present significant challenges for question-answering(QA) systems. While large language models(LLMs) excel in general tasks, they struggle with in budget auditing QA task. Retrieval-Augmented Generation(RAG) offers a promising approach by integrating external knowledge, yet traditional RAG methods falter in handling complex hierarchical structure and aggregating semantic relationships from extensive audit materials. To address these limitations, we propose BAG-RAG, a novel graph-based RAG framework tailored for government budget auditing. Our approach introduces a Multi-layered Semantic Graph Construction method to capture hierarchical structures and establish clear semantic relationships by linking structural nodes and instance nodes. Additionally, we design a Bidirectional Retrieval Strategy that combines top-down retrieval of structural nodes for global context and bottom-up aggregation of instance nodes to ensure comprehensive and accurate answers. We validate our framework using a newly curated QA dataset specific to government budget auditing. Experimental results show that BAG-RAG effectively addresses hierarchical complexity and incomplete retrieval, delivering precise and comprehensive answers for budget auditing questions. (This research was funded by Southeast University-China Mobile Research Institute Joint Innovation Center)