Reconstructing high quality 3D Magnetic Resonance Imaging (MRI) fetal brain volumes from motion-corrupted 2D slices is critical for accurate prenatal diagnosis. However, existing methods face challenges such as incomplete brain extraction regions and severe motion artifacts. To this end, we propose D-SVG (Diffusion-Based Slice-to-Volume Generation), a novel generation framework that integrates diffusion models with Implicit Neural Representation (INR) for reliable fetal brain volume generation. Our approach consists of two processes: (1) diffusion process, where a pre-trained diffusion model generates consistent 3D volumes with prior anatomical knowledge, and (2) slice constrained fidelity process, where an INR-based reconstruction scheme iteratively refines the generated volume using the scanned MRI slices in three stages. We conduct different experiments on clinical and simulated fetal brain datasets, including quantitative experiments, ablation study, visual analysis, demonstrating its superior performance over existing methods. We also perform two downstream tasks to prove the effectiveness and quality of generated brain volumes. The code of this paper is available at https://github.com/SCUT-Xinlab/dsvg .