This paper presents an advanced Vietnamese voice conversion system, called ViSWAP, that utilizes a diffusion model to achieve highly natural and intelligible speech synthesis. By incorporating cutting-edge techniques such as HiFi-GAN, Real-Time Voice Cloning, and speaker diarization, ViSWAP effectively converts voices in both single and multi-speaker contexts with precision and speed. The system processes audio through a structured pipeline, from pre-processing with mel-spectrogram generation and TextGrid alignment in Vietnamese, to encoding and decoding within the diffusion framework. The adoption of the diffusion model is crucial, as it excels in maintaining high-quality voice conversion by handling complex transformations with superior fidelity. Experimental evaluations across multiple audio frequencies demonstrate the system’s strength in minimizing key metrics such as DTW, Euclidean, and Cosine distances, MSE showcasing significant improvements in timbre accuracy and harmonic preservation. We have also published the dataset and implementation on Github ( https://github.com/Nguyen-Van-Nguyen-github/DiffusionVoiceVietNam ).