Brain inpainting presents unique challenges compared to natural image inpainting, including increased problem complexity arising from the volumetric nature of 3D MRI and the need to ensure anatomical plausibility in addition to visual realism. To address these challenges, we propose a Pseudo-Segmentation-Guided GAN (PSegGAN) for tumor-to-healthy brain inpainting. Our framework consists of three cooperating networks: (i) a generator that reconstructs masked brain volumes conditioned on both healthy and tumor masks, (ii) a discriminator that enforces perceptual realism by distinguishing reconstructed from inpainted regions, and (iii) a segmentator that provides structural guidance through a perceptual masked loss derived from whole-brain segmentation. We evaluate our method on the BraTS 2025 Inpainting Challenge dataset. While our results show that the proposed model does not surpass the state-of-the-art in standard voxel-wise metrics such as MSE, PSNR and SSIM, qualitative assessment demonstrates that it produces more realistic and anatomically plausible reconstructions. Our findings suggest that incorporating structural priors can enhance anatomical plausibility in medical image synthesis and may offer additional insights for supporting medical image analysis. We share our code in https://github.com/juhha/BraTS-inpainting-2025-PSegGAN .