Breast-conserving surgery (BCS) is the preferred treatment for early-stage breast cancer, offering survival rates comparable to mastectomy while preserving breast aesthetics. Accurate tumor segmentation is essential for surgical planning, yet segmentation models often exhibit biases toward specific tumor sizes, particularly underperforming on smaller tumors. To address this, we propose a novel approach that uses generative models to improve segmentation across tumor sizes. Specifically, we adapt the Stable Diffusion model and apply a Denoising Diffusion Probabilistic Model (DDPM) inversion approach to generate synthetic tumors of controlled sizes within real breast MRIs, helping to balance tumor size distribution in the training data. By augmenting the dataset with 10–20% synthetic tumor images, our method significantly improves segmentation accuracy for small tumors without compromising performance for larger tumors. This enhancement allows for more precise tumor assessment, leading to better-informed surgical decisions and potentially reducing unnecessary mastectomies.