Machine learning methods have achieved remarkable results in medical image processing but require large annotated training datasets, typically unavailable in the medical image domain. To overcome this issue, we propose a conditional diffusion model that generates brain MRIs with tumors directly with the corresponding ground truth annotations. An additional diffusion-based label modification approach is integrated in order to account for the tumor mass effect. Experimental results demonstrate that replacing up to 50% of real training data with generated samples does not significantly impact segmentation performance. Moreover, models trained exclusively on synthetic data still yield acceptable results, highlighting the potential of our approach to mitigate the problem of limited annotated data in medical imaging.