Supervised machine learning has enabled accurate pathology detection in brain MRI, but requires training data from diseased subjects that may not be readily available in some scenarios, for example, in the case of rare diseases. Reconstruction-based unsupervised anomaly detection, in particular using diffusion models, has gained popularity in the medical field as it allows for training on healthy scans alone, eliminating the need for large disease-specific cohorts. These methods assume that a model trained on normal data cannot accurately represent or reconstruct anomalies. However, this assumption often fails with models failing to reconstruct healthy tissue or accurately remove anomalies. In this work, we introduce CondDiff, a novel conditional diffusion model framework for anomaly detection and healthy image reconstruction in brain MRI. Our self-supervised approach integrates synthetically generated pseudo-pathology volumes into the modeling process to better guide the reconstruction of healthy imaging volumes. To generate these pseudo-pathologies, we apply fluid-driven anomaly randomization to augment real pathology segmentation maps from an auxiliary dataset, ensuring that the synthetic anomalies are both realistic and anatomically coherent. We evaluate CondDiff’s ability to detect pathology, using both synthetic and real pathology (ATLAS and BraTS) datasets. In our extensive experiments, CondDiff: (i) consistently outperforms variational autoencoders (VAEs), and conditional and unconditional latent diffusion; and (ii) surpasses on most datasets, the performance of supervised inpainting methods with access to paired diseased/healthy scans (Code is available at https://github.com/alawryaguila/CondDiff ).