Automatically detecting brain tumours can improve patient management and reduce clinicians’ diagnostic burden. Current supervised methods rely on scarce, costly annotations, limiting their scalability and transferability to other diseases. Weakly-supervised denoising diffusion probabilistic models (DDPMs) address this by modelling healthy data distributions to generate counterfactuals of diseased regions, enabling anomaly detection. However, challenges persist including poor anatomical encoding and high computational demands, restricting 3D magnetic resonance imaging (MRI) analysis to 2D models. We overcome these by introducing a novel 3D latent diffusion model (LDM) with a refined patch-based sampler for healthy tissue extraction. By leveraging exact diffusion inversion via coupled transformations (EDICT) encoding, we enhance anatomical preservation and facilitate significant image alterations with minimal encoding steps. Our 3D-LDM significantly outperforms state-of-the-art 2D weakly-supervised DDPMs in segmentation accuracy and false-positive reduction, advancing clinical implementation.