Research on computer-aided polyp segmentation in gastrointestinal endoscopy has spanned the past few decades. Despite notable progress, the challenge of achieving automatic accurate polyp segmentation remains unresolved. The majority of polyp segmentation methods rely on datasets with pixel-wise annotations. Creating these annotated datasets is laborious and time-intensive, particularly for physicians who need to prioritize their patients’ care. In opposition, it is cheaper and easier to obtain polyp bounding box annotations. Recently, large generative models have emerged and their strong generalizability makes them new baselines for many downstream tasks, even in depth estimation. In this paper, we propose a novel weakly supervised polyp segmentation framework, namely BBD-Polyp, which utilizes polyp bounding box annotations and polyp depth maps generated by a Diffusion-based depth estimator that provides zero-shot transfer. There are many background noises in coarse bounding box labels. Therefore, depth information from polyp depth maps is leveraged to suppress these noises. Moreover, we redesign the loss of learning segmentation masks in two terms. First, a fill-ratio loss to minimize the difference between predicted mask and coarse annotation. Second, an affine loss to preserve the consistency of predicted result within noised label under affine transformation. The depth map and the input polyp image are combined and fed to the segmentation model, in which a new Gaussian Enhanced Euclidean attention mechanism is incorporated to further exploit depth prior and eliminate background noises in bounding box annotations. Experiment results on three public datasets show that our proposed method achieves impressive results compared to fully supervised models with box labels and depth maps.