An important task in breast cancer detection and prognostication entails the precise delineation of the primary tumor, enabling insight in tumor size, shape characteristics and cancer progression in follow-up studies. However, manual segmentation is labor-intensive and subject to inter-observer variability, motivating the need for accurate automated methods. In light of the ongoing MAMA-MIA challenge, we propose a task-specific and data-centric adaptation of the nnU-Net framework that takes inspiration from the radiologist’s workflow for this task. We demonstrate that subtle architectural changes, such as increasing patch size and using multi-channel input, significantly improve performance over a vanilla nnU-Net. Quantitatively, our domain-inspired nnU-Net outperforms a baseline nnU-Net by 3% points in an internal validation setting and 5% points in an external validation setting with additional post-processing methods. Furthermore, we demonstrate that incorporating additional annotated and unannotated data in a semi-supervised manner boosts generalization to unseen domains. In conclusion, our work demonstrates that, despite being a strong general-purpose baseline, a vanilla nnU-Net may still yield a significant performance increase by incorporating domain-specific knowledge. More broadly, this finding corroborates previous research highlighting the need for rigorous validation in medical image segmentation, since common baseline methods may benefit greatly from task-specific adaptation.