Owing to its superior soft tissue contrast, Magnetic Resonance Imaging (MRI) has become a cornerstone modality in clinical practice. This prominence has driven extensive research on MRI-based segmentation, supported by the proliferation of publicly available benchmark datasets. Despite employing multi-expert consensus protocols to ensure annotation quality in public datasets, the inherent label noise, particularly prevalent at lesion boundary regions, remains unavoidable. To address this fundamental challenge, we introduce a novel machine learning paradigm that reframes dataset annotations as probabilistic weak supervision rather than deterministic gold standards. We proposed AffinityUMamba, a novel dual-branch Unet-like framework that synergistically integrates convolutional operations with state space models, leveraging local feature coherence and global contextual agreement. And a Local Affinity-guided Label Refinement (LALR) module to identify potential noisy labels in the training data and produce refined pseudo labels. A unified uncertainty constraint paradigm combining margin-based logit smoothing with local affinity refinement, enabling simultaneous optimization of segmentation accuracy and confidence calibration. Training is stabilized through a composite objective combining topological preservation constraints with margin-aware uncertainty penalization, enabling joint optimization of structural coherence and detail fidelity. We comprehensively evaluated the proposed method on 12 public datasets spanning multiple modalities: 10 MRI, 1 Ultrasound, and 1 CT. The results of our experiments demonstrate improved segmentation performance and reduced prediction uncertainty.