The Segment Anything Model (SAM) has achieved outstanding performance in both natural and medical image segmentation with extensive research validation. When applied to ultrasound images, which involve low contrast, indistinct boundaries and complex shapes, large models still suffer from significant performance degradation and limited generalization ability. We explore these challenges from a new perspective with the help of the segmentation foundation model SAM. In this paper, we propose Nora, a noise-robust fine-tuning framework for SAM to address domain generalized ultrasound image segmentation. Specifically, we introduce a feature-adaptive perturbation module, which applies well-designed noise to the fine-tuned features. We stimulate the model to segment the correct regions even under severe interference, thereby improving its robustness. Moreover, to further optimize SAM with prompts, we present an instance-aware prompt generation module. We introduce a set of tokens linked to distinct instances and then design a token-based augmentation strategy to prevent overcoupling and encourage tokens to capture more diverse information. Our Nora achieves state-of-the-art performance across extensive cross-domain experiments with three ultrasound image segmentation tasks, fully demonstrating its effectiveness and generalizability. The code is available at  https://github.com/wkklavis/Nora .