Although vision foundation models like SAM2 excel at natural image segmentation, their direct application to medical image segmentation faces severe domain gap challenges due to unique imaging principles, complex noise characteristics, and specialized semantic requirements. We propose AMoE-SAM2UNet, a cross-domain medical image segmentation framework that integrates an Adaptive Mixture of Experts Decoder (AMoE-Decoder) with the SAM2 encoder. The AMoE-Decoder comprises three specialized expert modules: the Multi-Scale Adaptive Expert (MSAE) captures lesion scale variations through parallel dilated convolutions and adaptive fusion operations; the Boundary Refinement Expert (BRE) sharpens blurred boundaries by enhancing high-frequency information via wavelet transform; and the Foreground Enhancement Expert (FEE) leverages multi-branch convolutions and spatial-channel attention to enhance foreground-background discrimination, addressing low contrast and complex background. Moreover, a lightweight gating network dynamically combines expert outputs based on input characteristics. Extensive experiments on five diverse medical imaging modalities—dermoscopy (ISIC 2017), ultrasound (TN3K), endoscopy (ClinicDB), CT (COVID), and histology (GLAS)—demonstrate that AMoE-SAM2UNet achieves consistent Dice coefficient improvements ranging from 1.8% to 3.2% over the SAM2-UNet baseline and outperforms existing state-of-the-art methods by margins of 2.5% to 3.4%, validating the effectiveness of adaptive mixture-of-experts for medical vision foundation model adaptation.