The application of SAM as a powerful pre-trained model encounters significant challenges in change detection task. First, SAM is training on large-scale natural images limits its direct use in remote sensing field. Second, as a single-image segmentation method, SAM itself cannot learn the semantic change in bi-temporal phases. Finally, the diverse surface changes and the pseudo-change problem pose challenges to SAM. To address the above issues, we introduce TS-SAM, aims to effectively guide the SAM to learn changes at the semantic level through a two-stage approach. More specifically, we develop a Spatial-Frequency Adaptation (SF-Adapter) module integrated with the SAM encoder, designed to bridge domain gaps by transferring spatial-frequency features from natural images to geospatial domains while preserving original segmentation capabilities. Next, we propose a Change Enhancement Module (CEM) to enable SAM to learn semantic changes in complex remote sensing environments. Finally, to address the pseudo-change issue, we analyze the training process of pseudo-change regions under SAM and propose a two-stage approach with a Change Refinement Module (CRM) to correct pseudo-change issues and improve prediction accuracy. Our experiments on three public datasets show that our model outperforms state-of-the-art methods by 0.3% to 3.2% in F1 score, demonstrating its effectiveness.