The change detection task of remote sensing images provides an effective means and technology to detect changes on the Earth’s surface, providing data support for disaster management. Although current methods mostly adopt hierarchical structures and variations of transformer-base models, they overlook the rich detailed features provided by shallow layers during the restoration process, as well as the accurate global features of deep layers, leading to the loss of edge details in the final change detection structure. As a solution to this problem, we suggest SFFAFormer, which employs a module design with enhanced channel learning in shallow layers to enhance edge details and feature transfer, and utilizes transformer-base modules with semantic accumulation computation in deep layers to ensure the accuracy of global information. Experimental results demonstrate that SFFAFormer surpasses many leading baselines and achieves outstanding performance on the LEVIR-CD and DSIFN-CD datasets.