Pansharpening, a key technique in remote sensing image fusion, aims to merge panchromatic (PAN) and multispectral (MS) images to generate high-resolution multispectral (HRMS) images. Despite the wide range of pansharpening methods available, achieving the perfect balance between spatial detail enhancement and spectral accuracy remains a challenge, the quality of the image and spatial resolution still have potential for further improvement. In this paper, we introduce ESNet, a novel multi-scale and multi-stage reconstruction fusion framework. Addressing the limitations of the HRNet model structure and the conflicting demands for spatial and spectral information, we have designed the enhanced spatial-spectral attention module (ESAM) to efficiently extract both spatial and spectral features, thereby highlighting the influence of useful information and suppress regions with fewer features in HRNet to obtain attention-enhanced features. Simultaneously, we incorporate a multi-stage reconstruction block to recover pansharpened images at each scale, which simplifies and stabilizes the training procedure while effectively capturing multi-level and multi-scale spatial-spectral features. A hybrid loss function is also designed to aggregate multiscale spectral feature information. To evaluate ESNet’s performance, we conducted experiments on the publicly available PAirMax dataset. These experiments demonstrated that ESNet achieved significant improvements on the PAirmax dataset, specifically a 10.22% increase in High-Quality Noise Ratio (HQNR) and a 9.37% boost in \(D_\lambda \) . Additionally, ESNet enhanced the performance by 2.96% in \({Q2}^{n}\) , thus demonstrating ESNet’s superior performance.