Insect noise (e.g., cicadas) is the primary noise interference in outdoor audio collection during summer and autumn. Current speech enhancement models suffer from ineffective multi-scale feature utilization and redundant information aggregation, hindering speech recovery. In order to more efficiently solve the interference of insect noise, this paper introduces a Multi-scale Adaptive Feature Sparse Network (MAFSNet). The Multi-scale Adaptive Sparse Conformer (MASC), serving as the cornerstone design of MAFSNet, comprises three pivotal constituent modules. Specifically, Multi-scale Adaptive Sparse Attention (MASA) module differentially processes multi-scale features at different levels, and then uses a dual-branch self-attention for adaptive redundancy reduction. Channel-guided Branch Modulation (CBM) module combines channel features with gating mechanisms to suppress irrelevant features of different branches. Meanwhile, Dual Scale Enhancement Fusion (DSEF) module implements learnable weighting for optimized feature fusion. The quantitative evaluations of the Insect Noise dataset and the Voice Bank+DEMAND dataset demonstrate that MAFSNet outperformed other models and achieved excellent results in objective evaluation metrics.