Segment Anything Model (SAM) has attracted considerable attention for its strong zero-shot segmentation capability, but its deployment in real-world scenarios is hindered by slow inference and high memory consumption, largely due to the attention mechanism. Existing works mainly focus on compressing the encoder, while leaving the inefficiency of attention computation insufficiently addressed. We propose SAM-Lightening, a lightweight variant of SAM that integrates a re-engineered attention operator, Dilated FlashAttention, to improve parallelism and efficiency while remaining compatible with FlashAttention. In addition, we design a progressive distillation strategy to transfer knowledge from the original SAM without expensive retraining. Experiments on COCO and LVIS demonstrate that SAM-Lightening achieves state-of-the-art efficiency and accuracy. It processes \(1024 \times 1024\) images in 7 ms— \(30.1\times \) faster than vanilla SAM and \(2.1\times \) faster than the previous best—while requiring only 244 MB memory (3.5% of vanilla SAM). These results highlight its potential for practical deployment on resource-constrained platforms.