Object detection in remote sensing images has advanced significantly, nevertheless, it still faces the dilemma of low performance in detecting small objects. Due to extremely limited area, small objects can easily be submerged in complex backgrounds and suffer from insufficient context semantic information. To alleviate the aforementioned issues, we design a lightweight plug-and-play module named Cascade Local-Global Context Module (CLGCM) to extract context information. The module contains one cascade operation and two novel Sparse Context Blocks. The cascade operation can obtain long-range context semantic information at a small computational cost. And Sparse Context Block focuses on the most relevant semantic information through a sparsification operation. When integrated with Faster-RCNN and Cascade R-CNN, our module further boosts detection performance on the small object detection benchmark, AI-TOD, which significantly outperforms other mainstream algorithms.