Text-based geo-localization, which aims to identify a target location in an aerial image using a natural language description. Existing methods often rely on global feature matching and local spatial information for alignment. However, fine-grained semantic matching between specific textual phrases and their corresponding visual regions is not effectively utilized. To this end, we propose the Region-level Cross-modal Matching Framework (RCMF), which forges robust correspondences across multiple levels of detail. Furthermore, we propose a novel Dual-stream Cross-Attention Fusion module (DCAF) that facilitates deep, reciprocal information exchange between image and text modalities. Finally, extensive experiments demonstrate that our approach achieves state-of-the-art performance on the GeoText-1652 benchmark, improving localization accuracy.