Graph Neural Networks have shown promise in spatial interpolation tasks. However, most existing methods rely on simple, coordinate-based heuristics for graph construction, often leading to suboptimal performance on target variables that are influenced by complex factors. Graph Structure Learning (GSL) aims to address this by generating task-relevant graph structures. However, Latent Context Information (LCI) can bias spatial correlations and limit the generalizability of the GSL module. To tackle this, we propose a self-supervised GSL module that approximates and isolates LCI, enabling more robust and generalizable graph structures. Experiments on diverse real-world datasets demonstrate consistent performance and stability improvements over baselines, showcasing our model’s adaptability across various tasks.