Despite the significant success of deep learning-based scene text detection methods, detecting text under adverse weather conditions, particularly in foggy weather, remains a challenge due to poor visibility. In this paper, we propose a novel fog text detection network that integrates visual-language models and fog feature decoupling. Specifically, we leverage the popular CLIP model to embed textual information into the detection network, achieving cross-modal assist scene text detection. In addition, we introduce a fog feature decoupling module that learns to extract fog-related factors from the visual style. By alternately optimizing the fog feature decoupling module and the text detection model, the gap between normal images and foggy images is progressively reduced, allowing the model to learn fog-invariant features. Experimental results demonstrate that our method outperforms existing approaches on real-world foggy scene text detection datasets.