Weakly supervised phrase grounding (WSG) aims to localize visual regions corresponding to text phrases using only image-level annotations, which is particularly challenging in open-world settings due to the lack of fine-grained supervision. In this work, we propose a novel Context-Aware Prompt Network (CAPNet) that enhances WSG by explicitly modeling image-conditioned prompts and performing dense pixel-text alignment. Specifically, we design a visual-guided prompt tuning strategy to adapt the CLIP text encoder, enabling it to capture richer contextual semantics from visual inputs. In parallel, we reformulate the grounding task as pixel-level matching between visual features and contextualized text embeddings, generating a pixel-text score map that guides dense localization. Extensive experiments on Flickr30K, ReferIt, and Visual Genome demonstrate that our approach significantly outperforms prior state-of-the-art methods in both weakly supervised and open-world grounding tasks, validating the effectiveness of context-aware prompting for fine-grained cross-modal understanding.