Vision-language models such as CLIP have demonstrated strong generalization in few-shot learning. Existing methods typically freeze CLIP’s visual encoder and rely on shallow classifiers and language models to generate class-level prompts for similarity matching. However, under extremely low-shot conditions, they face two main challenges: limited fine-grained discrimination and weak cross-modal interaction due to static feature matching. To address this, we propose a two-stage framework—Prompt-guided Contrastive Adaptation and Cross-modal Semantic Refinement (PCSR)—to enhance semantic alignment and generalization in few-shot vision-language tasks. In Stage I, we inject trainable dynamic prompts into CLIP’s visual and textual encoders. The model uses local image regions as input, guided by prompts to extract discriminative features, and learns via an image–text contrastive loss to adapt the joint embedding space. In Stage II, based on the frozen model adapted in Stage I, we introduce a cross-modal cross-attention mechanism. Local features are used to construct dynamic key-value pairs that drive collaborative modeling between global image representations and CLIP’s text classifier weights, thereby enhancing the model’s perception of fine-grained semantics. Experiments show that PCSR consistently outperforms existing methods on multiple few-shot benchmarks, demonstrating superior fine-grained recognition and cross-modal generalization.