Phenotypic virtual screening holds great promise for discovering therapeutic compounds by directly linking molecular effects to disease-relevant phenotypic outcomes. However, existing computational methods face critical limitations in generalizing across diverse biological contexts and mitigating experimental noise inherent in high-throughput screens. We present PhenotypeCLIP, a robust framework for phenotype-aware drug retrieval that integrates multi-modal representation learning and noise-invariant alignment. The framework leverages pretrained molecular encoders enriched with chemical property and protein target interaction knowledge to model compound-phenotype mechanisms, while explicitly capturing the interplay between molecules and baseline cellular states through latent variable modeling. An information bottleneck further enforces noise-robust representations by prioritizing predictive signals over batch-specific artifacts. Evaluated on cell line-stratified LINCS datasets, PhenotypeCLIP achieves a 3.85-fold improvement in top-1 retrieval recall on unseen cell lines compared to state-of-the-art methods and demonstrates superior few-shot learning performance in mechanism-of-action prediction and drug response prediction. These advancements establish a new paradigm for generalizable and interpretable phenotypic drug discovery, bridging molecular mechanisms with systemic therapeutic effects.