Vision-language models such as CLIP excel at multimodal tasks but struggle with fine-grained semantic distinctions, particularly when inputs share similar lexical or visual features. This dilemma often results in inaccurate image-text alignment and weak compositional reasoning. General coping methods following a two-phase contrastive learning paradigm (negatives construction & contrastive fine-tuning), where most focus on generating negative text samples, but negative image samples are often randomly selected and lack semantic relevance, offering limited training value. Consequently, critical hard negative pairings are underutilized, leading to imbalanced learning. To address this, we propose PB-CLIP, which augments CLIP with two modules: (1) PixelNeg, generating pixel-level hard negatives by perturbing local regions while preserving global semantics; (2) BalCL, a balanced multi-level contrastive module incorporating global-local hard image negatives and adversarial texts. PB-CLIP achieves new state-of-the-art results on Winoground and Visual Genome benchmarks.