In this paper, we present LRACLIP, a novel low-rank approximation framework built on CLIP that enhances visual-language consistency to improve language-guided age estimation from facial images. Existing approaches typically assume that pre-trained features are well-learned and directly utilize them for downstream tasks, which may introduce noise and degrade performance. Our method, grounded in canonical polyadic decomposition (CPD), employs low-rank filtering and residual protection to improve feature discrimination in the CLIP latent space. Specifically, we embed category labels into text prompts and leverage context optimization to reinforce semantic consistency. Then, the backbone image encoder and the CLIP-powered text encoder produce a shared feature space that contains redundancy and modality bias. Finally, the CPD module adaptively selects informative low-rank components from visual features to improve both semantic fidelity and ordinal consistency in visual-language alignment. Experimental results demonstrate that our approach effectively suppresses irrelevant visual information and significantly improves classification performance.