Fine-grained image retrieval remains a challenging task due to subtle inter-class distinctions and large intra-class variations. Existing fine-grained image hashing methods often employ attention or region-based localization techniques to enhance feature discrimination, which usually lacks explicit semantic guidance such as fine-grained annotations or textual descriptions. This absence of fine-grained labelled or linguistic supervision limits their ability to retain class-specific visual patterns accurately. To address this, we propose a novel \({\textbf {L}}\) anguage-guided \({\textbf {P}}\) atch \({\textbf {A}}\) ggregation \({\textbf {H}}\) ashing ( \({\textbf {LPAH}}\) ) framework, which, for the first time, utilizes automatically synthesized textual semantics to guide token aggregation in ViT, preserving discriminative patterns for improved hashing performance. Specifically, our framework introduces three plug-and-play modules: Cross-Layer Aggregation (CLA), Cross-Patch Aggregation (CPA), and Patch-Word Alignment (PWA). The CLA module fuses features from different ViT layers to enrich the representation with multi-level information. The CPA module then refines these fused features, aggregating them into a compact token set. Finally, the PWA module aligns the refined visual features with textual embeddings at the word level, establishing precise semantic correspondences between visual information and linguistic cues. Such aggregation and alignment procedures improve the capture of subtle yet critical visual distinctions in the fine-grained images. Extensive experiments on fine-grained retrieval benchmarks demonstrate that LPAH achieves state-of-the-art performance, significantly outperforming fine-grained and generic hashing methods. The source code is available at https://github.com/zhenglab/LPAH .