Image captioning (IC) aims to generate natural language descriptions of an image's visual content. Recent retrieval-augmented methods allow IC models to adapt to new domains using external databases, exploiting large-scale data without additional fine-tuning. However, treating all retrieved results equally introduces noise from less relevant captions, degrading output quality. To address this issue, we introduce Hierarchical Retrieval-Augmented Captioner (HRACap), a lightweight IC framework that selectively highlights relevant information and reduces the influence of irrelevant data. Specifically, we introduce the Hierarchical Retrieval-Augmented Prompt (HRA prompt), which structures retrieved captions hierarchically according to their semantic similarity to the input. We also enhance the decoder by embedding visual inputs as pseudo-words within the textual embedding space, enabling the model to concentrate more effectively on key visual elements. This combination of pseudo-words with carefully filtered captions constitutes the HRA prompt that efficiently guides the decoder's attention towards the image and the most pertinent text. Extensive experiments on COCO, Flickr30K, and NoCaps show that HRACap achieves outstanding performance with significantly fewer trainable parameters than existing methods, highlighting its superior efficiency and generalization capabilities.