GrowCap: Lightweight Image Captioning with Enhanced Hard Prompts

Image captioning is a core vision-and-language task aimed at generating textual descriptions for images. Beyond traditional applications, it also serves as a fundamental upstream task in multimodal knowledge graph (MMKG) systems, where captions provide semantic grounding for visual entities and relationships. These captions support the automatic construction of knowledge triples, enabling downstream reasoning, relation inference, and cross-modal alignment. Traditional approaches often scale model size and training data to improve performance, but at the cost of significant computation. Lightweight methods have emerged that freeze pretrained models to reduce trainable parameters. However, these models frequently overlook fine-grained visual details, relying heavily on high-level features and neglecting specific objects and their interactions. As a result, they may produce vague or incomplete descriptions in complex scenes. To address these challenges, we propose GrowCap, a lightweight captioning model based on enhanced hard prompts. GrowCap combines a frozen CLIP visual encoder and a GPT-2 language decoder with two key components: a visual feature enhancement module and a prompt refinement strategy. These additions help the model capture specific entities and their relationships more effectively. Experiments on MSCOCO and Flickr30k show that GrowCap achieves competitive performance while requiring far fewer trainable parameters compared to large-scale models, demonstrating its suitability for efficient and semantically rich caption generation.