Entity-aware multi-image captioning by common context text selection and joint entity prompting
摘要
The task of entity-aware multi-image captioning aims to generate coherent captions for multiple images within a news article. Since these images often contain rich entity information related to the article context, effectively mining and utilizing this entity information has been a research focus. Most prior work treats the images in the same article as independent and generates captions separately. However, we argue that images within the same article are likely to exhibit common relationships, as they may describe the same named entities. Motivated by this, we propose a two-step entity-aware multi-image captioning model to generate captions for pairs of images in a news article. Specifically, our model first employs a graph-structured network for conditional matching to select common sentences from the article and extract named entities associated with multiple images. Then, a joint entity generator is designed to select joint entities from these candidates. Finally, we use the images and the selected contextual text as inputs to the BART model, while incorporating the selected joint entities as prompts to generate captions for each image. Experiments on three public datasets demonstrate that the proposed model outperforms all baseline approaches in terms of BLEU, METEOR and entity recall.