Scholarlink

<p>Image captioning is an outstanding vision and language task that generates detailed descriptions for given images. Real-world applications require the balance between efficiency and effectiveness. <i>Non-Autoregressive</i> methods for <i>Image Captioning</i> (NAIC) have been proposed to address the high latency problem during inference. However, achieving a good trade-off between fast inference and high quality remains a challenge. To this aim, we propose a novel NAIC approach, i.e., <i>UniCap</i>. Our UniCap employs two types of decoders: a mask predictor for word prediction and a unified editor for automated editing operations. Furthermore, we propose an encoding method that encodes the editing operations. Meanwhile, we design a collision resolution algorithm to resolve the invalid outputs. To train UniCap, we improve the conventional two-stage training strategy. In cross-entropy stage, we propose a triple-path method to obtain editing-oriented training data by incorporating noise. In reinforcement learning stage, we propose a threshold sampling method by preferring non-confident words. Extensive experimental results on MS-COCO benchmark demonstrate that our UniCap achieves an excellent trade-off between latency and quality compared to previous methods. Our code is publicly available at <a href="https://github.com/uuam5/UniCap">https://github.com/uuam5/UniCap</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Towards balancing the efficiency and effectiveness: a unified edit-based framework for automatic image captioning

  • 期刊论文

  • 发表日期：

    2026年2月3日

查看全文
  • Ruifan Li,
  • Siwei Xu,
  • Pengyue Lin,
  • Fangxiang Feng,
  • Zhangyu Ma

摘要

Image captioning is an outstanding vision and language task that generates detailed descriptions for given images. Real-world applications require the balance between efficiency and effectiveness. Non-Autoregressive methods for Image Captioning (NAIC) have been proposed to address the high latency problem during inference. However, achieving a good trade-off between fast inference and high quality remains a challenge. To this aim, we propose a novel NAIC approach, i.e., UniCap. Our UniCap employs two types of decoders: a mask predictor for word prediction and a unified editor for automated editing operations. Furthermore, we propose an encoding method that encodes the editing operations. Meanwhile, we design a collision resolution algorithm to resolve the invalid outputs. To train UniCap, we improve the conventional two-stage training strategy. In cross-entropy stage, we propose a triple-path method to obtain editing-oriented training data by incorporating noise. In reinforcement learning stage, we propose a threshold sampling method by preferring non-confident words. Extensive experimental results on MS-COCO benchmark demonstrate that our UniCap achieves an excellent trade-off between latency and quality compared to previous methods. Our code is publicly available at https://github.com/uuam5/UniCap.