<p>Recently, image captioning has attracted significant research interest due to its wide range of applications. The training process for captioning generative models typically employs backpropagation to maximize the likelihood of the next word given the previous ground truth word. Evaluation of this method is usually conducted using metrics such as CIDEr, ROUGE, and others. However, a disadvantage of this approach is that the words sampled by the network must closely resemble the ground truth. Since the word generation process is not continuous, it is challenging to optimize these metrics directly through backpropagation. In this study, we propose a novel approach to image caption decoding that enhances the robustness of existing models by incorporating prior knowledge from pre-trained cross-modal CLIP models. The results of comparative experiments demonstrate that the Monte Carlo Tree Search (MCTS)-based approach yields favorable outcomes in terms of evaluation metrics.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CLIP guided image caption decoding based on monte carlo tree search

  • GuangSheng Luo,
  • ZhiJun Fang,
  • JianLing Liu,
  • YiFanBai Bai

摘要

Recently, image captioning has attracted significant research interest due to its wide range of applications. The training process for captioning generative models typically employs backpropagation to maximize the likelihood of the next word given the previous ground truth word. Evaluation of this method is usually conducted using metrics such as CIDEr, ROUGE, and others. However, a disadvantage of this approach is that the words sampled by the network must closely resemble the ground truth. Since the word generation process is not continuous, it is challenging to optimize these metrics directly through backpropagation. In this study, we propose a novel approach to image caption decoding that enhances the robustness of existing models by incorporating prior knowledge from pre-trained cross-modal CLIP models. The results of comparative experiments demonstrate that the Monte Carlo Tree Search (MCTS)-based approach yields favorable outcomes in terms of evaluation metrics.