CLIP guided image caption decoding based on monte carlo tree search
摘要
Recently, image captioning has attracted significant research interest due to its wide range of applications. The training process for captioning generative models typically employs backpropagation to maximize the likelihood of the next word given the previous ground truth word. Evaluation of this method is usually conducted using metrics such as CIDEr, ROUGE, and others. However, a disadvantage of this approach is that the words sampled by the network must closely resemble the ground truth. Since the word generation process is not continuous, it is challenging to optimize these metrics directly through backpropagation. In this study, we propose a novel approach to image caption decoding that enhances the robustness of existing models by incorporating prior knowledge from pre-trained cross-modal CLIP models. The results of comparative experiments demonstrate that the Monte Carlo Tree Search (MCTS)-based approach yields favorable outcomes in terms of evaluation metrics.