Image Captioning with Masked Diffusion Model

Some image captioning models adopt a non-autoregressive approach to independently generate each word, thereby speeding up the generation process. However, this generation method often sacrifices the quality of the generated captions. This paper aims to address this issue by proposing a novel diffusion model based on a non-autoregressive approach for image captioning tasks. Our model integrates a time-varying masking mechanism, gradually adding mask in the reverse diffusion process to guide image features selectively. Additionally, to further enhance the quality of generation, we introduce the CLIP model and fuse it with regional features to incorporate semantic information into the image features. This comprehensive utilization of visual and semantic information aids in generating richer and more accurate caption descriptions. To validate the performance of our model, we conducted extensive experiments and ablation studies on the MSCOCO benchmark. The experimental results demonstrate that our masked diffusion model combined with the CLIP model achieves highly competitive performance in caption generation tasks. Not only does it significantly improve generation speed, but it also yields satisfactory results in terms of generation quality. This study highlights the potential applications and importance of our approach in the field of image captioning.