Image Captioning with Visual Positional Embedding and Bi-linear Pooling

Recent approaches to image captioning typically follow an encoder-decoder architecture. The feature vectors extracted from the region proposals obtained from an object detector network serve as input to encoder. Without any explicit spatial information about the visual regions, the caption synthesis model is limited to learn relationship from captions only. However, the structure between the semantic units in images and sentences is different. This work introduces a grid based spatial position encoding scheme to learn relationship from both domains. Furthermore, bi-linear pooling is used with attention for exploiting spatial and channel-wise attention distribution to capture second order interaction between multi-modal inputs. These are integrated within the Transformer architecture achieving a competitive CIDEr score.