Lightweight Image Captioning Model Based on Knowledge Distillation

The performance of image captioning models based on deep learning has been significantly improved compared with traditional algorithms. However, due to the complex network structure and huge parameters, these image captioning models are hard to apply to resource-constrained appliances such as mobile embedded systems. To address this issue, we propose two lightweight image captioning models based on two pre-training models, i.e. \({M^2}\) Transformer and CaMEL. To improve the performances of the proposed lightweight models, we also develop an optimization training strategy based on knowledge distillation. Specifically, we design knowledge distillation loss functions in the encoder, decoder, and response modules, which can use the multi-modal feature mapping of the original models to transfer prior knowledge, improving the feature learning ability and generalization ability of the lightweight models. Experimental results show that the proposed lightweight models are superior to the original pre-trained models in performance when reducing the parameters greatly. Besides, the proposed lightweight models also have a good knowledge transfer ability.