Image Text Generation Based on Multimodal Fusion

The image text generation task involves both natural language processing and computer vision and is a representative task in multimodal research, which can provide relevant help for virtual assistants, image indexing, and people with disabilities. In this paper, we propose a “Pre-trained ResNet50 + LSTM” model to auto-generate image text for the image text generation task. The model uses the pre-trained ResNet50 network to extract the image features and the word embedding to extract the text features, and the image and text fusion feature vectors obtained by the earlier fusion method are used as the input to the LSTM network to obtain the output image description text. The model is trained and tested with two other models in the Flickr8k dataset, and the three are evaluated using accuracy and BLUE scores. The experimental results show that the proposed model in this paper is effective in improving the accuracy of the generated text compared with the other two models, and the model can successfully generate relevant explanatory text for images.