Image description using tags latent concepts in convolutional neural networks

Automatic image annotation systems are employed to describe the visual content of images via tag assignment. Most of these image description systems use deep convolutional neural networks as feature extractors or multi-label classification systems. However, the majority of these systems do not optimally employ latent variables and image concepts within the side information. The idea of this study is to add a latent variable to the CNN-based annotation problem. A Bayesian modeling framework is used for this purpose. High-Level Semantic Concepts of tags are used as an instance of a latent variable to implement the proposed model. The Bayesian modeling result interpretation leads to localizing the entire image annotation process and breaks it down into smaller problems. Thus, a CNN-based sub-network is trained independently for each sub-problem and converted to an expert in annotation for a particular high-level semantic concept. To estimate tags in the test phase, the tags obtained from all sub-networks are combined in weighted form. The weight of each sub-net is proportional to the relation of its high-level semantic concept with the test image content. These weights are obtained from a separate general CNN as a classifier of concept. At the end, a tag refinement method is used to remove noisy tags. The obtained results (F1 values are 40.3, 35.2, and 38.1 for Corel5k, ESPGame, and IAPRTC12 datasets, respectively) confirm the effectiveness of the proposed method and can be comparable with the state-of-the-art annotation systems.