Relevant Tag Extraction Based on Image Visual Content

Social media web services like Flickr allow users to share and freely annotate images with textual tags. These tags play a crucial part for text-based social image retrieval and browsing tasks. However, these tags are usually irrelevant and incomplete which limits their effectiveness and use. One fundamental problem is to interpret the relevance of tags with respect to the image visual content. Existing solutions have targeted either image visual content or user contributed tags separately to address the issue. Our proposed method Tag-tag, however, combines both aspects and exploit their semantic relationship. We use state of the art pretrained machine learning models in Tensorflow for object detection and NLP (Natural Language Processing) for semantic analysis. Our experiments on a dataset of 219 randomly collected Flickr images demonstrates the applicability of our method. Images with missing tags were excluded from the experiments. We identified two reasons where relevant tag was not found as: (1) relevant tag itself was missing from user contributed tag list, (2) relevant tag got skipped because it was either not found in the Wordnet dictionary or has no pretrained vector in word2vec-google-news-300 model. We identify this as one limitation of the proposed method.