Open-Vocabulary Scene Graph Generation via Synonym-Based Predicate Descriptor
摘要
This paper proposes a class balance-aware open-vocabulary scene graph generation (SGG) method that cooperates with predicate synonyms to improve unseen predicate inference. Existing SGG works address open-vocabulary relation predictions utilizing vision language pre-training through image caption datasets. However, there remains a gap between the dataset we use for pre-training and the open world; there are many unseen predicates, and the class imbalance in the training dataset further exacerbates the sensitivities to the unseen predicates. This paper introduces two solutions to improve the zero-shot capability: 1) Predicate Descriptor and 2) Fore-ground Relation Sampling on scene graph datasets. The Predicate Descriptor employs the mean pooling of augmented synonym text embeddings for calculating the similarity between text and visual features. The augmented synonym text embeddings make a relevant unseen predicate feature closer to the distribution in the feature space of the seen ones. As a result, our Predicate Descriptor can enhance the similarity score of the relevant unseen predicate. Fore-ground Relation Sampling can improve a wide range of types of predicate inference, including unseen ones, by reducing bias of scene graph datasets. The key to Fore-ground Relation Sampling is to avoid ignoring rare predicate samples, resulting in predicate class balancing. Comprehensive experiments on the public dataset demonstrate that our approaches cooperatively improve the zero-shot capability, resulting in a state-of-the-art accuracy of unseen predicate inference.