Open-vocabulary multi-label classification with visual and textual features fusion

Multi-label zero-shot learning (ML-ZSL) is a significant and challenging task in computer vision which aims to recognize multiple unseen classes from an image without corresponding annotated training data. However, existing ML-ZSL methods trained vision and language branches separately, ignoring inter-modal fusion to obtain semantic information. Open vocabulary multi-label classification (OV-MLC) is a more practical setting and more effective than ML-ZSL because it extends the recognition ability based on the rich semantic information in visual-language pre-training (VLP) model (e.g., CLIP). However, due to the limited expressive power of prompt in CLIP, the performance of existing OV-MLC methods in terms of representation of seen classes and generalization ability needs to be improved. To address these issues, visual and textual features fusion (VTFF) framework is proposed based on a vision-language pre-training model to deal with open-vocabulary multi-label classification. First, we explore an improved transformer decoder design by fusing visual and textual embeddings which enhances the ability of contextual and visual understanding. Then, to better adapt to the OV-MLC task and enhance visual-semantic consistency, we present a hybrid prompt learning module that combines learnable prompt and fixed prompt. The representation and generalization ability of our model is substantially enhanced by fusing textual and visual embeddings. Finally, extensive experimental results demonstrate that our method outperforms state-of-the-art ML-ZSL methods on public benchmark NUS-WIDE and Open Images datasets. The source code is available at https://github.com/tong79/OVMLC_VTFF.