Open-Set Cultural Object Detection Based on Vision-Language Pre-Trained Models Algorithm

With the rapid development of digital humanities and cultural heritage protection, it has become crucial to automatically identify and catalog cultural objects in a vast amount of image and video materials. However, traditional object detection algorithms usually operate under the “closed-set” assumption, they can only recognize pre-defined categories that have appeared in the training set, and it is difficult to handle the “open-set” scenario given the wide variety of cultural objects and the constant emergence of new categories. To address this challenge, this paper proposes an open-set cultural object detection algorithm, named VL-COOD, based on vision-language pre-trained models (VLPMs). This algorithm utilizes the powerful zero-shot recognition ability of VLPM (such as CLIP) and uses text descriptions as an open vocabulary, enabling the model to recognize “unknown” cultural objects not seen in the training set. Specifically, we design a “Semantic Calibration Module (SCM)”, which uses the semantic information of text embeddings to enhance the discriminability of visual features through an attention mechanism. In addition, we introduce an “Energy-Guided Open-set Loss”, which simultaneously optimizes the classification accuracy of known classes and the ability to distinguish known/unknown classes during the training process. We have conducted extensive experiments on the self-built “Cultural Heritage-Open Dataset (CH-Open)”. The results show that compared with the current mainstream open-set detection methods, the proposed VL-COOD algorithm in this paper has improved the detection accuracy (mAP) of known classes by 4.7% and the recognition recall rate of unknown classes by 8.2%, demonstrating the effectiveness and superiority of this method in the open-set cultural object detection task.