Semantic Segmentation Based Image Signature Generation for CBIR

Content-Based Image Retrieval (CBIR) leveraging semantic segmentation integrates semantic understanding with image retrieval, enabling users to search for images based on specific objects or regions within them. This paper presents a methodology for constructing image signatures, a pivotal element in enhancing image representation within a CBIR system. The efficiency and effectiveness of a CBIR system significantly hinge on the quality of the image signature, which serves as a compact and informative representation of raw image data. Our proposed methodology begins with emphasizing clear object or region boundaries through pixel-level semantic segmentation masks. A pretrained semantic segmentation model, such as DeepLab v3+, is employed to generate pixel-wise object class predictions, yielding the necessary segmentation masks. Subsequently, each image is segmented into meaningful regions based on these masks, and relevant features are extracted from each segmented region using a pre-trained Deep Convolutional Neural Network (DCNN) models AlexNet, VGG16 and ResNet-18. During the retrieval phase, when a user queries the system with an image, the query image is segmented using the pre-trained semantic segmentation model, and features are extracted from the segmented regions of the query image. These query features are utilized to search the database for the most similar regions or images. Similarity scores, calculated using Euclidean distance, are used to rank the database entries based on their similarity to the query, allowing for efficient retrieval of the top-k most similar regions or images. We found that for some classes semantic segmented based retrieval better performance in comparison to image based.