Audio-Text Retrieval: Exploring Shared Parameters and Intra-Modal Constraint Loss

Cross-modal retrieval involves retrieving information across diverse modalities, like image-text, image-audio and audio-text. It finds application in multimedia search engines, healthcare imaging, recommendation systems and more. While many contributions have focused on the bimodal domain of image-text, there has been a lack of focus on other domains, particularly the intersection of audio and text. This study aims to bridge this gap by introducing a model capable of retrieving relevant audio clips based on natural language queries and extracting relevant captions from audio inputs. Many approaches in this line of effort involve fine-tuning pre-trained encoders with a contrastive objective. Only some approaches explore the impact of correlating embeddings retrieved from encoders without fine-tuning. Progressing in this direction, this study examines the impact of aligning embeddings in the shared space. The main contributions include: the development of a Siamese neural network for cross-modal entity alignment, along with incorporating the Intra-modal constraint loss for learning useful representations in the audio-text domain. The experiments are conducted on the standard datasets of AudioCaps and Clotho. The results demonstrate that the architecture performs better than certain other architectures involving frozen encoders. This study evaluates the effect of projection layers, weight sharing and intra-modal similarity using Siamese neural network and Intra-modal constraint loss, thereby opening up further avenues in representational learning in the audio-text domain.