Audio meets text: a loss-enhanced journey with manifold mixup and re-ranking

Audio–text retrieval is the task of aligning natural language and audio such that instances from different modalities can be compared with a similarity metric. Contrastive representation learning is a popular way to address this problem by finetuning a dual encoder backbone. However, finetuning incurs a heavy computational burden and requires multiple graphics processing units. The main objective of this study is thus to drive enhanced retrieval performance in audio–text retrieval with lower computational burden and resources. Thus, a series of plug-and-play modules are introduced to elicit knowledge from pretrained audio and text encoders. From creating a competitive baseline with early and late fusion first, the study endeavors to improve the performance further by adapting deep metric Circle and Ranked List losses. This study utilizes Manifold Mixup as a strong regularizer and a novel post-processing step of re-ranking to refine the results further. These changes work in symphony to obtain superior performance than other models without the need to finetune encoders. Particularly, an improvement of 1–3% is obtained on the AudioCaps dataset, establishing a new state of the art. The results on the Clotho dataset remain competitive with other finetuning approaches utilizing heavy resources. Also, the model emerges as the best among the frozen encoder models across all the metrics. Moreover, the proposed modules are modality agnostic and hold great potential for other retrieval tasks beyond the domain of audio–text. Overall, this study establishes a strong and competitive baseline for future approaches in audio–text retrieval.