Dual-Grained Alignment with Dynamic Clip Modeling for Partially Relevant Video Retrieval
摘要
Partially Relevant Video Retrieval (PRVR) aims to retrieve the most relevant video from untrimmed videos based on a given query. Videos in PRVR are typically analyzed at two granularities: frame-level and clip-level. Previous methods employ static approaches for clip feature modeling. However, these approaches may not be ideal as they introduce irrelevant clip information. We propose a Dual-Grained Alignment framework with Dynamic Clip Modeling (DGADCM) for Partially Relevant Video Retrieval. Specifically, DGADCM is composed of three branches: Clip-Align, Frame-Align, and Frame-Align+. In the Clip-Align branch, we dynamically model clip features using Gaussian masks and learnable weights and perform clip-level cross-modal alignment between clip features and query features. The dynamic clip modeling method can generate compact and context-rich clip embeddings while reducing the negative impact of irrelevant clip information. In the Frame-Align branch, we directly perform frame-level cross-modal alignment between frame features and query features. In the Frame-Align+ branch, we incorporate knowledge distillation to transfer cross-modal alignment capabilities from the CLIP model. Additionally, we introduce a query diversification loss that encourages queries originating from the same video to spread out within the embedding space. Our motivation is to achieve better cross-modal alignment by preventing queries from being overly close in the embedding space and aligning with the same video frame or clip. We conducted extensive experiments and ablation studies on two publicly available datasets, and the results demonstrate that our proposed method achieves state-of-the-art performance, validating its effectiveness.