Test-Time Tuning for Zero-Shot Spatio-Temporal Action Localization
摘要
Zero-Shot Spatio-Temporal Action Localization (ZS-STAL) aims to spatially and temporally localize persons, and recognize corresponding unseen action categories. Existing ZS-STAL methods typically fine-tune the vision-language models on extensive training data. Though promising, these training-based methods rely heavily on large-scale annotated data for training, which may be impractical in real-world scenarios. Additionally, the supervised-learning process inevitably introduces domain bias, limiting their generalization capabilities. Thus, we perform Test-time Tuning for Spatio-Temporal Action Localization (T \(^{2}\) Stal) by only utilizing test samples, eliminating the need for training data. T \(^{2}\) Stal comprises two main modules. Concretely, Spatio-Temporal Visual Prompt Pre-tuning (STVP2) initializes and pre-tunes visual prompts to reasonable states by making use of spatial prior knowledge and temporal visual continuity. Furthermore, Multi-view Multi-modal Prompt Adaptation (M2PA) synchronously tunes visual and textual prompts by enforcing the prediction semantic consistency among augmented views. Experiments on three benchmarks demonstrate that T \(^{2}\) Stal outperforms state-of-the-art VLMs’ baselines, suggesting the effectiveness of our method.