Cross-modal spatio-temporal fusion weakly supervised video anomaly detection based on large-scale vision-language models
摘要
Video anomaly detection plays a vital role in the development of intelligent industry and smart city by effectively identifying and responding to abnormal events, which enhances production efficiency and the safety of urban operations. Existing large-scale visual-language models still suffer from insufficient extraction of spatio-temporal features and semantic disconnection in the fusion of multimodal data. To address this, this paper proposes a weakly supervised video anomaly detection method based on large-scale visual-language models with cross-modal spatio-temporal fusion (WSDL-CAM). In this task, we designed a Spatio-Temporal Context Adapter Module (STCA) that effectively uses the Mamba network’s ability to capture long-range dependencies in the spatio-temporal context information extracted by the spatio-temporal graph convolutional network, achieving sufficient extraction of spatio-temporal features from the video. In addition, we designed an Image-Text Semantic Perception Fusion Module (ITSPF) that associates fine-grained visual features with text features to judge semantic consistency, further enhancing the semantic relevance between images and text. Finally, we implemented hyperparameter self-optimization in training and evaluation through Bayesian optimization algorithms. We conducted extensive experiments on two large-scale datasets with real-world surveillance scenarios (XD-Violence and UCF-Crime). Experimental results show that our model has achieved state-of-the-art performance in video anomaly detection.