Video data are vulnerable to illumination changes and background variations, while the scarcity of annotated samples in publicly available violence datasets further constrains the generalizability of existing recognition schemes. As a result, these schemes often fail to accurately detect violent actions in unseen scenarios, leading to limited robustness. To address this challenge, we propose a novel video violence recognition scheme that integrates multi-modal visual features and language-based semantic understanding enhancing robustness across diverse environments. Specifically, to mitigate the impact of lighting and background fluctuations, we extract spatio-temporal features from raw video frames and human motion cues from pose representations. These heterogeneous features are fused via a cross-attention mechanism based on a Transformer architecture, generating a unified representation that captures both appearance and structural motion information. To reduce reliance on large annotated datasets, we incorporate a vision-language framework trained with instruction tuning on video-centric question-answering data, enabling prompt-based inference. We evaluate our method in a zero-shot setting on two public violence benchmarks, where it achieves accuracy gains of 3.0% and 1.9% over prior state-of-the-art approaches, respectively. In addition, few-shot experiments demonstrate that the model maintains competitive performance with limited labeled samples, confirming its strong generalization capability.