Traditional anomaly detection algorithms have become important for assisting security personnel in monitoring surveillance footage. However, these methods lack semantic understanding and robustness to scenarios outside of the training domain. In this paper, we redefine Surveillance Video Understanding (SVU), focusing on identifying and describing key events in surveillance footage that represent anomalous situations, such as violence, crime, and accidents. We leverage the zero-shot and descriptive capabilities of Large Vision-Language Models (LVLMs) and assess their performance in SVU. To evaluate LVLMs in SVU, we propose and validate a LLM judge, that rewards the detection of key events while not penalizing the omission of background details. From the judge’s assessments, we translate two quantitative metrics and report performance of four state-of-the-art LVLMs. Our analysis shows that LVLMs can capture contextual cues reasonably well but struggle to recognize the main abnormal events of the video, especially if only subtle visual cues that disclose individuals’ intent are available. This work offers a modern perspective on SVU and enables a more interpretable and generalizable evaluation approach.