The rapid growth of surveillance video challenges traditional cloud computing with high bandwidth costs and transmission delays, while edge computing’s limited resources hinder efficient processing. Surveillance video analytics face unique challenges, including substantial content redundancy, varying inference complexity across frames, and strict real-time processing requirements. To address these issues, we propose CSVA, an efficient Complexity-driven and Semantic-aware Video Analytics approach leveraging edge-cloud collaboration. CSVA optimizes video preprocessing, handles consecutive similar video frames efficiently, and implements effective scheduling strategies by considering video semantic features and system workload variations. Our approach dynamically balances the accuracy-efficiency trade-off, adapting to changing video content and environmental conditions. Experiments on a real-world edge-cloud testbed demonstrate that CSVA improves average performance by 32.74% compared to baseline methods.