<p>Detecting abnormal behavior in densely populated environments presents a major challenge for intelligent video surveillance systems, particularly in achieving real-time, computationally efficient analysis of massive video streams. Traditional deep CNN architectures, although effective, often involve excessive parameters and computational overhead. To address this, the proposed research introduces a hybrid deep learning model that combines lightweight and optimized CNN variants, including an improved YOLOv8 for abnormal behavior detection and a Modified Deep Residual Shrinkage Network (MDRSN) for classification, with the Adapted Aquila Optimization Algorithm (AAOA) for hyperparameter tuning. This architecture significantly reduces redundancy and computational cost while maintaining high detection accuracy. The proposed YOLOv8-MDRSN model is validated on three publicly available datasets: UCSD Ped1, UCSD Ped2, and HaJJV2. It achieves state-of-the-art performance, with accuracy rates of 99.97%, 99.93%, and 99.95%, respectively. Corresponding F1-scores are 99.96%, 99.91%, and 99.93%, and AUC scores reach 99.30% for Ped1 and Ped2, and 98.30% for HaJJV2. These results confirm the model’s superior capability in real-time abnormal behavior classification with minimal false detection rate (FDR), achieving values of 0.083, 0.092, and 0.120 for the three datasets, respectively. This research demonstrates the practical applicability and robustness of the proposed method in complex surveillance scenarios.</p>