With the wide-ranging installation of security systems in smart cities, academics can now analyze large amounts of data in order to ensure automated surveillance. To minimize casualties that could have a social, economic, or environmental impact, smart cities, educational institutions, healthcare facilities, and other monitoring domains need to have an increased protection system in place to detect hostile or anomalous activities. Automated violence detection is crucial for prompt action and can effectively aid relevant agencies. In this study, we provide an extensive deep learning architecture for three-stage violence detection. In the beginning, a lightweight simple convolutional neural network (CNN) model is used to distinguish persons under observation. Second, a series of 16 frames containing identified persons is delivered to a 3D CNN, which captures spatiotemporal properties and feeds them into the Softmax classifier. We additionally utilized Intel’s open source visual inference and neural network tuning toolkit to improve the 3D CNN model. This toolkit translates the training model to an intermediate representation and optimizes its execution on the ultimate platform to predict violent behavior. When hostile behavior is identified, an alarm is transmitted toward the appropriate security agency or police station, enabling for immediate preventive action. For a variety of datasets that serve as benchmarks, we observed that our suggested methodology outperforms current state-of-the-art methods.