Recognition of violent activity is crucial for ensuring the safety and security of all individuals and working professionals in various environment including public sectors, private sectors, workplaces, schools, and rather sensitive areas like emergency care facilities. The main objective circumvents the possibility of detecting unauthorized intrusion, unwanted attacks, or similar physical violence which causes discomfort, harm or injuries, without requiring constant human surveillance. This study presents an advanced methodology to train a model on aggressive behaviors, gestures, and movements for rapid violence detection so that actions can be taken is very less response time. The proposed model uses DenseNet121 for spatio-temporal feature extraction and TCN for temporal analysis and classification which has demonstrated success in accurately detecting violence from videos and generalizes well to unseen data. For this, a novel Indian violence dataset was introduced for evaluating the effectiveness of the presented model. The Real Life Violence Situations dataset as well as the novel Indian Violence dataset were used for training the model, as well as testing it.