Surveillance systems are being deployed and widely used in various areas such as traffic monitoring, airports, shopping centers and colleges. Although surveillance systems are active 24 × 7, detection of unusual activities is not possible by manual monitoring as it could be prone to errors. Hence, detection of unusual activity is a demanding area of research. This work aims to identify unusual activities happening in a campus setting such as fights and vandalism using videos from CCTV cameras. It proposes a smart system using a Long-Term Recurrent Convolution Network (LRCN). LRCN handles long video frame sequences and variations in lighting, making it suitable for real-world surveillance scenarios. The dataset used in this work is collected from college campuses and comprises of normal and unusual activities with the test and train split ration being 75% and 25% respectively. The dataset consists of a total of 1000 videos of which 500 are usual videos and remaining 500 are unusual videos. The paper aims to contribute to intelligent surveillance systems by improving accuracy and adaptability in recognizing activity in video frames. This proposed model achieved an accuracy of 99% which is comparatively better than the existing models.