With the rapid growth of digital education and online assessments, maintaining the integrity of remote examinations has become a critical challenge. This study evaluates the effectiveness of pre-trained convolutional neural networks (CNNs), including InceptionV3, DenseNet121, YOLOv5, and YOLOv8, for detecting abnormal activities in online exams. The proposed system leverages YOLOv8, an advanced object detection model, to enhance real-time cheating detection. The research involves dataset preprocessing, model training, performance evaluation, and real-time deployment using Flask. Experimental results demonstrate that YOLOv8 outperforms other models, achieving 99% accuracy, precision, recall, and mean average precision (mAP), making it the most effective solution for automated proctoring. The study confirms that deep learning-based systems can significantly reduce the need for human invigilators, ensuring secure and scalable online exam monitoring.