In the education technology field, there is a need to transform present smart classrooms into artificially intelligent-enabled classrooms. These classrooms can assist teachers in automating mundane activities such as student engagement and behavior monitoring and feedback through vision-based frameworks, leading to effortless and effective classroom management for optimal teaching and learning outcomes. Student classroom behavior monitoring is vital in classroom learning as it directly relates to their engagement. Recently, many vision-based frameworks have been proposed in this research direction. However, most of these works have attempted direct visual engagement estimation using facial expressions, and couldn’t extend to behavior monitoring. Furthermore, these works process independent video frames for this purpose, omitting temporal information in video sequences for more accurate behavior classification considering context. Additionally, there is hardly any student behavior video dataset recorded in real classroom settings. This paper attempts to address these research gaps by developing a novel classroom behavior video dataset named the Student Classroom Behavior Video Dataset (SCBVD). The SCBVD comprises 10 distinct commonly observed classroom student behaviors, such as using mobile devices, talking to others, and looking around. This paper also presents baseline results on this dataset by experimenting with two renowned deep learning-based video classification methods: Long-Term Recurrent Convolutional Network (LRCN) and 3D Convolutional Neural Network (CNN). Our results show that our customized 3D CNN architecture outperforms the LRCN model. Furthermore, both models were further evaluated against two benchmark video datasets, UCF50 and JHMDB, highlighting the effectiveness of the 3D CNN model in achieving higher accuracy.