The widespread adoption of online learning has made traditional engagement evaluation methods, which rely on teachers’ subjective judgments, no longer practical. It is essential to develop an accurate and automatic evaluation method. However, most existing methods for assessing student learning engagement rely on CNNs, whose limited receptive fields restrict their capacity to model long-range dependencies in video data. Additionally, the scarcity of low-engagement students compared to highly engaged ones in real classrooms introduces model bias, resulting in the misclassification of low-engagement (rare) instances as high-engagement (common) classes. They lead to limited accuracy in engagement recognition methods. This paper investigates approaches to enhance recognition accuracy by optimizing video representation and addressing data imbalance. Specifically, we introduce a dual-stream pretrained video transformer framework into student engagement recognition, which jointly learns spatial representations and temporal dynamics to generate more discriminative video representations. Furthermore, we improve model performance on rare categories by calibrating prediction scores to achieve better inter-class balance. We conducted extensive experiments on the challenging DAiSEE benchmark, with our method achieving promising results. Comparative experiments show that our method achieves a 9.02% improvement in accuracy compared to Vision Transformer, which also employs transformer, and outperforms all CNN-based methods.