Yoga, with a history spanning hundreds of years, is often referred to as a “treasure of the world.” As global emphasis on health and fitness increases, yoga, which integrates physical, mental, and spiritual practices, has gained significant popularity. Correct yoga poses are crucial for achieving optimal results. Therefore, accurate recognition and classification of yoga poses are of great importance to practitioners. This paper introduces a novel intelligent yoga pose classification system, XcepSENet, which combines the feature extraction capabilities of Mediapipe with an improved Xception model and the SE blocks of SENet. Our system estimates and classifies five major types of yoga poses with low latency, aiming to provide high-accuracy and low-latency classification to assist practitioners in correcting their poses, thereby enhancing safety and effectiveness. Furthermore, in the Yoga dataset, the XcepSENet network is compared with three deep learning models—VGG16, InceptionV3, and MobileNetV2—evaluating metrics such as accuracy, precision, recall, and F1 score to draw conclusions, To prove that the model can provide feedback more timely and accurately to achieve higher training effects.