In recent years, the landscape of education has been profoundly transformed by the shift towards online learning, accelerated by global events such as the COVID-19 pandemic. While this transition has increased accessibility, it has also highlighted significant challenges in mathematics education, such as the “one-size-fits-all” approach that fails to address the diverse needs and emotional states of students. This issue is particularly acute in subjects like calculus, which require not only personalized feedback but also real-time interactions to effectively address misconceptions and foster comprehension. This study investigates the effectiveness of integrating advanced Facial Expression Recognition (FER) technology and ChatGPT prompts within a “Personalized Interactive Online Learning System”, to enhance the online learning experience using Moodle. Employing Convolutional Neural Networks, the system detects emotional cues from learners’ facial expressions, categorizing them into four types—active, passive, evaluative, and non-learners—to tailor content and teaching approaches. Additionally, ChatGPT provides real-time, context-aware support to promptly address student queries and clarify concepts. An experimental evaluation with learners using a mixed method approach demonstrated that those using this integrated system reported greater satisfaction and showed higher confidence and improved performance in calculus compared to a control group. This work holds considerable relevance for the global scientific community, as it advances AI-driven emotional intelligence in digital pedagogy and offers a scalable model adaptable across various disciplines. By supporting personalized, interactive, and effective learning environments, this research aligns with broader efforts to democratize quality education and improve digital education outcomes worldwide.