Predicting Course Performance on a Massive Open Online Course Platform: A Natural Language Processing Approach

Massively open online courses (MOOCs) and platforms such as Udemy have proliferated in recent years. These courses run the gamut from highly successful and high-rated to courses with very low ratings and little engagement. This research aims to address the challenge of preemptively identifying potentially low-rated courses by leveraging instructor-provided textual information. Our approach involves a two-stage process. First, we employ transformer-based Large Language Models (LLMs) to extract semantic information from the text provided by the instructors on the Udemy platform. In the second stage, we incorporate the extracted information as additional features into an upstream predictive model. To the best of our knowledge, this is the first attempt to use extracted semantic information from MOOC courses as features in a predictive model. In general, we find that existing consumer research findings hold and identify three key takeaways. First, we find that an instructor’s prior performance is a strong indicator of future ratings. Second, we see that including semantic information contained in instructor-provided text can have an additive effect on model performance. Finally, we demonstrate that fine-tuning language models on Udemy-specific text have an appreciable positive effect on upstream model performance.