The proliferation of online learning platforms allows for greater scaling of educational experimentation than ever before. This includes attempts to personalize the delivery of online educational content to meet individual student needs, often through some form of reinforcement learning. This work explores different reinforcement learning approaches for personalization, paying particular attention to supplemental instructional materials in the form of hints and explanations. Incorporating lessons learned from prior work, we discuss a framework for improving bandit algorithms through feature engineering, and detail our preliminary attempts to apply bandits with engineered features to a real-world context. The goal of this work is to learn which features are important for personalization, to identify qualitative interactions that exist between these features, and develop strategies for using historical data to give our learning agents a “warm start.”