Automated question-asking systems play a crucial role in enhancing student learning by enabling repeated practice and frequent testing with flexible scheduling. With the rapid advancement of large language models (LLMs), we propose a pipeline for generating questions using LLMs while ensuring reliability in high-stakes assessments. Rather than directly generating student-facing questions, our approach produces code for PrairieLearn, an autograding platform that supports extensive randomization and customizable components, allowing for individualized questions with immediate and accurate feedback. To improve question quality, we augment instructor prompts with relevant information retrieved from example PrairieLearn questions and documentation. Additionally, we implement custom validators to automatically detect and repair non-functional questions, reducing the need for manual debugging. We evaluate our pipeline through a user study and an ablation analysis, demonstrating that each component contributes to improved question functionality. Our approach significantly lowers the barrier for instructors unfamiliar with PrairieLearn, enabling them to benefit from automated question generation and grading.