Investigating Large Language Models for Prompt-Based Open-Ended Question Generation in the Technical Domain

We explore the automated generation of open-ended questions from technical domain textbooks. These questions are more diverse than those typically examined in the field of question generation (QG) for reading comprehension. To facilitate this endeavor, we curate EngineeringQ, a prompt-based QG dataset that contains triples of (1) Context: a segment from which the questions are formed; (2) Prompt: a prompt is a concise and specific keyphrase that serves as a short guiding reference to generate a question based on a given context; (3) Question: a question that aligns with the context and is coherent with the prompt. We evaluate the performance of several fine-tuned encoder-decoder based large language models (LLMs), including Pegasus, BART, Flan-T5, and T5 on EngineeringQ. We introduce a novel prompt-tuning method for these encoder-decoder based LLMs. We also investigate the potential of general-purpose decoder-only LLMs such as GPT \(-\) $-$ 3.5 Turbo, text-davinci-003, and GPT-4 in zero-shot setting. Evaluation involves automated metrics and human evaluation by domain experts. Extending our study beyond EngineeringQ, we apply these methods to subjects such as biology, chemistry, earth science, and physics. Furthermore, we investigate the potential for domain adaptation by fine-tuning the best-performing LLM on school-level subjects and assessing its effectiveness on undergraduate-level computer science and information technology subjects for zero-shot and few-shot QG. To gain insights into generated question complexity, we also utilize Bloom’s revised taxonomy to categorize questions into different levels, enhancing our understanding of their educational value. Experimental results highlight T5_LARGE’s superiority in automated evaluation metrics and text-davinci-003’s excellence in human evaluation metrics in EngineeringQ. For subjects such as biology, chemistry, earth science, and physics, Flan-T5_BASE excels in automated metrics, while GPT-4 leads in human evaluation. Notably, human baseline methods consistently outperform all AQG approaches, including fine-tuned LLMs, prompt-tuned LLMs, and zero-shot GPT models.