<p>This study evaluates the quality of questions generated by large language models, such as ChatGPT, in the context of cybersecurity education, using the CompTIA Security + certification as a case study. By analyzing 360 human-authored questions across 17 chapters and comparing them with AI-generated counterparts, the study employs Bloom’s Taxonomy to assess the cognitive levels achieved. The results reveal that AI-generated questions are predominantly limited to lower-order cognitive tasks, such as remembering and understanding, with significant gaps in addressing higher-order cognitive tasks, including applying, analyzing, evaluating, and creating. To address these limitations, a multi-layer agent framework was developed and implemented on a website. This framework integrates the ChatGPT API and processes the generated questions through multiple stages of evaluation, aligning them with Bloom’s Taxonomy and enhancing their quality. The framework includes scenario-based refinements, domain-specific fine-tuning, and a feedback mechanism to iteratively improve the cognitive depth of the questions. The agent systematically aligns AI-generated content with higher Bloom’s levels, making the questions more robust and applicable to real-world contexts. Evaluation of the framework demonstrates notable advancements in the quality of AI-generated questions, achieving closer alignment with human-authored content in terms of cognitive complexity, scenario depth, and relevance. The agent achieved substantial improvements in generating higher-order cognitive tasks, addressing the limitations of baseline AI performance. This study provides a scalable foundation for leveraging large language models in high-stakes educational assessments, with implications for adaptive learning and enhanced question design across diverse domains.</p>