Strengthening Cyber Security Education: Designing Robust Assessments for ChatGPT-Generated Answers

Cyber security education has become a hot topic in Australia and many OECD countries due to increasing job demands for cyber security professionals. Designing authentic cyber security assessment tasks is an ongoing challenge, especially in the context of ChatGPT and similar AI-generated content (AIGC) tools. Some early studies suggest that the risks of using ChatGPT tools can be mitigated, but these studies overlooked cyber security education. This paper addresses this gap in the literature, focusing on assessment design in cyber security education in the presence of ChatGPT. While existing research has examined the transition from in-person to online education and ChatGPT’s capabilities, our study emphasizes the assessment structure and pedagogical approaches related to cyber security education. We conducted a systematic analysis by creating questions with four distinct prompts, feeding them to ChatGPT, and analyzing the answers with statistical tools. Our findings highlight the significance of question types and fact-checking in ChatGPT’s responses. We propose practical recommendations to enhance cyber security assessment design when incorporating ChatGPT. Our recommendations include incorporating recent academic references, using long essay questions, and thorough fact-checking to ensure the integrity of assessments.