This article presents a comparative analysis of generative AI (GenAI) model efficiency in supporting Chief Information Security Officer (CISO) tasks across operational (Op), technical (Tech), human (Hum), and physical (Phy) security domains. The study evaluates multiple GenAI models (GPT-4, GPT-4o, O1, O3-mini, O3-mini-high) by assessing their performance in handling security-specific prompts related to data analysis, anomaly detection, and regulatory compliance. A structured methodology was developed using Lithuanian and EU cybersecurity legislation to define evaluation criteria and domain-specific requirements. Models were tested through repeated trials using uniform prompts, with expert evaluation focused on response accuracy, consistency, and contextual relevance. Token usage was analysed to assess computational efficiency, revealing that higher token counts do not always correlate with better performance. Findings indicate significant variability in model responses due to stochastic behaviour and differences in architectural design, highlighting the need for human oversight in security-critical applications. The study concludes that while GenAI holds promise as a CISO support tool, it must be integrated cautiously and with clearly defined boundaries.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Comparative Analysis of Generative AI Model Efficiency in Chief Information Security Officer (CISO) Tasks

  • Renata Danielienė,
  • Kęstutis Driaunys,
  • Ilona Veitaitė,
  • Martynas Bartnykas,
  • Rokas Stankūnas,
  • Ieva Šilingaitė,
  • Dainora Kuliešienė

摘要

This article presents a comparative analysis of generative AI (GenAI) model efficiency in supporting Chief Information Security Officer (CISO) tasks across operational (Op), technical (Tech), human (Hum), and physical (Phy) security domains. The study evaluates multiple GenAI models (GPT-4, GPT-4o, O1, O3-mini, O3-mini-high) by assessing their performance in handling security-specific prompts related to data analysis, anomaly detection, and regulatory compliance. A structured methodology was developed using Lithuanian and EU cybersecurity legislation to define evaluation criteria and domain-specific requirements. Models were tested through repeated trials using uniform prompts, with expert evaluation focused on response accuracy, consistency, and contextual relevance. Token usage was analysed to assess computational efficiency, revealing that higher token counts do not always correlate with better performance. Findings indicate significant variability in model responses due to stochastic behaviour and differences in architectural design, highlighting the need for human oversight in security-critical applications. The study concludes that while GenAI holds promise as a CISO support tool, it must be integrated cautiously and with clearly defined boundaries.