Evaluation of a Red Team Automation Tool in Live Cyber Defence Exercises

This paper presents an evaluation of the red team automation tool Lore in two live-fire cyber defense exercises (CDX). During the CDXs, Lore and manual “red” teams subjected 72 network security analysts (i.e., defenders; the “blue” side) to various threats such as software exploits and shell commands. Ten hypotheses related to how the actions by manual red teams and Lore are perceived and managed by the security analysts are examined. Evaluations were made by studying the subjective judgements of the analysts and by comparing the objective ground truth to their submitted incident reports. The results show that none of the null hypotheses could be rejected. In other words, the security analysts could not tell the difference between the actions made by the manual red team and those made by Lore, and their performance was similar regardless of the source of the threats.