Within autonomous cyber defence, building scalable agents that can generalise across attack behaviours is crucial to developing a truly autonomous system. These generic agents are pivotal, as over time, attackers will inevitably change their behaviour, requiring the defence mechanisms to adapt accordingly. Current approaches for generic agents use deep reinforcement learning policies to learn multiple attack behaviours and mitigate them. When a new attack behaviour is introduced, the generic policy is retrained to incorporate this behaviour and not forget previous attack behaviours. In this paper, we propose a novel solution based on a modified version of the Proximal Policy Optimisation (PPO) reinforcement learning algorithm that retains previously acquired knowledge, enabling a scalable and generic framework in which new attack behaviours can be incorporated modularly. The modified PPO algorithm demonstrates a 22.11% performance improvement compared to standard PPO when trained to sequentially learn two distinct attack behaviours. These results show a step towards building more scalable autonomous cyber defence systems capable of incorporating evolving cyber threats.