Security awareness and training are often mooted as the path forward in mitigating security breaches. Cyber-security training encourages users to take a proactive role in protecting the systems they use. Yet, in spite of increased cyber-security training schemes, statistics indicate that the number of online harms due to scams, phishing, and exposed/stolen personally identifiable information (PII), is on the rise. In this paper, we provide insights on why security training does not translate to increased security awareness on the part of users. We tested our hypothesis with an online security education course delivered on a Massive Open Online Course (MOOCs) platform to 1929 participants aged between 18 and 70. Following a training (education) phase, participants were given a variety of exercises tailored to mimic real-life security decision scenarios. Our findings indicate that participants showed increased levels of security awareness, but were not always able to translate their learnings to handling realistically complex scenarios effectively.