We propose a computational model of the auditory cortex inspired by recent developments in time-frequency analysis, as a defense against adversarial attacks on audio signals. Our proposed cortical network emulates the first stages of audio processing in human brain, by appropriately adapting the concept of the Fourier scattering transformation. To test the model’s efficacy we apply several white-box iterative optimization-based adversarial attacks to an implementation of Amazon Alexa’s HW network, and a modified version of this network with an integrated cortical representation, and show that the cortical features help defend against universal adversarial examples. At the same level of distortion, the adversarial noises found for the cortical network are always less effective for universal audio attacks.