The robustness of Artificial Intelligence (AI) models is particularly relevant for trusty, fair, and reliable applications such as healthcare, surveillance, face recognition, self-driving cars, among others. The Quaternion Monogenic Convolutional Neural Network Layer (QMCL) has demonstrated a remarkable performance to be robust in front of different types of contrast changes. In this work, we analyze the response of a more stable version of QMCL, with only two trainable parameters, to brightness changes and adversarial attacks. The experimental results on benchmark datasets and with face gender classification reveal that QMCL significantly improves the resilience of Convolutional Neural Networks, maintaining high accuracy classification under diverse lighting conditions and adversarial attacks (white and gray). One limitation of the QMCL is that the performance with the original images (no perturbation) is lower than that with the same ConvNet with a regular layer. The code of this paper is available at Github