Deep learning has shown great promise in the medical image analysis domain. Medical professionals and healthcare providers have begun to adopt this technology to accelerate and enhance their work. These systems use deep neural networks (DNNs) which are vulnerable to adversarial samples: images with imperceivable changes that can alter the model’s prediction. Prior research has proposed defenses aimed at making DNNs more robust or detecting the adversarial samples before they can do any harm. However, none of the studies considered an informed attacker capable of adapting the attack to the defense mechanism. In this qualitative study, we show that an informed attacker can evade five advanced defenses, successfully fooling the victim deep learning model and rendering the defense useless. We also propose two alternative means of securing healthcare DNNs from such attacks: (1) hardening the system’s security, and (2) using digital signatures. Finally, we discuss measures the healthcare community should take to mitigate this threat and explore the evolving threat landscape as large language models (LLMs) become increasingly integrated into the industry.