In the realm of modern Machine Learning and Deep Learning, ensuring data privacy is paramount. Encrypted inference holds tremendous potential in this arena and Fully Homomorphic Encryption (FHE) stands at the forefront of privacy-preserving technologies, offering a pathway to secure computation on encrypted data. While encrypted inference offers promising solutions to safeguard sensitive data, a critical question arises: Are the vulnerabilities and attacks that threaten conventional ML (Machine Learning) and DL (Deep Learning) models also applicable in the encrypted domain? This study particularly investigates this question in the context of evasion attacks. The major challenge of evasion attack in encrypted domain is that it is not possible to decide upon perturbations similar to plaintext methods. From Advesrary’s point of view, there are no known methods to generate adversarial examples by solving data-dependent optimization problems homomorphically over ciphertext space and it is also not possible to get attack assurance without knowledge of secret key. We present a solution where adversaries use partial training data distribution and Universal Adversarial Perturbations (UAPs), which are image-agnostic and capable of inducing misclassification across multiple images sampled from data distribution. Our research reveals that the equivalent notion of UAPs varies depending on whether a symmetric or public key FHE scheme is used, which we investigate specifically in the context of two libraries: Concrete ML and TenSEAL. While TenSEAL supports more direct attack methods, Concrete ML requires intermediate compiled representations for a successful attack in API-only setting. Additionally, Concrete ML’s integer-based quantization alters targeted UAP behavior. We propose a Quantization-aware UAP generation algorithm, which preserves UAP property, ensuring consistent attack success rates in encrypted and plain classifiers.