Biometric-based authentication systems are evolving as a robust and viable method as it involves personalized biometric features of the person who is being authenticated. Biometric features are extracted and stored in the server. An untrusted server or third-party cloud server is more vulnerable to spoofing attack and Denial of Service (DoS) attack. Apart from this, there is a demand to preserve the privacy of the individual person. In this work, we have proposed the Privacy-preserving and anti-spoofing-based biometric Authentication System (PAS). As the Discrete Logarithm Problem (DLP)-based crypto model is evolving as a dominant representative for security, the Genus-2 Hyper Elliptic Curve (HEC) is explored for biometric template protection. HEC over prime field is designed with 256 bits’ prime number and is corroborated with biometric signals in our previous work. Face is used as a biometric and the face template is transformed into Genus-2 HEC divisors on the fly. In order to allow the scalability, these divisors are stored in the third-party cloud after performing inline compression that reduces storage need by a factor of 2. The novelty is in the formulation of a one-way function that ensures privacy-preserving template protection in the untrusted cloud. Anti-spoofing is reinforced by detecting liveness of the face biometric by incorporating blurring and distortion. The proposed model is verified against statistical attacks and proved that it preserves privacy, reluctant to spoof attacks. The security of the face protection phase is validated by MSE and PSNR which has attained highest ideal MSE between original and encrypted, PSNR with zero between original and decrypted. The face liveness detection phase is endorsed with confusion plot and gains 92.8% accuracy. As the proposed system is light-weight, memory consumption and computing capacity is very less when compared to state-of-the-art authentication systems. The proposed system can be used for authenticating remote users in e-Health and any e-Commerce and e-Banking applications which demand privacy and security. This framework is also analyzed against state-of-the-art cancellable remote biometric authentication systems and has substantiated its strength against other models.