Server-based face recognition access control (SFRAC) is a widely used biometric authentication method where a camera captures a user’s face image and sends the face information to a backend server to determine an authentication result. In existing systems, user face images or raw facial features are sent to the backend server, which induces privacy concerns. This paper proposes a method to protect user face privacy in SFRAC systems, such that user face images or raw facial features are never sent to the server, but the system can still do face-based access control. The method uses a novel partially order-preserving hash-induced transformation (POP-HIT) to perform feature transformation on facial embeddings. POP-HIT maps a user’s facial features to a transformed space in order to preserve privacy. Authentication is performed by checking the transformed facial embedding of the user’s face image captured at the time of authentication against the user’s pre-stored transformed facial embeddings at the server, based on similarity metrics or machine learning methods. Since the server only sees transformed facial embeddings, which cannot be used to recover the facial image, facial privacy is protected. Analysis and experimental evaluations show that our method has a high authentication accuracy, and it achieves privacy protection without being vulnerable to impersonation attacks.