Privacy-Preserving Federated Learning (PPFL) enables efficient model training and prediction while safeguarding clients’ data privacy. However, malicious clients can launch poisoning attacks by manipulating local training data or model updates sent to the server which may lead to an incorrect global model. Existing defense methods for detecting malicious clients typically fail to balance privacy, overhead, and accuracy. To address this, there is a pressing need for an efficient PPFL framework that can resist poisoning attacks. In this paper, we propose a lightweight PPFL framework LBRFLthat simultaneously achieves privacy, efficiency, and accuracy. Specifically, we introduce a lightweight privacy-preserving clustering method that leverages secure Euclidean distance computation to process masked gradients uploaded by clients efficiently. To minimize the impact of malicious clients and maintain the accuracy of the global model, we propose an efficient Byzantine-Robustness secure aggregation method by introducing removable masks. In addition, experiments on the MNIST and CIFAR-10 datasets demonstrate that, compared to existing methods, LBRFLeffectively defends against common poisoning attacks in both independent and identically distributed (IID) and non-IID data settings without heavy encryption costs.