Federated learning (FL), a distributed machine learning paradigm, has attained significant attention in recent years. FL addresses privacy concerns by enabling multiple parties to collaboratively train a model through gradient exchange instead of sharing raw data. However, inherent privacy risks persist in FL, such as the potential inference of user training samples from gradients. Additionally, the emergence of poisoning attacks poses a significant security threat to FL, where malicious users may tamper their local data or submit crafted gradients to compromise model integrity. The integration of privacy-preserving federated learning solutions with defenses against data and model poisoning attacks will significantly improve the overall security and privacy protection in FL. To address this dilemma of simultaneously mitigating data and model poisoning while ensuring the privacy of transmitted weights, this paper develops and introduces GuardedLearn (GL) framework. GL is a novel privacy-enhanced FL framework engineered to safeguard against poisoning attacks. It employs techniques, including local masking and gradient analysis, to empower the server in penalizing malicious behavior. Through meticulous feature extraction, resilient aggregation strategies, and streamlined clustering algorithms, GL sets a new standard in mitigating both data and model poisoning in FL settings. We conducted extensive experiments and comprehensive analyses, which clearly demonstrate the efficiency of GL and its superiority over existing models.