Federated Learning (FL) is a distributed machine learning technique that enables clients to train Deep Neural Networks (DNNs) collaboratively without centralized data storage. However, this distributed nature introduces security challenges, making FL vulnerable to backdoor attacks. In these attacks, an attacker injects triggers into local models and tampers with updates during aggregation, causing incorrect predictions for specific inputs. As backdoor attacks evolve to become more covert, detecting these stealthy backdoors has become a critical issue. In this paper, we design the Lightweight Focus Layer (F2L), an adjustable layer inserted into server aggregation rounds. F2L detects subtle variations in backdoor attack parameters and fine-tunes them to minimize their impact. Our approach can be combined with existing defenses, significantly enhancing the robustness of FL models. Testing on three public datasets showed that F2L maintained the main task’s accuracy while significantly reducing backdoor attack effectiveness.