Strong security measures are required due to the growing use of IoT devices and constantly growing network sizes. In order to tackle some of the most important issues in IoT security, this paper investigates the use of optimization metaheuristics in XGBoost hyperparameter tuning. In particular, we suggest a brand-new modified metaheuristic algorithm that is intended to improve diversity throughout the search process and is modeled after the firefly algorithm (FA). Experiments with simulations on a newly released IoT security dataset show how well the proposed optimizer works to enhance model performance. While tackling important issues related to hyperparameter optimization, such as striking a balance between exploration and exploitation, the method achieves a noteworthy accuracy of 0.996853. These findings demonstrate how the suggested approach may strengthen network security by using more accurate predictive modeling, opening the door for scalable and effective IoT systems in progressively complex settings.