<p>The increased adoption of Internet of Things (IoT) devices across different applications exposes them to many cyber-attacks, emphasizing the importance of securing these devices. However, the resource limitation of IoT devices makes implementing traditional security mechanisms challenging. The use of Machine Learning (ML) and Deep Learning (DL) techniques for attack detection in IoT shows impressive results but requires some mechanism to reduce its complexity. Feature selection techniques are widely used to reduce the number of selected features. This work introduces novel feature selection algorithms based on improving the Firefly Algorithm (FA). The first feature selection algorithm is based on enhancing FA using a new multi-objective fitness function based on the sum of features’ chi-squared scores, the ratio of the selected features to evaluate the selected features and a new population diversity method based on chi-squared and cosine similarity. The second feature selection technique is based on hybridizing FA and Harris Hawk Optimization (HHO) algorithms. Furthermore, an optimized Random Forest (RF) classifier is used to classify the traffic. The proposed work was tested using the IoTID20 dataset and shows the success of the new multi-objective fitness function and population diversity method in reducing the selected features while increasing the convergence speed. Similarly, the new hybrid feature selection algorithm shows impressive results in selecting the most relevant feature for attack detection, making the ML model reliable in detecting true positives, reducing false positives, and selecting fewer features. In addition, the proposed feature selection algorithms were tested using the UNSW-NB15 dataset to evaluate their performance across different types of network traffic and attacks.</p>