Network attacks via zero-days continue to be a challenge in cybersecurity due to their unknown nature, making both signature-based approaches and supervised learning ineffective because of a lack of information needed for classification. On the other hand, anomaly-based models have been proposed to overcome this issue by modeling normal behavior and then detecting anomalies. However, these approaches run into the problem that the penalty of the false positive is rigid for every type of attack and does not vary based on the number of occurrences. In this regard, our solution is Adaptive- \(\lambda \) PSO-IF. Isolation Forest is combined with Particle Swarm Optimization (PSO) algorithm so that feature selection and anomaly threshold optimization can be conducted simultaneously for each type of attacks without requiring any labeled attack samples during the optimization process. The novelty of our approach is the rarity-aware regularization of the \(\lambda \) -coefficient, an approach that, to our knowledge, has not been previously explored in PSO-driven anomaly detection frameworks and directly addresses the systematic underdetection of rare zero-day families. Experiments on the UNSW-NB15 dataset under a strict Leave-One-Family-Out zero-day evaluation protocol demonstrate that Adaptive- \(\lambda \) PSO-IF achieves a mean objective score of 0.176 ± 0.266 across five independent random seeds, outperforming fixed- \(\lambda \) PSO-IF by up to 50.9% on rare attack families and standard Isolation Forest by 104.6% in mean objective score, while maintaining competitive performance with LOF at significantly lower inference complexity. The gains are most pronounced in rare attack families, with F1 improvements of 15.0% on Analysis, 20.8% on Reconnaissance, and 50.9% on Backdoor over the fixed- \(\lambda \) baseline. The online inference complexity of the deployed system remains \(O(t\cdot log \psi ),\) identical to standard Isolation Forest, confirming that the performance gains carry no additional cost at detection time.
