Advanced persistent threats (APTs) are becoming more advanced, requiring the creation of strong cybersecurity measures. This study suggests a new method to address advanced persistent threats (APT) by utilizing machine learning, particularly by implementing the “Non-dominated Sorting Genetic Algorithm” II (NSGA-II). The proposed model attains a true-positive rate (TPR) of 98.62% and a precision of 97.59%. Extensive experiments were conducted using a diverse dataset containing APT and benign data samples to assess the effectiveness of NSGA-II compared to other popular machine learning algorithms such as random forest, isolation forest, and LightGBM. The NSGA-II model consistently surpassed other algorithms in terms of true-positive rate (TPR) and precision, demonstrating its superiority in detecting and mitigating advanced persistent threats (APT). The study explores the structure and functionality of NSGA-II, highlighting its ability to enhance the efficiency of machine learning models for APT detection. The algorithm's efficiency in exploring the solution space, balancing multiple objectives, and identifying nondominated solutions greatly contributes to its success in dealing with the complex nature of APT. This study emphasizes the practical implications of using NSGA-II for APT detection in real-world situations. The model shows a high level of accuracy in differentiating malicious activities from normal network behavior, offering an effective defense against APT. The suggested method employing NSGA-II for APT detection provides a promising solution to the changing cybersecurity threat environment. The high true-positive rate (TPR) and precision rates of NSGA-II exceed those of random forest, isolation forest, and LightGBM, demonstrating its potential as a strong and effective tool in addressing advanced persistent threats (APT) and improving cybersecurity resilience.