<p>The rapid expansion of computer networks and widespread use of network applications globally have significantly increased cyberattack risks. This increased connectivity has exposed network vulnerabilities, enabling hackers to infiltrate systems. Overcoming such threats, an Intrusion Detection System (IDS) helps protect data confidentiality and integrity, but conventional approaches struggle with unpredictable attacks. Current IDS inefficiencies are largely due to outdated datasets, especially with rare attack types. These datasets are also highly class-unbalanced, deteriorating the model’s performance. This study aims to enhance intrusion detection efficiency through a novel three-phase anomaly-based network IDS. The first phase preprocesses the data, addresses class imbalance with Synthetic Minority Over-sampling Technique (SMOTE), and applies dimensionality reduction using Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). In the second phase, four well-known machine learning classifiers, SVM, Random Forest, K-Nearest Neighbor, and Decision Tree, are used to classify benign and attack instances. Different network attacks are then detected in the final phase using the proposed modified Picture Fuzzy Clustering technique, mP_<i>ic</i>FC. Unlike outdated datasets like KDD Cup’99, this study uses the CSE-CIC-IDS2018 dataset, providing a robust benchmark that includes modern attack types. Statistical analysis is performed using <i>k</i>-fold and Leave-one-out cross-validation on both original and balanced datasets. The experimental results demonstrate that on the SMOTE-balanced dataset, the PCA-Random Forest model achieved the highest accuracy of 99.94%, and the proposed mP_<i>ic</i>FC achieved a detection rate of 89.98%, surpassing state-of-the-art clustering methods. These results highlight the potential of the proposed IDS to improve real-time detection and enhance cybersecurity.</p>