This paper introduced a robust anomaly detection framework utilizing the BPF-extended tracking honeypot (BETH) dataset, designed to benchmark cybersecurity applications with real-world kernel-process and network logs. Comprehensive preprocessing, including feature engineering, numerical scaling, and ADASYN for class imbalance handling, prepared the dataset for effective machine learning application. Multiple classifiers, such as LightGBM, CatBoost, support vector machines (SVM), K-nearest neighbors (KNN), and logistic regression, were evaluated using precision, recall, F1-score, and accuracy metrics. CatBoost emerged as the top-performing standalone model with high precision and recall. An artificial neural network (ANN) was subsequently introduced to complement traditional models, leveraging its capability to detect nonlinear relationships. The hybrid approach combined CatBoost’s robust feature extraction with ANN’s pattern recognition strengths, further refining anomaly detection. Ensemble techniques, such as stacking and voting classifiers, were also explored to aggregate predictions from multiple models. The hybrid model demonstrated exceptional performance, achieving an accuracy of 99.0% and an F1-score of 98.6%, outperforming standalone models. This research highlights the synergy of machine learning, feature engineering, and hybrid ensemble strategies, providing an effective solution for anomaly detection in complex cybersecurity scenarios.