Zero-day malware presents a significant cybersecurity threat by exploiting unknown vulnerabilities and evading traditional signature-based detection methods. This study proposes a behavior-based malware detection framework that leverages machine learning to identify anomalies in system and network activity. The framework captures behavioral indicators such as CPU and memory usage, network traffic patterns, file system alterations, and process activity, which are commonly linked to zero-day malware attacks. The methodology includes thorough data preprocessing, feature engineering, dimensionality reduction, and the application of three ensemble learning models: Random Forest, XGBoost, and Light Gradient Boosting Machine (LightGBM). These models were evaluated using k-fold cross-validation to ensure robustness and reliability. All three achieved high detection performance, with accuracy scores exceeding 99% and ROC-AUC values close to 1.0. Among them, LightGBM demonstrated the highest accuracy, with XGBoost and Random Forest following closely. The framework emphasizes early detection and aims to minimize false positives, making it viable for real-world integration. While the results are promising, scalability for real-time deployment and adaptability to the evolving nature of malware remain challenges. Future enhancements will focus on incorporating deep learning models for sequential behavioral analysis, testing on real-world datasets, and optimizing the approach for large-scale environments. This research supports the potential of behavior analytics and machine learning in effectively identifying zero-day malware and contributes toward the development of adaptive, intelligent cybersecurity solutions.