Digital systems are seriously threatened by the quick innovation of the harmful software (malware) industry, with attackers leveraging advanced techniques to evade detection and cause widespread damage, including data corruption, theft, and disruption of critical services. Traditional detection techniques often find it difficult to keep up with the increasing amount and diversity of malware, highlighting the urgent need for scalable and effective solutions. Effective malware detection is essential not only to protect sensitive information but also to ensure the security and reliability of modern digital infrastructures. Machine learning has emerged as a powerful approach to tackle the challenges of malware detection, offering the ability to analyze vast data sets and adapt to evolving threat landscapes. Due to their excellent accuracy and scalability, gradient boosting algorithms such as LightGBM, XGBoost, and CatBoost have demonstrated tremendous promise for classification tasks among several machine learning techniques. However, less-than-ideal hyperparameter setups frequently reduce the efficacy of these models. By effectively examining intricate hyperparameter spaces, advanced optimization methods like Particle Swarm Optimization (PSO) and frameworks like Optuna have shown great promise in overcoming these constraints. This study systematically evaluates the application of advanced optimization techniques to enhance gradient-boosting-based malware detection systems. Starting with a LightGBM model that achieves an accuracy of 81%, the application of PSO improves the performance to 83%. Further optimization with Optuna results in a remarkable accuracy of 98%, accompanied by an F1 score of 0.98, a precision of 0.97 and a recall of 0.98. These results highlight the significance of hyperparameter tuning in maximizing model performance. Optuna’s pruning process boosts efficiency by discarding unpromising trials early, reducing computation time.