In this paper, we present a novel Gradient Penalty Sine Cosine Algorithm (GP-SCA) and its implications in optimizing complex functions. The GP-SCA introduces a regularization term to penalize large gradients in order to improve the exploration and exploitation capabilities of the algorithm. The regularization term is computed by calculating the norm of the gradients and adding it to the objective function. This penalty encourages the algorithm to explore different regions of the search space and prevents it from getting stuck in local optima. The GP-SCA algorithm achieves better performance by penalizing large gradients. The computation of the regularization term involves taking the square root of the sum of squared gradients and multiplying it by a user-defined coefficient. The user-defined coefficient allows for adjusting the impact of the penalty on the algorithm’s behavior, striking a balance between exploration and exploitation. A higher coefficient value will increase the penalty and encourage more exploration, while a lower value will prioritize exploitation and exploitation. The choice of the coefficient value depends on the specific problem and the desired balance between exploration and exploitation. The source code will be available after the acceptance of the paper at: https://github.com/Alqushaibi/GP-SCA.git .