Optimization on Riemannian manifolds has attracted the attention of many in the machine learning community due to its potential in solving constrained optimization problems. However, high cost of geodesic-based retraction algorithms often limits the applications of Riemannian optimization algorithm in practical problems that are intensely resource consuming. We focus on the positive orthant constrained optimization problems, a special type of constraint that are present in many application areas from evolutionary game theory to non-negative matrix factorization. We propose an accelerated natural gradient descent algorithm modified from evolutionary game dynamics and show that our algorithm converges to second-order stationary points. Our theoretical analysis and experiments verify that a geodesic free acceleration scheme has the potential to balance the convergence guarantee and computational efficiency in constrained non-convex optimization problems like non-negative matrix factorization.