This paper presents a novel approach to optimize the non-convex loss functions used in neural networks using powerful parallel computing language Julia. There are two types of loss functions- convex, where there is only one local maximum, and non-convex loss function where there are multiple local minima. To get most optimal results in model evaluation, reaching the global minima is preferred. This paper proposes a method to reach the global minimum and thus optimize the non-convex loss functions by using the optimization algorithms in the Julia packages. In this paper, we have used the IPOPT (Interior Point Optimizer) package to find the global minimum in loss functions so that the neural network’s accuracy increases and we have maximum convergence between model output and target value. This paper also provides detail comparison between other loss function optimization algorithms and the Interior Point Method, which we have adopted for this research paper.