Basic Optimization Concepts

Optimization aims at finding a unique point, a unique state, or a unique function (a sequence of points or states) that is associated with the minimum cost or maximum benefit with respect to a certain criterion, subject to constraints. Function optimization is largely based on calculus developed by Sir Isaac Newton and Gottfried Wilhelm Leibniz. The most comprehensible optimization process is finding the minima or maxima of a single variable function \(y=f(x)\) by setting the first derivative \(\frac{dy}{dx}=0\) , which can be extended to multi-variable functions \(y=f(x_1,x_2,\ldots {x}_n)\) by setting all partial derivatives to zero \(\frac{\partial y}{\partial x_i}=0; i=1,2,\ldots {n}\) . More often, the functions to be optimized are costs that can be modeled in the quadratic form \(f(\textbf{x})=\textbf{x}^T\textbf{Q} \textbf{x}\) , which allows wider applicability of mathematical tools in the optimization process, hence quadratic optimization.