Complete stability analysis of iterative adaptive critic designs with discounted cost

In this paper, the stability of nonlinear systems under infinite-horizon discounted optimal control via adaptive dynamic programming method is analyzed. First, considering the adoption of function approximators during value iteration (VI), the iterative value function and control policy are shown to be continuous. Then, based on a verifiable condition on the approximation errors caused by the critic network, it is proved that the approximate value functions are bounded and positive definite. Further in the stability analysis, a stability condition as the termination criterion of approximate VI (AVI) is developed, which guarantees that the control policy derived from the obtained critic network makes the controlled system \(\mathcal{K}\mathcal{L}\) $\mathcal{K}\mathcal{L}$ -stable. Also, an upper bound function of the approximation errors caused by the action network is derived for ensuring that the system controlled by the trained action network remains \(\mathcal{K}\mathcal{L}\) $\mathcal{K}\mathcal{L}$ -stable. The \(\mathcal{K}\mathcal{L}\) $\mathcal{K}\mathcal{L}$ -stability of the closed-loop system is established by using the approximate value function to act as the Lyapunov function and estimate the region of attraction. Finally, the present theoretical results are applied to the simulation studies of the spacecraft rendezvous.