This paper proposes a gradient descent reinforcement learning algorithm with a non-convex constraint operator to solve the optimal containment control problem for linear multi-agent systems with unknown dynamics and input constraints. By transforming the control problem into a dynamic design based on local containment error, we employ the Q-function from reinforcement learning and design a policy iteration method which based on gradient descent and non-convex constraint operator. We analyze its convergence and use an Actor-Critic neural network framework to approximate the optimal Q-function and control policy. Simulation results demonstrate the effectiveness of the proposed approach.