Constrained reinforcement learning with statewise projection: a control barrier function approach

Safety is a critical issue for reinforcement learning (RL), as it may be risky for some actual applications if the learning process involves unsafe exploration. Instead of formulating constraints as expectation-based in constrained RL, considering statewise safety in constrained RL is more meaningful. This work aims to address the issue of safe projection in RL by introducing a control barrier function that inherently learns a safe policy through a set certificate. We seek to analyze some theoretical properties of safe projection in the learning process, including convergence and performance bound, and extend the discussion into ensembles and guided controllers. Moreover, we approach analytical solutions for deterministic and stochastic system dynamics. Experimental results in different tasks show that the proposed method achieves better effects in terms of both performance and safety.