Value Function Evaluation with Data Augmentation for Offline Reinforcement Learning

Offline reinforcement learning, also known as batch reinforcement learning, aims to utilize pre-collected offline datasets for training without further interaction with the environment. It is suitable for training in complex real-world environments. However, offline datasets often have fixed sample distributions and highly correlated samples, which may lead to biased or overly conservative learned policies. Additionally, the data collection process is typically slow and potentially risky. To overcome these challenges, we propose a Value-Function-Evaluation with Data Augmentation (VEDA). The goal is to improve the robustness and generalization ability of offline reinforcement learning algorithms, mainly based on the potential distribution of state learning datasets and generate data to improve generalization ability. Subsequently, in order to ensure the quality of the generated data, a value function evaluation module is adopted for state evaluation. By combining with several existing standard baselines, experimental results demonstrate that our approach significantly improves performance and achieves better model stability compared to the original algorithm.