Soft Actor-Critic Based Deep Reinforcement Learning Method for Production Optimization

Production optimization is a crucial technology for efficient development of water-driven reservoirs. By adjusting the injection and production rate of oil and water wells in a reservoir block, the optimal production solution can be provided for the field to maximize the economic benefits while minimizing the costs. In this paper, a soft actor-critic (SAC) based reinforcement learning offline production optimization method are proposed, which models the production optimization problem as a Markov sequence decision process. Specifically, the deep reinforcement learning (DRL) agents aimed at maximizing the economic efficiency. The agent updated the policy model incrementally using the data obtained by interactive sampling with the environment to accelerate the convergence of the optimization process. In addition, to achieve offline optimization, a state transfer model is constructed that captures the dynamics of the reservoir under time-varying well control conditions using historical regulation experience. In the offline deployment stage of the cloud platform, the trained policy network and state transition network are utilized. In this way, the well control scheme for multiple future time steps can be calculated using only the current state of the reservoir. Reservoir instances show that this method is highly efficient and can provide optimized solutions within seconds, and the optimization performance is also remarkable. With the good effect of water control and oil increment, the target model can achieve higher net present value (NPV). The proposed offline method, which embedding control strategies into the model and utilizing a state transition model to capture the dynamics of the system, offers a novel approach to intelligent production optimization. By enabling offline optimization deployment on a cloud platform, this approach provides a practical solution to meet the demand for intelligent oilfield construction.