<p>As drilling activity intensifies across the oil and gas industry, there is a growing need to enhance the efficiency, reliability, and safety of drilling operations. Traditional methods are often constrained by high costs, non-productive time (NPT), and operational risks. These challenges are exacerbated by inconsistencies in real-time decision-making and uncertainties in subsurface rock properties. Existing uncertainty modeling approaches, such as analog models, geostatistical simulations, and Bayesian expert systems, offer limited scope and rely heavily on operator expertise and simplistic forward modeling techniques. This paper introduces DRUID (deep reinforcement learning used to improve drilling), a real-time optimal control agent that automates and optimizes drilling parameters using deep reinforcement learning. DRUID employs proximal policy optimization (PPO) to model subsurface conditions and dynamically recommend key control parameters, including rate of penetration (ROP), weight on bit (WOB), rotations per minute (RPM), and bit inclination. By integrating expert domain knowledge with static reservoir data, DRUID supports geosteerers in making informed decisions while accounting for geological variability. A comparative study with a value-function approximation method, deep Q-network (DQN), demonstrates that the PPO-based agent achieves superior drilling performance, benefiting from trust-region updates that ensure stable training and controlled exploration across the state–action space. To further understand the agent’s learning behavior, a reward sensitivity analysis was conducted to evaluate the impact of reward shaping on convergence and final policy quality. The analysis revealed that the structure and scaling of intermediate and terminal rewards significantly influence the agent’s ability to balance exploration and exploitation, underscoring the importance of carefully designed reward functions in reinforcement learning-based control systems. The effectiveness of DRUID is further validated through a real-world case study using the Volve field dataset. Applied to the F-12 well, DRUID achieved a simulated rotary drilling time of 38.7 h, closely approximating the robustly optimized benchmark of 36.7 h sand significantly shorter than the actual field drilling time of 139.2 h. These results highlight DRUID’s ability to generalize from synthetic training environments to real-world data, capturing the complex relationships between rock properties and drilling dynamics. By combining reinforcement learning with expert-informed modeling and real-time optimization, DRUID represents a significant advancement in intelligent drilling automation. Its deployment at rig sites and drilling support centers offers a scalable, data-driven solution for reducing NPT, improving operational consistency, and enabling end-to-end automation across both rotary and non-rotary drilling phases.</p>

Application of Proximal Policy Optimization for Directional Drilling

期刊论文
发表日期：

2026年5月9日

查看全文

Kshitij Dawar,
Joshua Scott Hoskinson

摘要

As drilling activity intensifies across the oil and gas industry, there is a growing need to enhance the efficiency, reliability, and safety of drilling operations. Traditional methods are often constrained by high costs, non-productive time (NPT), and operational risks. These challenges are exacerbated by inconsistencies in real-time decision-making and uncertainties in subsurface rock properties. Existing uncertainty modeling approaches, such as analog models, geostatistical simulations, and Bayesian expert systems, offer limited scope and rely heavily on operator expertise and simplistic forward modeling techniques. This paper introduces DRUID (deep reinforcement learning used to improve drilling), a real-time optimal control agent that automates and optimizes drilling parameters using deep reinforcement learning. DRUID employs proximal policy optimization (PPO) to model subsurface conditions and dynamically recommend key control parameters, including rate of penetration (ROP), weight on bit (WOB), rotations per minute (RPM), and bit inclination. By integrating expert domain knowledge with static reservoir data, DRUID supports geosteerers in making informed decisions while accounting for geological variability. A comparative study with a value-function approximation method, deep Q-network (DQN), demonstrates that the PPO-based agent achieves superior drilling performance, benefiting from trust-region updates that ensure stable training and controlled exploration across the state–action space. To further understand the agent’s learning behavior, a reward sensitivity analysis was conducted to evaluate the impact of reward shaping on convergence and final policy quality. The analysis revealed that the structure and scaling of intermediate and terminal rewards significantly influence the agent’s ability to balance exploration and exploitation, underscoring the importance of carefully designed reward functions in reinforcement learning-based control systems. The effectiveness of DRUID is further validated through a real-world case study using the Volve field dataset. Applied to the F-12 well, DRUID achieved a simulated rotary drilling time of 38.7 h, closely approximating the robustly optimized benchmark of 36.7 h sand significantly shorter than the actual field drilling time of 139.2 h. These results highlight DRUID’s ability to generalize from synthetic training environments to real-world data, capturing the complex relationships between rock properties and drilling dynamics. By combining reinforcement learning with expert-informed modeling and real-time optimization, DRUID represents a significant advancement in intelligent drilling automation. Its deployment at rig sites and drilling support centers offers a scalable, data-driven solution for reducing NPT, improving operational consistency, and enabling end-to-end automation across both rotary and non-rotary drilling phases.