Integrating Deep Reinforcement Learning into Agent-Based Models for Predicting Farmer Adaptation Under Policy and Environmental Variability

In designing models of integrated socio-environmental systems, human behaviors in Agent Based Models (ABMs) have been generally modeled with variety of simple to complex behavioral theories predicting behavioral change, such as bounded rational theory, rational choice theory, value-belief-norm theory, and the theory of planned behavior. In an attempt to improve the representation of intelligent agents in ABMs, whose cognition is embedded with memory and foresight, and who are responsive to dynamically evolving feedbacks between human and environmental systems, this paper presents a novel ABM, in which intelligent agents are embedded with deep reinforcement learning derived behavioral rules. A Deep Reinforcement Learning enabled ABM (DRL-ABM) is calibrated in this paper to simulate the behavior of 480 farmer agents engaged in joint production of food and water pollution in Missisquoi Watershed of trans-boundary Lake Champlain Basin. The DRL-ABM specifically focuses on predicting the adoption of adaptation actions by simulated farmers that reduce the adverse impact of agricultural production on water quality in the face of global climate change induced shifting frequencies and intensities of extreme events (e.g., floods and droughts). State-Action-Reward pairs exploiting the conflict between the agent goal to maximize profit while minimizing their impact on environment are configured in this DRL-ABM. Experimental simulations manipulate incentives (taxes and subsidies) endogenously offered by policy agents to farmer agents, frequencies of extreme events and the memory of farmer agents to these extreme events that can cause direct damage to their agricultural production in the absence of adaptation actions. We also test variations in learning related hyper-parameters for homogeneous and heterogeneous configurations of intelligent agents. From the experimental simulations, we discover that adoption of adaptation actions is highly sensitive to the thresholds of incentive change used by policy agents, efficacy of adaption action in improving water quality and reducing damages from extreme events, and the memory span of agents. Extended applications of DRL-ABM in other integrated socio-environmental systems can be tested in future research to improve the integration of Artificial Intelligence with Intelligent Systems and address pressing socio-environmental sustainability challenges.