Accelerated Approximation of Bellman Equation Solutions: Agent Policy Optimization With a Feedforward Neural Network
摘要
Solving recursive equations through iteration can be a computationally expensive endeavour, and the time required to reach an optimal solution delays the progress of any dependent processes. To address this issue for a specific use case of decision-making in an agent-based model, a method of replacing the iterative function used in said model, a Bellman equation, with a feedforward multilayer perceptron was developed. A hyperparameter grid search was performed to determine the combination of architecture, learning rate, and batch size which produced results deviating the least from those of the original iterative method. With the resulting neural network, accepting four inputs and yielding two outputs, the time required to compute outputs scales sublinearly with the number of agents. Excluding training time, for a set of 1,000 agents, the selected neural network produces output at over 66,000 times the speed of the original function. It achieves this acceleration while maintaining a 99.3% accuracy in adaptation strategy selection and 0.10 mean absolute error in consumption, leading to its ready adoption as an acceptable replacement for the original method.