Function Approximation and Deep Learning

The previous three chapters looked at various approaches to planning and control—first at the dynamic programming (DP), then at the Monte Carlo approach (MC), and finally at the temporal difference (TD) approach. In all these approaches, you saw problems where the state space and actions were discrete. Only in the previous chapter, toward the end, did I talk about Q-learning in a continuous state space. You discretized the state values using an arbitrary approach and trained a learning model. This chapter extends that approach by talking about the theoretical foundations of approximation and how it impacts the setup for reinforcement learning. It will then look at the various approaches to approximating values, first with a linear approach that has a good theoretical foundation and then with a nonlinear approach with neural networks. This aspect of combining deep learning with reinforcement learning is the most exciting development and has moved reinforcement learning algorithms to scale.