Model-Based Approaches

RL agents learn optimal actions for any given state, which maximizes the total reward the agent gets by following an optimal set of actions as the agent transitions from one state to another. Equations 2-19 and 2-21 clearly indicate that V(s) and Q(s, a) depend on two components, the transition dynamics and the next state/state-action values. To lay the foundations of RL learning, this chapter starts with the simplest setup—one in which the transition dynamics Pr{ St + 1 = s′, Rt + 1 = r | St = s, At = a} are known. It also assumes that the number of states and actions possible in a given state form a closed, small set of discrete values. Such a simplifying assumption will help you develop the foundations of RL learning algorithms. The agents in this chapter use the transition knowledge to “plan” a policy that maximizes the cumulative return of the state value vπ(s) or qπ(s, a). All these algorithms are based on dynamic programming, which allows you to break the problem into smaller subproblems and use the recursive relationship of Bellman equations explained in Chapter 2 . Next, I will extend the approach to come up with a more general framework of policy improvement