Model-Free Approaches

The previous chapter looked at dynamic programming, used when you know the model dynamics p(s’, r| s, a), and the knowledge is used to “plan” the optimal actions. This is also known as the planning problem. This chapter shifts focus and looks at learning problems—that is, setups where the model dynamics (aka transition dynamics) are not known. You will learn to calculate/learn state values and state-action values by sampling—that is, collecting experience by following some policy in the real world or running the agent through a policy in simulation. There is another class of problems where the model-free approach is more applicable. In some problems, it is easier to sample than to calculate the transition dynamics, such as a problem of finding the best policy to play a game like blackjack. There are many combinations to reach a score that depend on the cards seen so far and the cards still in the deck. It is almost impossible to calculate the exact transition probability from one state to another, but it is easy to sample states from an environment. To summarize, you use model-free methods when either you do not know the model dynamics or you know the model, but it is much more practical to sample than to calculate the transition dynamics.