Multi-Agent RL (MARL)

The book so far has covered most of the popular RL approaches, including the state-of-the-art PPO with its application in Large Language Models for RLHF fine-tuning. You may have noticed that the focus has always been on only one agent in the environment that learns to act optimally using RL training algorithms. However, there is a whole range of settings with more than one agent. These agents in the environment—either individually or in a collaborative manner—try to achieve some goal. A setup involving multiple agents in the same environment is the focus of this chapter. It is called Multi-Agent RL (MARL). MARL is a very fascinating and vast topic. To do proper justice to the topic would require a complete book of its own. This chapter introduces the key topics with simple examples and finally concludes the chapter with an example of applying learning in MARL setup for a simple environment. I will introduce various terms and concepts and make some assertions without getting into depth. The key purpose is to introduce MARL with the expectation that readers interested in MARL will refer to other detailed MARL-related books.