Proximal Policy Optimization (PPO) and RLHF

This is an exciting chapter. It starts from the foundations and builds toward one of the most exciting uses of RL. Have you have used ChatGPT or another Large Language Model (LLM) and found it amazing how these models seem to follow your prompts and complete a task that you describe in English? Apart from the machinery of generative AI and transformers-driven architecture, RL also plays a very important role. Proximal Policy Optimization (PPO) using human annotated (or machine annotated) preferences over pairs of sentences is used as a reward model to fine-tune LLMs to follow the human preferences. This makes LLMs safer and better. This chapter starts with the basics, building toward a deeper understanding of PPO, which even after so many years is still the state-of-the-art policy-based optimization technique in RL. This is followed by a quick overview of LLMs—the architecture, the training process, and the overall LLM ecosystem. The chapter walks through a complete demo of RLHF tuning on a LLM using the state-of-the-art approaches.