Human Alignment
摘要
This chapter delves into the critical issue of human alignment in large language models (LLMs), emphasizing the need to align their behavior with human values, intentions, and social norms to mitigate risks such as bias, offensive content, and factual inaccuracies. It outlines the background and criteria for human alignment, focusing on three key principles: helpfulness, honesty, and harmlessness. The primary technique for achieving alignment is reinforcement learning from human feedback (RLHF), which involves collecting human preferences, training a reward model, and fine-tuning the language model using reinforcement learning algorithms like proximal policy optimization (PPO). This chapter also explores non-reinforcement learning methods, with a special focus on direct preference optimization (DPO), which simplify the alignment process by eliminating the need for reinforcement learning. Additionally, it discusses advanced RLHF methods, including process-supervised alignment and reinforcement learning from AI feedback. Finally, the chapter highlights the challenges of hallucination in LLMs, categorizing them into different types, and proposes mitigation strategies such as alignment training, retrieval-augmented generation, and improved prompt design. In general, the chapter provides a comprehensive overview of human alignment techniques and practical approaches to improving the reliability and ethical behavior of LLMs.