Pre-training forms the foundational stage of large language models (LLMs), primarily built on the objective of predicting the next token. While pre-training is highly effective, models immediately after this stage often struggle to adapt effectively to task-solving scenarios. To enhance their task-solving capabilities, instruction tuning or supervised fine-tuning is employed, teaching LLMs to solve tasks described in natural language instructions. This approach has proven effective in eliciting and unlocking the task-solving abilities of LLMs. This chapter provides a comprehensive introduction to instruction fine-tuning methods, covering various aspects from dataset construction to training strategies. We begin by exploring methods for constructing instruction datasets, including the adaptation of existing datasets and the synthesis of high-quality data. We also discuss key factors in instruction data construction and their impact on model performance. Next, we examine typical training strategies for improving instruction tuning, such as optimization settings and data organization. Additionally, we provide an overview of parameter-efficient fine-tuning methods, particularly in low-resource scenarios, with a focus on the widely used lightweight fine-tuning technique, LoRA (Low-Rank Adaptation). We also briefly introduce several variants of parameter-efficient fine-tuning. Finally, we present code implementations and conduct an empirical study to analyze the effects of different instruction datasets.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Instruction Tuning

  • Wayne Xin Zhao,
  • Kun Zhou,
  • Junyi Li,
  • Tianyi Tang,
  • Ji-Rong Wen

摘要

Pre-training forms the foundational stage of large language models (LLMs), primarily built on the objective of predicting the next token. While pre-training is highly effective, models immediately after this stage often struggle to adapt effectively to task-solving scenarios. To enhance their task-solving capabilities, instruction tuning or supervised fine-tuning is employed, teaching LLMs to solve tasks described in natural language instructions. This approach has proven effective in eliciting and unlocking the task-solving abilities of LLMs. This chapter provides a comprehensive introduction to instruction fine-tuning methods, covering various aspects from dataset construction to training strategies. We begin by exploring methods for constructing instruction datasets, including the adaptation of existing datasets and the synthesis of high-quality data. We also discuss key factors in instruction data construction and their impact on model performance. Next, we examine typical training strategies for improving instruction tuning, such as optimization settings and data organization. Additionally, we provide an overview of parameter-efficient fine-tuning methods, particularly in low-resource scenarios, with a focus on the widely used lightweight fine-tuning technique, LoRA (Low-Rank Adaptation). We also briefly introduce several variants of parameter-efficient fine-tuning. Finally, we present code implementations and conduct an empirical study to analyze the effects of different instruction datasets.