Diverse and Task-Specific Data Selection for Instruction Tuning

Data selection for instruction tuning is an important and challenging task, which selects an effective subset within a given budget to maximize instruction-following performance. Recent state-of-the-art data selection studies typically consider both data diversity and data quality. However, for diversity, existing methods usually rely on simple cluster algorithms to get optimal groups, which is difficult. Moreover, they treat all tasks equally and use a unified selection strategy across task categories, overlooking inter-task differences. To address these issues, we propose a two-stage framework called Diverse and Task-Specific Data Selection (DaTS). In the first stage, inspired by the intrinsic hierarchical nature of data, we introduce a coarse-to-fine clustering strategy to alleviate the grouping difficulty. Coarse clustering captures category-level diversity, while fine clustering within each coarse cluster captures the semantic diversity of subcategories. In the second stage, DaTS selects data from each fine cluster by jointly considering sample-level quality, inter-sample differences, and model-perceived difficulty in a task-specific manner, enabling each task to prioritize its key considerations. Experiments on three data pools and multiple benchmarks show that using just 5% instruction data selected by DaTS yields an average improvement of 2.1% over the best baseline and 2.4% over full-data training.