Smaller Can Be Better: Efficient Data Selection for Pre-training Models

To reduce the resource consumption for pre-training models on large-scale corpora, existing research has primarily focused on refining network architectures and training methods. Nevertheless, the concomitant cost tends to be expensive. This paper proposes to enhance pre-training efficiency by judiciously selecting a subset of the entire corpus. To this end, we introduce a sentence-level domain classifier to quantify the relevance of each pre-training sample to target downstream tasks. This fine-grained classifier enables the selection of the most relevant samples, facilitating efficient domain transfer between pre-training and fine-tuning stages. To improve selection efficiency, we extend the classifier to the coarse-grained level, where consecutive texts from the same source are organized into blocks and selected accordingly. We further explore the multi-grained data selection from coarse-grained to fine-grained in a hierarchical manner, providing a smoothing mechanism to balance efficiency and accuracy. Extensive experiments across diverse corpora and tasks verify the efficiency and effectiveness of our data selection approach.