Dataset Distillation for Enterprise Applications
摘要
Deep learning has achieved remarkable success in recent decades, which is largely attributed to the sheer amount of data. However, the constraints of computing resources result in significant time and memory required to train deep learning models on large-scale data. To address this, dataset distillation has attracted increasing research attention in the community. Dataset distillation aims to derive informative small datasets that distill the most important knowledge from large datasets. These small curated datasets can replace the original large datasets to benefit scenarios such as efficient model training and inference, neural architecture search, and privacy protection. In this chapter, we aim to provide a comprehensive summary of dataset distillation. We first introduce the dataset distillation problem and group existing methods into four categories: performance matching, gradient matching, trajectory matching, and distribution matching. Specifically, we provide a formal task definition and establish a unified framework for these methods. Then, we systematically review the technical details of methods in each category and discuss the application of dataset distillation across various data modalities such as image, graph, text, and time-series data. Finally, we provide concluding remarks on the pitfalls and prospects of dataset distillation.