Deep learning has achieved remarkable success in recent decades, which is largely attributed to the sheer amount of data. However, the constraints of computing resources result in significant time and memory required to train deep learning models on large-scale data. To address this, dataset distillation has attracted increasing research attention in the community. Dataset distillation aims to derive informative small datasets that distill the most important knowledge from large datasets. These small curated datasets can replace the original large datasets to benefit scenarios such as efficient model training and inference, neural architecture search, and privacy protection. In this chapter, we aim to provide a comprehensive summary of dataset distillation. We first introduce the dataset distillation problem and group existing methods into four categories: performance matching, gradient matching, trajectory matching, and distribution matching. Specifically, we provide a formal task definition and establish a unified framework for these methods. Then, we systematically review the technical details of methods in each category and discuss the application of dataset distillation across various data modalities such as image, graph, text, and time-series data. Finally, we provide concluding remarks on the pitfalls and prospects of dataset distillation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dataset Distillation for Enterprise Applications

  • Juntong Ni,
  • Shengbo Gong,
  • Wei Jin

摘要

Deep learning has achieved remarkable success in recent decades, which is largely attributed to the sheer amount of data. However, the constraints of computing resources result in significant time and memory required to train deep learning models on large-scale data. To address this, dataset distillation has attracted increasing research attention in the community. Dataset distillation aims to derive informative small datasets that distill the most important knowledge from large datasets. These small curated datasets can replace the original large datasets to benefit scenarios such as efficient model training and inference, neural architecture search, and privacy protection. In this chapter, we aim to provide a comprehensive summary of dataset distillation. We first introduce the dataset distillation problem and group existing methods into four categories: performance matching, gradient matching, trajectory matching, and distribution matching. Specifically, we provide a formal task definition and establish a unified framework for these methods. Then, we systematically review the technical details of methods in each category and discuss the application of dataset distillation across various data modalities such as image, graph, text, and time-series data. Finally, we provide concluding remarks on the pitfalls and prospects of dataset distillation.