Dataset distillation aims to create compact synthetic datasets that retain the generalization properties of real datasets. This study employs dataset distillation by matching training trajectories (DDMTT), a novel approach that utilizes expert trajectories (precomputed sequences of network parameters trained on the full dataset) to guide the distillation process. Experiments with the extremely increased number of images per class (IPC) were conducted using standard datasets such as CIFAR-10 and CIFAR-100, as well as medical benchmarking datasets from MedMNIST. The proposed method of generative data augmentation by dataset distillation (GDADD) demonstrated that, for CIFAR datasets, their smaller distilled versions containing 40,000 images achieved higher validation accuracy than the full datasets with 50,000 images, surpassing the original dataset’s performance by 3.1% for CIFAR-10 and 2.9% for CIFAR-100. For the considered MedMNIST datasets (PathMNIST, DermaMNIST, RetinaMNIST), some distilled datasets (PathMNIST) exceeded the performance of models trained on full datasets, confirming the method’s robustness across different domains and demonstrating the better results for the well balanced datasets.