Knowledge distillation (KD) is a technique used to transfer knowledge from a complex teacher model to a simpler student model. The core idea of KD is that the learned mapping of a complex model is superior to a one-hot label, the student model can achieve higher accuracy. However, it is important to consider whether the student should blindly follow the teacher as an oracle. It has been found that not all of the teacher’s knowledge is useful to the student. In fact, some of the teacher’s logits output can be detrimental to the student’s model performance in general cases. To address this issue, we introduce a simple but effective method called Adaptive Selective Knowledge Distillation (ASKD), where the student model focuses on valuable knowledge, leading to a lower empirical risk bound compared to vanilla KD. Furthermore, in large-scale classification tasks, quantitative experiments show that ASKD improves performance on tasks such as CIFAR-100, ImageNet-1K, large-scale face recognition, and ASKD wins second place of the iQIYI-VID test track in the lightweight face recognition challenge.