In previous chapters, we introduced research works that employ KD to compress and accelerate neural networks. In these works, the teacher models are usually defined as the models that have more parameters while the student models are defined as the models that are more lightweight and efficient. In this setting, when the knowledge is distilled from the teacher to the student, the teacher model is actually compressed into the student model. Besides model compression, KD can also be utilized for targets besides model compression, by using different manners to define the students and teachers. In this chapter, we introduce how to leverage KD in model training to improve the robustness of computer vision models in Sect. 1, and to reduce the communication cost of updating neural network-based software in Sect. 2.