Knowledge Distillation (KD) is an effective technique for compressing large language models through the teacher-student framework without compromising accuracy. Once a large model is compressed to a lightweight one, it can be stored within limited disk space and trained at higher speed with fewer computational resources. Previous work on feature distillation mainly applied an exact matching between the hidden representations of the student and the teacher. However, since the student has a lower capacity compared to the teacher, it may struggle to mimic its exact hidden representations. This leads to a large discrepancy between their features as demonstrated in preceding research. Therefore, we propose intra-class similarity-guided feature distillation, a novel approach to make the task easier for the student. In this chapter, we map each sample representation by the student to its k-nearest neighbor samples representations by the teacher that are within the same class. This method is novel and can be combined with other distillation techniques. Empirical results show the effectiveness of our proposed approach by maintaining strong performance on benchmark datasets. Furthermore, we evaluate our proposed KD approach on the hate-speech detection task, where we aim to compress HateBERT into a smaller, more efficient model.