Two Heads Are Better Than One: Teaching MLPs with Multiple Graph Neural Networks via Knowledge Distillation

Knowledge Distillation (KD) has been successfully applied to compress and accelerate Graph Neural Networks (GNNs) in recent years. Recently, KD has been adeptly applied to boost Multi-Layer Perceptrons (MLPs), enabling them to parallel GNNs in performance. This is notable as MLPs depend solely on node features and require extensive knowledge from multiple teachers for enhancement. However, since existing multi-teacher KD methods often rely on a graph’s topological data, they are typically limited to GNN-based student models, thus excluding MLPs. In this paper, we introduce an innovative method that amalgamates various GNNs into a super teacher, which is then distilled into an MLP student. Moreover, our experiments suggest the promising number of selected teachers configurations to boost the performance of the student model. Extensive experiments on five benchmark datasets show that our proposed approaches outperform the state-of-the-art methods and achieve even higher accuracy than the teacher models. Also, the inference time of our approach is 30 \(\times \) -60 \(\times \) faster than GNNs and other KD methods.