Iterative Transfer Knowledge Distillation and Channel Pruning for Unsupervised Cross-Domain Compression

Practical applications of deep learning are challenged with critical issues that the distributions of training data and testing data are different and the labels of testing data are insufficient. To address these problems, unsupervised domain adaptation (UDA) based transfer learning has gained significant attention. However, advanced deep learning models of UDA are too complex for real-time and resource-constrained applications. In this paper, we propose an iterative transfer model compression (ITMC) method with two key modules, i.e., transfer knowledge distillation (TKD) and adaptive channel pruning (ACP). During each epoch, the TKD module achieves model compression by distilling the knowledge from the teacher model to the student model, while facilitating the transfer of cross-domain knowledge to enhance the performance in the target domain. Concurrently, with the aid of the ACP module, redundant channels in the student model are pruned to reduce the computational cost while retaining the model accuracy. In particular, the alternation of ACP and TKD ensures effective knowledge transfer, balancing the model size and its performance in the target domain. Experimental results demonstrate that ITMC approach achieves higher accuracy under the same compression ratio compared with the state-of-the-art methods.