Heterogeneous Transfer Learning Using Pre-trained Feature Mapping and Exchange

Transfer Learning (TL) is an efficient approach for training neural networks with improved performance and lower costs compared to training from scratch. However, traditional TL techniques have drawbacks in mapping knowledge to the parameters of new networks, as they must necessarily use the same backbone as pre-trained models. To overcome this challenge, we introduce TLVFC, a novel method that enables knowledge transfer between networks with different architectures. For convolution layers, target model weights are initialized by identifying suitable pre-trained candidate layers and adjusting their parameters through variance to achieve alignment. For fully-connected layers, a new weight initialization approach is implemented that utilizes the distribution of pre-trained weights. To make the most of the model trained on a large dataset, an innovative feature exchange technique is further applied in the fine-tuning pipeline. Our method demonstrates remarkable convergence speed, outperforming the current state-of-the-art DPIAT [1] by over 2% in accuracy across various image classification benchmarks. The source code is available at https://github.com/ThuanNaN/TLVFC .