Transfer learning is widely used as a means to leverage knowledge obtained from a source dataset to solve a downstream task on a target dataset. In the realm of image classification, models trained on large datasets often exhibit high degrees of invariance in the output spaces against intra-class variations such as geometric or chromatic changes that leave the class labels invariant. Leveraging these invariant characteristics from one model to enhance the performance of another is an attractive prospect. However, a previous study has pointed out that conventional transfer learning approaches often compromise the robustness of the pre-trained model when transferred. In this paper, we propose a novel transfer learning method called TransInv, aimed at transferring the invariant properties of a teacher model to a target model. The invariance of the teacher model is expressed by a set of augmented samples produced by our proposed data augmentation module, which is jointly optimized with the target model parameters. We demonstrate that our proposed method effectively transfers the invariant properties of the teacher model to the target model, resulting in superior model performance compared to baseline methods. The code will be released upon acceptance.