Fuse Tune: Hierarchical Decoder Towards Efficient Transfer Learning

This paper shows that integrating intermediate features brings superior efficiency to transfer learning without hindering performance. The implementation of our proposed Fuse Tune is flexible and simple: we preserve and concatenate features in the pre-trained backbone during forward propagation. A hierarchical feature decoder with shallow attentions is then designed to further dig the rich relations across features. Since the deep buried features are entirely exposed, back propagation in the cumbersome backbone is no longer required, which greatly improves the training efficiency. We make the observation that Fuse Tune performs especially well in self-supervised and label-scarce scenarios, proving its adaptive and robust representation transfer ability. Experiments demonstrate that Fuse Tune could achieve \(184\%\) temporal efficiency and \(169\%\) memory efficiency lead compared to Prompt Tune in ImageNet. Our code is open sourced in https://github.com/JeavanCode/FuseTune .