Towards the Necessity of Pre-trained Heads in Model-Agnostic Meta-Learning

In this paper, we explore the necessity of meta-training the final layer of the network in model-agnostic meta-learning (MAML) for few-shot learning. Previous research has shown that updating only the final layer during fine-tuning can improve performance. We go beyond this by randomly re-initializing the final layer before optimizing the inner loop and not updating its weight in the meta-step to discover the necessity of pre-training the last layer. Our findings indicate that pre-training the final layer is slightly beneficial when the task distribution does not change between training and testing. However, our novel approach excels in cross-domain adaptation when the tasks change during testing. Re-initializing the final layer forces the body of the network to learn better representations. We perform experiments on various in-domain, cross-domain setups, and mixed-way scenarios and conduct a representation similarity analysis to analyze these networks.