Saliency-guided meta-hallucinator for few-shot learning

Learning novel object concepts from limited samples remains a considerable challenge in deep learning. The main directions for improving the few-shot learning models include (i) designing a stronger backbone, (ii) designing a powerful (dynamic) meta-classifier, and (iii) using a larger pre-training set obtained by generating or hallucinating additional samples from the small scale dataset. In this paper, we focus on item (iii) and present a novel meta-hallucination strategy. Presently, most image generators are based on a generative network (i.e., GAN) that generates new samples from the captured distribution of images. However, such networks require numerous annotated samples for training. In contrast, we propose a novel saliency-based end-to-end meta-hallucinator, where a saliency detector produces foregrounds and backgrounds of support images. Such images are fed into a two-stream network to hallucinate feature samples directly in the feature space by mixing foreground and background feature samples. Then, we propose several novel mixing strategies that improve the quality and diversity of hallucinated feature samples. Moreover, as not all saliency maps are meaningful or high quality, we further introduce a meta-hallucination controller that decides which foreground feature samples should participate in mixing with backgrounds. To our knowledge, we are the first to leverage saliency detection for few-shot learning. Our proposed network achieves state-of-the-art results on publicly available few-shot image classification and anomaly detection benchmarks, and outperforms competing sample mixing strategies such as the so-called Manifold Mixup.