Decoding BatchNorm statistics via anchors pool for data-free models based on continual learning

Generating high-quality samples reversely from existing models is a significant technique in continual learning and knowledge distillation. Existing approaches either fail to generate valid samples from models or require massive storage space to accommodate a large number of examples generated at once. To address the above issues, this paper presents a simple but effective method called DBSAP, referring to decoding Batchnorm statistics via anchors pool. In the proposed method, an anchor pool is constructed for each BatchNorm layer in a neural network to store the corresponding feature representation of a small number of examples sampled. During reverse-generating samples, the anchors are further sampled from anchor pools to participate in the optimization of BN loss, which can generate valid samples even with a small batch size, i.e., very low storage requirement. When multiple models are available in continual learning, more loss optimizations can be performed to further guarantee the validity of generated samples. The extensive experiments on datasets CIFAR10, CIFAR100, ImageNet-200 and ImiageNet-1000 demonstrate the effectiveness and superiority of the proposed method. Compared with the state-of-the-art methods, the DBSAP method can generate the images samples with much higher quality based on multiple metrics and require only about 1/4 GPU memory on average for sample generation.