Fooling Downstream Classifiers via Attacking Contrastive Learning Pre-trained Models

Nowadays, downloading a pre-trained contrastive learning (CL) encoder for feature extraction has become an emerging trend in computer vision tasks. However, few works pay attention to the security of downstream tasks when the upstream CL encoder is attacked by adversarial examples. In this paper, we propose an adversarial attack against a pre-trained CL encoder, aiming to fool the downstream classification tasks under black-box cases. To this end, we design a feature similarity loss function and optimize it to enlarge the feature difference between clean images and adversarial examples. Since the adversarial example forces the CL encoder to output distorted features at the last layer, it successfully fools the downstream classifiers which are heavily relied on the encoder’s feature output. Experimental results on three typical pre-trained CL models and three downstream classifiers show that our attack has achieved much higher attack success rates than the state-of-the-arts, especially when attacking the linear classifier.