Stylized image captioning aims at generating captions that accurately describe the image content while aligning with the desired style. In semi-supervised setting, existing methods typically first pre-train models on large-scale factual image-caption pairs and then fine-tune the pre-trained models on small-scale unpaired stylized corpus, requiring significant resources. In this paper, we propose PPCap, a novel Plug and Play framework for stylized image captioning, where only a stylized image captioning model needs to be trained on small-scale unpaired stylized corpus. Then it will form a generative style discriminator via Bayes rule by the contrast of the captions in different styles, guiding an off-the-shelf large-scale factual image captioning model to generate stylized image captions in a post-processing manner, which is flexible and efficient. Experimental results on SentiCap and FlickrStyle10k show that our framework achieves comparable performance to the state-of-the-art methods in the same setting while reducing training time by over 90%. Our code is available at https://github.com/gWeiXP/PPCap .