Automatic prediction of dose distribution maps wields considerable influence in clinical radiotherapy treatment. Recently, deep learning-based approaches have been explored to automatically predict the dose map from structure images and obtain promising results. However, these methods mainly focus on extracting anatomical features from CT and organ masks, ignoring abundant visual knowledge inherent in the domain of dose map. To address this limitation, we innovatively propose a visual prompt-guided dose prediction model, named ViPDose, to effectively predict radiotherapy dose distribution for cancer patients. Specifically, our ViPDose is structured with two key stages: 1) a prompt pre-training stage and 2) a prompt generation stage. In the pre-training stage, we train a prompt encoder to encode dose maps alongside structure im-ages into compact prompt vectors. Then, in the prompt generation stage, we design a fast prompt generator fulfilled with a diffusion adversarial network (DAN) to efficiently produce the prompt vectors that closely approximate those generated by the prompt encoder, thus enriching the model with abundant visual prompt information. By adopting DAN in such highly compressed latent space, our method can guarantee high-quality predictions with relatively low computation costs. Comprehensive experiments on a clinical rectal cancer dataset with 130 cases have verified the superior performance of our method over other state-of-the-art methods.