“Poetry in pictures and pictures in poetry” elucidates a naturally direct correlation between the classical Chinese poetry and vision, making the image and poetry retrieval crucial. Most existing works employ the photograph as the query, but obtaining a suitable photograph brings additional difficulties. In contrast, the free-hand sketch has served as a more convenient tool to depict human perception since ancient times. In this paper, we introduce a new task of Sketch-Based Poetry Retrieval. The task is challenging due to the following factors: (i) the significant differences between sketches and images, as well as between poetry and modern Chinese; (ii) the high time cost of collecting the parallel cross-modal data for the traditional supervised learning. To address these challenges, we construct a sketch-and-poetry pre-training model based on unsupervised vision-and-language learning named SKP-CLIP. Specifically, we utilize a multi-modal knowledge graph for poetry to bridge the semantic gap and then learn the modal alignment through sketch and its caption as well as poetry and its corresponding image entities instead of sketch-poetry pairs. Furthermore, we semi-automatically assemble a dataset for evaluation. Experiments confirm the effectiveness of our method and establish benchmarks for this new task.