In recent years, AI-powered social bots have become more anthropomorphic and deceptive, posing a serious challenge to combat the spread of misinformation in social networks. However, the lack of high-quality annotated data severely limits the further development of social bot detection technology. Moreover, existing graph-based approaches analyze social networks as static graphs, overlooking the inherent dynamic nature of the evolving social network. To address the above drawbacks, we propose BotTGCL, a novel social bot detection framework that jointly utilizes the generalization patterns of the social graph structure and the dynamic nature of evolving social networks to improve the detection of social bot. Specifically, we construct the social network as a dynamic graph and employ a graph contrastive learning module to learn the topological patterns of graph structure in unlabeled social networks. We then propose a graph temporal module to integrate historical context and extract temporal patterns from the evolving graph. Finally, we fuse topological patterns and temporal patterns to classify users as social bots or humans. Extensive experiments conducted on two comprehensive social bot detection benchmarks demonstrate that BotTGCL achieves superior performance compared to state-of-the-art methods and exhibits exceptional performance in real-world scenarios with scarce labeled data. Additional studies also confirm the effectiveness of our proposed graph structure contrastive learning and graph temporal pattern learning.