Spatiotemporal prediction (STP) utilizes historical properties to predict future trends. However, most STPs only consider the single granularity data and ignore the diversity of spatiotemporal patterns, i.e., different granularities, thus performing mediocrely. In this paper, we propose an Intra- and Inter-granularity Contrastive Learning Framework (IICLF) to enhance STP with several key challenges: i) Difficult to learn reliable spatiotemporal representations under different granularities; ii) STP suffers from uneven data distribution. To address the first challenge, we devise an intra- and inter-granularity contrastive learning module, which enhances the spatiotemporal representations by incorporating the commonalities and differences across granularities. For the second, we design a dual hypergraph convolutional module integrating geographical hypergraph and semantic hypergraph to capture higher-order dependencies among nodes and mitigate the problem of uneven data distribution. In addition, we introduce a supervised task that distinguishes the importance of different granularities and makes compelling predictions. Extensive experiments on three real-world datasets validate that our proposed IICLF is superior to various state-of-the-art STP methods.