Image clustering, which involves in grouping images into different clusters without labels, is a key task in unsupervised learning. Although previous deep clustering methods have achieved remarkable results, they only explore the intrinsic information of the image itself but overlook external supervision knowledge to improve the semantic understanding of images. Recently, visual-language pre-trained model on large-scale datasets have been used in various downstream tasks and have achieved promising results. However, there is a gap between visual representation learning and textual semantic learning, and how to deeply utilize the representation of different modalities and distinctive information within modality for clustering is still a key challenge. To tackle the challenges, we propose a novel image clustering framework, named Dual-Level Cross- and Intra-Modality Contrastive Clustering (DL-CICC). Firstly, external textual information is introduced for constructing a semantic space which is adopted to generate image-text pairs. Secondly, the augmented image and image-text pairs are respectively sent to pre-trained encoder to obtain image and text embeddings which subsequently are fed into four well-designed networks. Thirdly, dual-level cross- and intra–modality contrastive loss is conducted between discriminative representations of different modalities and distinct level. Extensive experimental results on five benchmark datasets demonstrate the superiority of our proposed method. Source code is released  https://github.com/SandLYJ .