The rapid development of joint profiling methods makes it possible to simultaneously measure multi-modal data from the same cell. It enables the more comprehensive insight of cellular heterogeneity. However, due to the heterogeneity of features across different modalities and the complexity of each modal data, effectively integrating multi-modal data to achieve more accurate cell heterogeneity analysis remains challenging. Here, we propose a novel contrastive learning-based approach for single-cell multi-omics data clustering, named scCLC. Taking contrastive learning as backbone, scCLC integrates scRNA-seq data and scATAC-seq data for cell presentation learning with dedicated data augmentation strategy and self-supervised learning model. scCLC leverages the topological structure derived from single-cell multi-omics data to determine positive pairs, which not only increases the training sample size but also provides basic label priors for the model to learn more cluster-friendly cell representations. The experimental results on single-cell multi-omics datasets show the superior clustering performance of scCLC. When applied to visualization, scCLC can effectively separate the cell subtypes.