CSSeg: Cross-Supervision for Unsupervised Semantic Segmentation

Unsupervised semantic segmentation has shown progress. Recently, approaches based on self-supervised vision pretrained models, i.e., Vision Transformers (ViT), have demonstrated superior performance. However, most existing methods rely solely on a single self-supervised prediction model, making the model overly dependent on its own prediction. Furthermore, the single-branch framework limits the ability to correct potential biases and leads to unsatisfactory segmentation results. To address the above challenges, we propose a dual-branch framework with cross supervision to achieve a comprehensive prediction. Specifically, we first adapt the Cross-Supervised Learning (CSL) mechanism, in which the framework consists of two networks that generate diverse predictions, each equipped with a teacher-student predictor for pseudo-label generation, and mutually guide each other through cross supervision. This dual-network collaboration mechanism facilitates the integration of multi-perspective information, enhancing the diversity of feature representations, thereby improving the robustness and generalization ability of the model. However, CSL cannot guarantee the accuracy of the semantic information learned by each branch, which may lead to the propagation of incorrect information during cross-supervision. To this end, we introduce the Patch Shuffle Entropy Select (PSES) module, which evaluates the entropy differences between original images and their patch-perturbed counterparts as a criterion to identify the network with more stable predictive performance, thereby guiding the update of the other network. With the proposed training strategies, our method achieves significant performance improvements on both the Cityscapes and Potsdam-3 datasets.