Multi-media processing has achieved great success based on semantic segmentation. Semantic segmentation can be viewed as pixel clustering based on semantic prototypes. However, existing methods focus more on consistent semantics while ignoring the consistency in vision, making this task challenging. Motivated by the success of discrete visual representation learning, we propose Multi-group Visual Semantic Centroid (MVSC) to better cluster the pixels while maintaining consistent semantics of the dense features for any image encoder. Specifically, we randomly initialize multiple groups of prototypes as multi-groups in visual space. The visual features are also randomly split into the same groups and forced to be aligned with the corresponding prototypes. Then these visual prototypes are projected into the semantic space and supervised by the same classifier as the dense features. Compared with existing methods, MVSC further considers the visual space and thus facilitates the task. Experimental results on COCO-Stuff show great improvements compared with previous methods.