Transformer-based semantic segmentation models have played a revolutionary role in the computer vision field. The current methods primarily rely on self-attention mechanisms to enhance model performance, thereby overlooking the significant inference cost of transformer models. To address this issue, we propose a token fusion method for transformer-based segmentation models, without requiring retraining. To identify the optimal token fusion strategy, we employed the following three techniques: (i) a novel token grouping strategy; (ii) a lightweight token matching method; (iii) selection of the optimal token fusion areas. We apply our method on Segformer, Segmenter, MaskFormer and Mask2Former, and we evaluate its effectiveness on the ADE20K datasets. Our framework achieves up to a 1.12 \(\times \) reduction in FLOPs and a 1.11 \(\times \) speedup in inference latency without compromising accuracy.