Lots of popular calibration methods in medical images focus on classification, but there are few comparable studies on semantic segmentation. In polyp segmentation of medical images, we find the most diseased area only occupies a small portion of the entire image, resulting in previous models not well-calibrated for lesion regions but well-calibrated for background, despite their overall seemingly better Expected Calibration Error (ECE) scores. Therefore, we proposed the four-branches calibration network with Mask-Loss and Mask-TS strategies to focus more on the scaling of logits within potential lesion regions, which serves to mitigate the influence of background interference. In the experiments, we compare the existing calibration methods with the proposed Mask Temperature Scaling (Mask-TS). The results indicate that the proposed calibration network outperforms other methods both qualitatively and quantitatively.