Fact Error Correction (FEC) aims to identify and rectify factual errors in claims, thereby enhancing the accuracy and reliability of information. Existing work primarily focuses on unsupervised FEC methods to improve model faithfulness and ensure the factual consistency of outputs to the provided evidence. These methods identify informative spans within a given claim, mask these spans to generate questions, and then employ a Question Answering (QA) model to answer these questions, ultimately generating corrected claims. However, current unsupervised FEC models overlook the valuable hints provided by these informative spans, which contain semantic and type constraints crucial for error correction. To address this issue, we propose the method for training a Fact Error Correction model, named CorrectFEC, to correct factual errors more effectively. Specifically, CorrectFEC generates claim-evidence pairs using QA datasets, employs a vanilla T5 model to predict masked spans, and subsequently applies contrastive training to Pretrained Language Models (PLMs) to correct factual errors by leveraging the span prediction results. As a result, CorrectFEC can correct errors in claims by fully incorporating the semantic hints from these informative spans during inference. Our experimental results on the FEVER and SCIFACT datasets demonstrate that the CorrectFEC model outperforms existing unsupervised FEC models, achieving a 5% improvement in the SARI score. The code is available at https://github.com/NEUIR/CorrectFEC .