Much progress has been made in the field of person re-identification, but changes in clothing have hindered the practical application of long-term person re-identification. Cloth-changing person re-identification (CC-ReID) aims to address this problem, with the main challenge being the extraction of discriminative features unrelated to clothing. Existing methods, which mainly focus on introducing clothing-irrelevant cues such as key points, contours, and 3D shapes, require additional modules for feature extraction, resulting in increased complexity and potential inaccuracies due to dependence on the performance of external models. Few studies directly use the original RGB images to make the model constantly focus on clothing-independent information. In this paper, we propose a Confidence-Guided Feature Alignment Network (CGFA) for CC-ReID. Specifically, we design a confidence module that automatically learns to make confidence adjustments to fine-grained information, prompting the model to mine clothing-independent discriminative features without introducing other modal cues. By transferring knowledge, we encourage the model to learn discriminative identity features that are independent of clothing bias. As a result, the confidence module can be removed during the inference phase. The proposed simple but efficient method uses only RGB modality without additional cues, and can serve as a powerful baseline for CC-ReID to drive future research. Extensive experiments on the CC-ReID datasets demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance.