Text-based person retrieval (TBPR) is a challenging topic in cross-modal retrieval tasks, aiming to query corresponding person images based on textual descriptions. This task is complicated by noisy correspondences between images and text due to incorrect text annotations and low-quality images. Although many robust noise learning methods have been proposed with satisfactory performance, they tend to directly filter out a portion of the noise in the dataset, which decreases the ability to distinguish the noise as the number of training epochs increases. This paper proposes a noise-robust learning strategy based on false positive detection (FPD). Precisely, FPD consists of two main components: 1) The key-value library (KVL) module utilizes a set of clean samples, assigns confidence weights to each training sample during the training process, and adaptively adjusts the contribution of each sample to reduce the accumulation of noise. 2) To improve the performance of TBPR, this paper designs a local token-based selection method, which utilizes the contribution of each local token to the weights in the self-attention matrix and selects the local tokens with vital information as global representations. Extensive experiments are conducted on three datasets (CUHK-PEDES, ICFG-PEDES, and RSTPReID) to evaluate the performance and robustness of FPD under noisy conditions. The experimental results show that the FPD has better results at high noise ratios, and all the metrics outperform the state-of-the-art method under the RSTPReID dataset with a noise ratio of 80%, where Rank-1 outperforms the best method by 2.28%, 1.64% for mAP, and 1.92% for mINP.