Text-based person retrieval (TBPR) retrieves pedestrian images of the same identity across different camera views based on a query text. It is a challenging task due to obvious cross-modality discrepancy and intra-class variation. Previous methods primarily aim to encode the visual and textual modalities into a common semantic space via contrastive learning paradigm. However, they ignore the matching ambiguity issue due to the failure to consider the one-to-many relationships from the textual modality and the weakly matched pairs caused by the dislocation noise from the visual modality. The matching ambiguity issue from visual and textual modality leads to sub-optimal alignment learning in the common semantic space. In this paper, we propose a novel Soft Multi-View Representation Learning (SMVRL) model to address the matching ambiguity for the TBPR task. Specifically, SMVRL generates multiple region-based representations to build one-to-many relationships. Based on the multiple representations, SMVRL further utilizes both intra-modal and inter-modal similarity clues to comprehensively compute the proper soft correspondence labels for those weakly matched pairs to alleviate the noise inference. Experiments on three representative benchmark datasets, i.e., ICFG-PEDES, RSTPReid, and CUHK-PEDES prove that our method outperforms existing state-of-the-art methods by 2.9%, 2.0% and 1.0% in terms of mAP with a more cost-effective architecture. The source code is available at: https://github.com/Zjamie813/SMVRL .