Text-based person search (TBPS) faces the severe problem of domain shifts in practical applications, which causes significant performance degradation and has to be carefully addressed. Test-time Adaptation (TTA) offers a feasible approach for quick adaptation during inference, but there are two major challenges when directly applying existing TTA approaches to TBPS: 1) Less Discriminative Cross-modal Distributions: the text-to-image prediction distributions are with flat characteristics and low discriminability caused by numerous irrelevant images inside the gallery. 2) Modality-specific Domain Shifts: domain shifts exist independently in textual and visual modalities. To address these challenges, we propose a novel test-time adaptation method named Distribution-discriminative and Modality-aware Domain Adaptation (DMDA). Firstly, the Discriminative Pairs Learning (DPL) module is proposed to discard irrelevant images and select a credible one for each text query based on cross-modal similarity, producing more discriminative distributions for cross-modal alignments. Secondly, the Modality-aware Stable Adaptation (MSA) module explicitly aligns domains within respective modalities by an anchor-based stable adaptation strategy. Finally, extensive experiments and analyses have been conducted to validate the effectiveness of our DMDA across three publicly available benchmarks for TBPS.