Image-text matching is a basic task in the multimodal field to achieve semantic connection between images and texts. Recent methods have demonstrated advanced performance by focusing on cross-modal interactions of embeddings. However, these methods cause the model to prioritize shared representations across modalities, weakening their ability to preserve modality-specific discriminability and resulting in reduced representation diversity. To address the above problem, this paper proposes an intra-modality information transfer method RIMD, which aims to perform cross-modal interactions while preserving the intra-modality discriminative information. Our method consists of two parts: 1) an intra-modality discrimination learning module, which learns discriminative information within a single modality based on message passing, and 2) a Brownian Bridge-based cross-modal interaction module that leverages the learned intra-modality information to mitigate the modality gap. To ensure the effective leverage of high-quality global embeddings for enhanced intra-modality discrimination, we draw inspiration from differential denoising and further optimize the implementation of Local Feature Aggregation. Extensive experimental results demonstrate the effectiveness of RIMD on Flickr30k and MSCOCO datasets.