Speech-to-text (S2T) retrieval aims to retrieve corresponding textual information from large-scale data based on a speech query. To efficiently build such a S2T retrieval system, existing works proposed to initialize it with pre-trained large language models (LLMs). However, text-only LLMs lack attention to speech, which limits the generalization ability of retrieval systems built on these pre-trained models. To address this, we turn to the large-scale pre-trained automatic speech recognition (ASR) model Whisper, which has been pre-trained on web-scale multi-modal speech-text pairs, and propose extending it to the S2T retrieval task. By removing the cross-attention layers in the Whisper model, we decouple its audio encoder and text decoder, thus constructing a dual-tower architecture. This architecture independently processes both speech and text and leverages contrastive learning to optimize the alignment of speech and text embeddings. To indicate the efficiency of our design, we fine-tune this structure only on limited resources. Experimental results demonstrate that the proposed method outperforms current state-of-the-art open-source models on the FLEURS dataset, achieving a notable performance boost. In particular, our model is capable of cross-lingual speech-text matching, enabling it to perform speech-to-text translation (S2TT) retrieval tasks.