With the increasing demands for privacy preserving of speech in cloud storage, the complex features along with the high overhead of speech computation and storage challenge the search mechanism. Therefore, we proposed a secure multi-user encrypted speech search scheme in cloud-edge-end environments (MUSES). Firstly, the MUSES scheme extracts speech feature vectors through CNN and RNN, employing secure proximity algorithms to obtain high similarity query results. Secondly, for the demands of multi-user scenarios, the MUSES scheme designs a trapdoor generation mechanism that generates specific trapdoors for each user, preserving the privacy of users. Finally, the proxy re-encryption (PRE) technology is utilized to transmit the symmetric key in the way that the edge server cannot recognize, mitigating the risk of data leakage. The analysis of the scheme proves that the MUSES scheme has high efficiency and does not cause privacy leakage in the speech search process. Meanwhile, the performance analysis also indicates the efficient and accurate search characteristics of the MUSES scheme.