In cloud-based image privacy protection, encrypted image feature extraction is critical for constructing efficient retrieval systems. Outsourcing the feature extraction task to cloud servers alleviates the computational burden on data owners compared to conventional plain-image processing. However, encryption operations disrupt the visual structure of images, making deep learning models struggle to learn discriminative representations from cipher-images. To address this limitation, we construct a Contrastive Image-Edge Image Learning Framework (CIELF) by utilizing the handcrafted features of cipher-images and their corresponding edge cipher-images, aiming to enhance the expressive ability of handcrafted features. Additionally, an Interactive Feature-Sequence Mixing Multi-Layer Perceptron (IFSxMLP) is built. It learns feature sequences and their subsequences through the interaction between the feature dimension and the sequence dimension, which can mine more fine-grained representations from the handcrafted features. Experimental results on two benchmark datasets indicate that mAP@300 reaches 92.44% on Corel1K and mAP@100 reaches 85.73% on Corel10K. Compared with current state-of-the-art scheme, our scheme achieves a remarkable performance improvement while ensuring image security.