Electronic Health Records (EHRs) play a vital role in healthcare by providing structured data that supports precise decision-making. With the exponential growth of medical data, cloud-based storage and retrieval systems have become prevalent for managing EHRs. However, handling sensitive medical information in the cloud often poses challenges related to efficiency and security. To address these challenges, we propose PCMedIR, a privacy-enhancing cross-modal medical information retrieval system that ensures both high performance and data privacy. Our approach utilizes the Contrastive Language-Image Pre-Training (CLIP) model to extract multimodal features from medical images and text reports, and the Deep Pairwise Hashing model is used to generate similarity-preserving hash codes from these features for effective retrieval. Medical images are first encrypted using a hyperchaos-based image encryption algorithm, after which each image’s corresponding text report is embedded into the encrypted image using steganography and stored securely along with their respective hash codes. During retrieval, a text query retrieves similar medical images, while an image query returns relevant text reports. Experimental results demonstrate that PCMedIR performs effectively, achieving a precision of 80% for image-to-text retrieval and 90% for text-to-image retrieval. It offers enhanced security, ensuring privacy-preserving storage and access to EHR data in cloud environments by protecting them against unauthorized access, inference attacks, and data tampering.