The paper aims to use fundus images to determine the patient’s sex. Admittedly, such a possibility has been confirmed by previous studies, but the paper will consider how transfer learning from the RETFound foundation model for retinal images dedicated to the diagnosis of diabetic retinopathy can be used to build such a system and whether the quality of the transfer is sufficient when analyzing data from a different population than the one used to build the fundus model. The study presented here is based on a non-public dataset of fundus images collected from individuals of Polish ancestry. Our study shows that transfer learning, a proven tool for applying and achieving better results when reusing models tuned for different tasks, has some limitations when applied to medical data. In addition, we address emerging concerns about data leakage in medical imaging, as such data may contain overt patient metadata and hidden patterns invisible to humans, which may encode sensitive information that is impossible to capture even by medical experts aided by Explainable artificial intelligence (XAI) techniques.