Detecting and localising lesions is a key task in the staging phase of diagnosing and treating prostate cancer (PCa). After a positive digital rectal examination or rise in prostate-specific antigen, pinpointing lesion positions for biopsy using multiparametric magnetic resonance imaging (mpMRI) is crucial. mpMRI and ultrasound (US) imaging already aid in collecting cores for biopsy accurately, a procedure called FBx: mpMRI-targeted US-guided prostate fusion biopsy. Yet, physicians face challenges, e.g., due to limited resolutions of both mpMRI and US. This affects patients’ therapy choices, as malignancy assessment accuracy depends on FBx. Recent research aims to improve lesion detection in both mpMRI and US using more objective markers, such as shear wave elastography (SWE). AI can improve FBx using both mpMRI or US data, which has been demonstrated in various studies. However, in the case of mpMRI, labelled lesion examples are still limited, which hinders the performance of state-of-the-art models. Self-supervised learning (SSL) provides a solution by utilising large unannotated databases to create robust feature extractors, enabling the training of case-specific AI models with limited data. Thus, in this paper, we investigate how to improve the models for PCa lesion detection by combining mpMRI and US. We show that recent joint embedding predictive architectures may be a good choice for mpMRI-SSL pretraining. Moreover, we present a false-positive-filtering approach based on real and AI-based SWE, that further improves the mpMRI-model’s specificity. Our model achieves state-of-the-art performance of 0.626 average precision in mpMRI-based segmentation and carries the potential to significantly improve lesion detection and localisation accuracy.