In this paper, we propose a novel medical image registration framework called Det-SAMReg, aiming to fully leverage foundation vision models while minimizing training efforts. More specifically, the Grounding DINO-based detection module is first used to predict the target organ (e.g., prostate)’s bounding box using engineered text prompts specific to the anatomy-of-interest, with which the SAM-based segmentation module then automatically generates organ masks for both the moving and fixed images. These pairs of segmented masks are subsequently fed into an iterative optimization-based registration process to estimate the voxel-wise dense displacement field. Furthermore, it is found that fine-tuning detection and segmentation modules with substantially fewer samples achieved comparable performance to using the full training set. The impact of noise in localising bounding boxes on detection, segmentation and registration performances is also investigated. Experimental results of MRI-to-US registration demonstrate that Det-SAMReg significantly outperforms competitive learning-based methods, achieving a Dice Similarity Coefficient (DSC) of 0.867 and a Target Registration Error (TRE) of 5.548 mm, compared to those being 0.724 and 8.897 mm achieved by the strongest tested baseline method TransMorph ( \(p <0.001\) ). Det-SAMReg remarkably maintains superior performance and significantly outperforms ( \(p <0.001\) ) compared methods in MRI-to-US registration when detection and segmentation modules are fine-tuned with only 14 training samples. The source codes have been released at: https://github.com/ymttt111/Det-SAMReg.git .