Purpose <p>Foundation models, trained on multitudes of public datasets, often require additional fine-tuning or re-prompting mechanisms to be applied to visually distinct target domains such as surgical videos. Further, without domain knowledge, they cannot model the specific semantics of the target domain. Hence, when applied to surgical video segmentation, they fail to generalise to sections where previously tracked objects leave the scene or new objects enter.</p> Methods <p>We propose <i>SASVi</i>, a novel re-prompting mechanism based on a frame-wise object detection <i>Overseer</i> model, which is trained on a minimal amount of scarcely available annotations for the target domain. This model automatically re-prompts the foundation model <i>SAM2</i> when the scene constellation changes, allowing for temporally smooth and complete segmentation of full surgical videos.</p> Results <p>Re-prompting based on our <i>Overseer</i> model significantly improves the temporal consistency of surgical video segmentation compared to similar prompting techniques and especially frame-wise segmentation, which neglects temporal information, by at least 2.4%. Our proposed approach allows us to successfully deploy <i>SAM2</i> to surgical videos, which we quantitatively and qualitatively demonstrate for three different cholecystectomy and cataract surgery datasets.</p> Conclusion <p><i>SASVi</i> can serve as a new baseline for smooth and temporally consistent segmentation of surgical videos with scarcely available annotation data. Our method allows us to leverage scarce annotations and obtain complete annotations for full videos of the large-scale counterpart datasets. We make those annotations publicly available, providing extensive annotation data for the future development of surgical data science models.</p>