Robust segmentation of surgical tools is essential to improve robot-assisted surgery, but is affected by challenging visual scenes such as smoke, bleeding, and low light. Deep learning models cannot be generalized to such diverse domains, typically suffering from catastrophic forgetting and data privacy issues. To overcome this, we present a Domain-Incremental Continual Learning (CL) framework for robust and privacy-preserving segmentation of surgical tools. We construct our solution based on Segment Anything Model 2 (SAM2) and utilize parameter-efficient Low-Rank Adaptation (LoRA) for domain-specific adaptation learning. The foundation of our solution is a K-Means clustering strategy on CLIP embeddings that dynamically selects the appropriate LoRA adapter for the current visual domain, isolating domain knowledge, and avoiding forgetting. We perform a rigorous evaluation on the challenging SegSTRONG-C endoscopic video dataset. Our findings show that our solution is substantially better than CL baselines at segmentation accuracy as well as knowledge retention, presenting a promising path to reliable and adaptive AI for real-world surgical procedures. The code is made public at https://github.com/DonWick32/image-segmentation .