Binary surgical tool segmentation is a crucial component of Computer-Assisted Intervention (CAI) applications in minimally invasive surgery (MIS). Accurate segmentation is essential for instrument tracking, surgical scene understanding, and augmented reality overlays. However, a complete CAI system consists of multiple parallel processes, necessitating fast, accurate, and robust segmentation models with low computational cost and minimal memory footprint. In this study, a comprehensive benchmarking analysis of various backbone architectures and decoder topologies for binary surgical tool segmentation is conducted. The evaluation assesses memory efficiency, computational complexity, and generalization ability across unseen surgical datasets, including robotic, laparoscopic, and endoscopic procedures. The experimental results reveal the trade-offs between segmentation performance and resource constraints, offering valuable insights for selecting efficient models suitable for real-time surgical applications.