One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-operative morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilities by observing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted instruments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This chapter addresses the evolution of the computer vision necessary to achieve surgical autonomy, through the study of the anatomical environment, exploring the technology present and what is needed to analyze the scene. The issue of hand-eye calibration is then addressed, aligning the vision system and the robot into a unified reference frame, thereby enhancing the precision of the surgical work plan. This chapter delves into the integration of vision and calibration within the context of 3D reconstruction. In minimally invasive surgery (MIS), simultaneous localization and mapping (SLAM) can be employed to determine the pose of the endoscopic camera and construct a 3D model of the tissue surface. Another crucial aspect for MIS is maintaining real-time awareness of the positions of surgical tools relative to the surgical camera and the underlying anatomy.