Artificial intelligence (AI) has been gradually introduced into the fields of radiological and endoscopic diagnosis as supports for clinical practices. In the field of laparoscopic surgery, AI-based image recognition technology is also promising for the development of surgical navigation, skill assessment, and OR (operating room) management. Some of the active research and developments in this area using deep learning approach include the identification of surgical phases, surgical instruments, and anatomical structures. The major tasks of AI can be divided into image classification (the task of assigning a whole image to a specific class), object detection (the identification of the location of lesions, organs, or other objects with a circle or a box region of interest), and semantic segmentation (the recognition of the precise pixel-wise borders of objects). To develop these AI-based systems, a large number of still images from surgical videos are required as the training set, a dataset of annotations for each still image. Therefore, the construction of the high-quality surgery video database which contributes to the development is desired. We need to establish this infrastructure to boost the developments of AI-based surgical systems with global collaborations.