Computer assisted navigation is important in surgeries. In this paper, we focus on the field of total knee arthroplasty, where computer aided navigation is already widely used to increase surgical precision. For the purpose, marker-based optical measurement systems are used, which are able to determine the position and orientation of surgical tools as well as femur and tibia. For the purpose of tracking the bones, optical locators must be drilled into the patient’s femur and tibia to determine the position and orientation precisely. However, the temporarily inserted locators slow down the patient’s healing process, due to the additional drilling. This article presents a solution that aims to replace the marker-based measurement system used in total joint arthroplasty with an image-based measurement system. The 3D model of the knee required for computer-aided navigation is to be reconstructed in real time from 2D images using photogrammetric methods. This requires the relevant image data (femur and tibia) to be reliably identified and separated from the image background. For this purpose, an AI-based segmentation was implemented to pre-process the 2D image data. The difficulties and requirements are shown and a first proof-of-concept solution with initial results is presented.