High-precision interaction in augmented reality applications using tool-mounted cameras

The use of tools is fundamental in many augmented reality (AR) applications, particularly in fields such as education and industry. These applications often rely on cameras integrated into the head-mounted display (HMD) for tool tracking; however, this setup suffers from several limitations, including a restricted viewpoint, partial occlusions, sensitivity to ambient lighting, and limited precision. This paper presents a novel system specifically designed for AR applications, combining HMD-mounted cameras with cameras embedded directly in the tools. As part of the system, we propose a global optimization framework for the seamless integration of all devices, along with a dedicated calibration procedure for the tool-mounted camera. Furthermore, we have conducted an extensive experimental evaluation to assess the system’s precision and robustness in comparison with common alternatives. The results demonstrate improved robustness under limited visibility, greater independence from the HMD, and a notable increase in detection accuracy. To the best of our knowledge, this is the first system in the literature to adopt such a configuration.