In a product-based manufacturing layout, the operator must inspect multiple similar products, making the process monotonous and strenuous. The quality control stage is mandatory in any manufacturing industry before the final product is dispatched. Of many quality inspection procedures, visual inspection is the most widely used. However, the visual inspection process becomes repetitive due to the vast number of products to be inspected daily. Hence, the operator carrying the inspection process is prone to human limitations like fatigue, which results in error, i.e. missing some of the inspection points. To mitigate this problem, a Computer-Assisted Visual Inspection System (CAVIS) has been proposed in this work, which monitors and ensures inspection carried out by the operator. In CAVIS, a hand-tracking model detects and tracks the operator’s hand motion, i.e. the pointing finger used to touch the inspection markings to be checked. The present work is divided mainly into two parts: developing the inspection system algorithm to determine the coordinates of inspection marks in the image captured by a vision camera and subsequently tracking and ensuring the operator touches all the inspection markings using the pointing finger in real time, i.e. validating the developed system through experimental investigation.