Object Detection and Speech Recognition Based Motion Analysis System for Pointing and Calling

Pointing and calling is a method for ensuring safety during working by pointing at work objects or tools and verbally calling and telling the situation in order to predict accidents and raise safety awareness. Therefore, pointing and calling are implemented at production sites to reduce human error and prevent accidents. This method can be used for detecting processing deficiencies and confirming the number of products to be shipped. However, due to a lack of work experience and insufficient recognition of the work importance, there are many cases where the correct motion and procedures are not followed. To ensure the safety of workers in a factory, it is necessary to continuously monitor their actions. Also, the instructors spend a lot of time for monitoring. In this paper, we propose a motion analysis system for pointing and calling based on object detection and speech recognition. The proposed system takes into account the safety measures in order to prevent accidents and injuries at production sites where people are involved. The experimental results show that the proposed system correctly detects the motion sequence of pointing, looking and speaking.