Recognizing the activities of construction workers is crucial for enhancing productivity. Traditionally, analyzing worker activities on construction sites often rely on manual supervision, which can pose challenges for project managers due to the potential inaccuracies in the recorded data. This paper presents a system for recognizing the activities of construction workers and automatically calculating their effective work hours. Firstly, after identifying workers’ activity (e.g., bricklaying) as the subject of study in construction industry, Mediapipe as a vision-based framework to detect human activities is used to track the joints of construction workers and convert this information into coordinate data. In this process, the number of frames corresponding to each action in the video is recorded to obtain data related to the duration of each activity automatically. Secondly, an LSTM model is used to process videos and classify construction activities. And a dataset which concludes the segmented videos of construction activities is developed to train the LSTM model. Finally, an evaluation system for worker efficiency concluding the Mediapipe and the LSTM model can be used to count workers’ effective hours automatically. The effectiveness of each activity is determined based on its classification. The test accuracy of the system is 82.23%. The established system facilitates the extraction of information regarding categories and durations of construction activities, which offers significant assistance in enhancing construction productivity.