In manufacturing, monitoring task execution is crucial for optimizing efficiency in production units. Traditionally, task execution tracking on the production line has been done manually using stopwatches, which is labor-intensive, time-consuming and prone to human error. While recent AI-based approaches have explored activity recognition, these methods often struggle with real-world challenges such as noise, occlusion, and visual clutter. Additionally, their reliance on complex architectures and high-end computational resources hinders practical deployment on the factory floor. To address these limitations, we present a robust hybrid system for automated activity recognition and time tracking in industrial settings using egocentric vision. Unlike the manual stopwatch method or high-resource AI models, our lightweight framework combines gesture recognition and object detection via a dynamic confidence fusion strategy. Trained on real-world assembly-line data collected from the factory floor, our model achieves 96.7% object detection and 98.6% gesture recognition accuracy. Our multi-modal framework offers a practical and scalable solution for automated task monitoring in complex manufacturing settings.