Enriching Scene-Graph Generation with Prior Knowledge from Work Instruction

With the current focus on human resources in Industry 5.0, analysing the work movements of industrial operators is the important first step in optimising labour performance. Thanks to the popularity of camera sensors, vision-based Human Activity Recognition models have become useful engines for real-time monitoring tools, in which scene-graphs play an important role. Traditional scene-graph generation methods rely primarily on visual data for perception, neglecting a valuable source of process-oriented prior knowledge: the work instruction. Therefore, an extension of the scene-graph paradigm by integrating ground truth elaborated on elements from the work instruction is elaborated to complement and enhance the understanding of human activities in industrial environments, and improve the tracking capability with micro and repetitive movements. This conceptual paper discusses the basic design of this approach with potential applications in industrial environments, which is validated by a simulated use case of an electronic assembly process. Based on the proposed extension, the Human Activity Recognition model can be lightweight and robust. Further integration of multi-modal sensory inputs beyond visual cues, such as environmental and human-centric data, can enrich scene interpretation and provide a more comprehensive understanding of work behaviour, paving the way for more effective labour utilisation and improved productivity.