HaM3D: generalized XR-based multimodal HRI framework with haptic feedback for industry 4.0

The current generation of Industry 4.0 involves the integration of automation and data in manufacturing, while the upcoming generation, Industry 5.0, aims to enable seamless interaction between humans, robots, and machines for sustainable and personalized product and service creation. This necessitates manufacturing components according to customer wishes, leading to frequent changes in the production line. In this context, automation proves unsustainable and could be addressed through Human–Robot Cooperation. In this paper, an Extended Reality (XR)-based Human–Robot Interaction with haptic feedback has been proposed to assist users in remotely performing tasks. The haptic feedback, generated using the Artificial Potential Field method, aids users in controlling robots without collisions. This paper introduces a multimodal XR-based framework for effective communication between humans and robots, utilizing vision models to provide information about the remote side to local users. This seamless communication is facilitated through multiple modalities such as haptic, speech, visual, and text. The system’s effectiveness through pick-and-place activities and adaptation to it for complex industry use cases such as peg in the hole and welding are further evaluated. The results demonstrate the system’s efficacy in two specific human–robot collaborative scenarios in industries: arc welding and assembly tasks.