Context-aware assistance systems are becoming essential for aiding decision-making and task management for the workers. However, integrating these systems with various other entities such that they interact seamlessly, poses a challenge. This paper proposes a solution that leverages advanced interaction protocols and context-aware robotic assistance to ensure smooth and effective collaboration in the factory environment supported by three pillars: 1) an ontology-based Digital Twin (DT) representing the semantic dimension and providing object-specific information, 2) an egocentric perception system using HoloLens2 to observe the worker with individually tailored dataset for context recognition, and 3) a robotic assistant guided by an intention-recognition module that responds to emerging worker needs, based on behavioral and environmental observations. In a manual assembly scenario, our framework provides useful contextual information as well as automated and adaptive robotic assistance. We present the results from a user experiment which was conducted to compare the stress level of participants with and without the use of the assistance system in a manual assembly scenario. Physiological sensors from the Empatica E4 wrist-watch were used to collect physiological data on which the stress levels were detected using a deep learning model. Additionally, the user ratings on the System Usability Scale (SUS) were used to assess the usability and acceptance of robotic-assistance technology.