Modern railcar assembly lines show increased complexity and variability, requiring assembly robots to make more context-aware decisions by leveraging diverse sensory and textual inputs. Most conventional robotic systems are limited to isolated sensory data sources and often struggle to combine multi modal data, resulting in limited adaptability to dynamic manufacturing environments. This research proposes using cross-modal transformers to fuse multiple sensory modalities with textual data in an assembly robot, thereby enhancing situational understanding and operational precision. Our approach allows the incorporation of cross-attention mechanisms to dynamically weigh each modality’s relevance based on the task context. This multimodal fusion enables the robot to autonomously adapt its actions based on real-time feedback and contextual information. We propose using Deep Deterministic Policy Gradient, a deep reinforcement learning algorithm, to enable real-time task adaptation and optimised robot decision-making. This algorithm lets the robot learn continuously from interacting with the environment to refine its decision-making policies for various assembly tasks. Extensive testing in both simulation and experimental scenarios validates the proposed system. Preliminary results prove that our proposed system improves the accuracy of the task, can smoothly adapt to unforeseen changes, and requires less human intervention. This contribution represents a big step toward better autonomy and flexibility for assembly robots capable of autonomously adapting to complex situations.