In modern industrial manufacturing, manual assembly remains essential. Expert assistance systems have been developed for some areas but lack flexibility and the ability to perceive dynamic working scenes. A deeper and more comprehensive understanding of the entire working scene within an assembly cell enables the development of more complex and advanced assistance features. By integrating promising deep learning techniques, the expert assistance system will automatically capture and analyze assembly operations, providing appropriate guidance and error detection. This paper focuses on leveraging advanced large models and multiple data modalities, enhancing the expert systems’ ability to understand complex and dynamic assembly scenes.