Multimodal Deep Learning for Manufacturing Systems: Recent Progress and Future Trends

The development of sensing technology provides large amounts and various types of data (e.g., profile, image, point cloud) to describe each stage of a manufacturing process. Deep learning methods have the advantages of efficiently and effectively processing and fusing large-scale datasets and demonstrating outstanding performance in different tasks such as process monitoring and diagnosis. However, multimodal monitoring data raise new challenges to apply existing deep learning methods to solve manufacturing tasks: (1) features across modalities contain complementary yet redundant information (inter-modal data fusion); (2) single modal data can also contain information from different viewpoints (intra-modal data fusion);(3) besides the fusion among data, domain knowledge in advanced manufacturing should also be actively fused into the feature extraction (domain knowledge fusion). This chapter provides three examples of cutting-edge multimodal deep learning methods focusing on inter-modal, intra-modal, and domain knowledge fusion, respectively. From the application aspect, they are designed to solve online process monitoring, product quality inspection, and manufacturing process design, which cover both the forward prediction task and backward optimization task in manufacturing systems. Several prospective directions in multimodal deep learning for advanced manufacturing are further discussed.