Industrial Large-Scale Multimodal Foundation Model for Complex Equipment Anomaly Detection with Incomplete Data (WCCM2024)

With the arrival of Industry 4.0, the demand for intelligent anomaly detection (AD) of complex equipment is becoming increasingly high. However, massive monitoring data not only have complex modalities, but also lack labels, making it difficult for intelligent AD models to make accurate decisions. To fuse multimodal information, an industrial multimodal large-scale foundation model (ILSMF) is proposed, which utilizes massive and incomplete unlabeled data from complex equipment for unsupervised learning. Firstly, ILSMF utilizes a pre-trained multimodal scale-variable feature extractor to fuse feature of incomplete multimodal data, transforming the original multimodal dataset into a new feature set. Then, under the constraint of the feature set, the incomplete data are restored through the generation module. To construct anomaly indicators for data, the recovered data is reconstructed through a graph autoencoder network embedded with structural knowledge. Finally, the ILSMF will determine whether the anomalies have occurred based on AD indicators. To demonstrate the effectiveness of the proposed method, we have conducted sufficient case studies and result analysis, where the dataset is from 19 static firing tests of a certain engine model. The experimental results show that the proposed method could efficiently fuse multimodal data and accurately detected the anomalies with the incomplete data.