Broad Learning System Based on Fusion Features

Multimodal fusion is the integration of data from different modalities according to the principle of optimality to produce more reliable perception of objects. Deep learning has been widely used in multimodal fusion domain, but how to reduce model parameters and training time while maintaining good performance is still an academic problem. To address these issues, we design a system capable of learning and fusing features from both modalities, the Deep Residual network Broad Learning System (DR-BLS). First we use a pre-trained deep residual network (ResNet18) to extract features of different modalities separately. And then we use canonical correlation analysis to project the two obtained features in the same feature space for correlation learning and fusion of features. Finally the broad learning system is used as classifier for target recognition and classification. Experiments on target classification recognition on the University of Washington RGB-D dataset show that the method pro-posed in this paper is more stable and faster than other algorithms.