GFIDF:gradual fusion intent detection framework

Multimodal intent detection integrates various types of information to identify user intent, which is crucial for developing dialog systems that function effectively in complex, real-world environments. Current methods show potential for improving the exploration of connections between patterns and extracting key semantic features from nontextual data. Many researchers opt to fuse data at a single level. In this paper, the gradual fusion intent detection framework (GFIDF), which consists of two main modules, is proposed. The first module, the conical multilayer convolutional attention (CMCA) module, uses a conical multilayer convolutional architecture. This architecture allows the module to capture both local and global contextual information, refining feature representations. The CMCA module is designed to eliminate noise and enhance feature quality by leveraging adaptive convolutional operations. These operations produce a clearer characterization of multimodal data that facilitates alignment and fusion in subsequent processing stages. The second module, the multimodal split and recombination attention (MSRA) module, matches and integrates augmented features from the CMCA module with textual information. This module segments multimodal features into distinct blocks to focus attention on individual segments. By utilizing a block-level attention mechanism, the MSRA module captures interdependencies between modalities, aiding in the understanding of user intent. Four performance metrics are employed for evaluation: accuracy (ACC), F1 score, precision (P), and recall (R). Compared with the baseline model, all the metrics show improvements ranging from 1% to 3%. Experiments validate the CMCA module’s noise reduction effects when processing video and audio modalities. Additionally, the results demonstrate the effectiveness of the MSRA module in fusing the three modal features.