<p>Multimodal intent detection is an important task for understanding human language in real-world multimodal scenarios. Previous intent detection methods rely on textual information, ignoring the potential connections between different modalities, and face the challenge of efficiently extracting semantic features from non-linguistic modalities. In response to these challenges, we propose the MIMO model, which employs a multimodal feature extraction and processing approach to address the complexity of text, video, and audio data. In our model, we employ a feature selection flow (SwinVTRSFlow) to process video features, reduce feature dimensionality, and improve model performance. To establish an optimal multimodal semantic environment for text modalities, we introduce Perceptual Fusion Representation (PFR). PFR efficiently aligns and fuses the features of different modalities and utilizes intent tags to obtain optimal textual semantic insights, guiding the learning process of other modalities. Experiments show that our video and audio feature enhancement and transformation approach significantly outperforms the baseline, improving the processing efficiency and performance of multimodal data. These improvements are crucial for multimodal intent detection tasks.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Integrated feature-enhanced multimodal intent detection method

  • Qimeng Yang,
  • Lanlan Lu,
  • Yi Liu,
  • Jinmiao Song

摘要

Multimodal intent detection is an important task for understanding human language in real-world multimodal scenarios. Previous intent detection methods rely on textual information, ignoring the potential connections between different modalities, and face the challenge of efficiently extracting semantic features from non-linguistic modalities. In response to these challenges, we propose the MIMO model, which employs a multimodal feature extraction and processing approach to address the complexity of text, video, and audio data. In our model, we employ a feature selection flow (SwinVTRSFlow) to process video features, reduce feature dimensionality, and improve model performance. To establish an optimal multimodal semantic environment for text modalities, we introduce Perceptual Fusion Representation (PFR). PFR efficiently aligns and fuses the features of different modalities and utilizes intent tags to obtain optimal textual semantic insights, guiding the learning process of other modalities. Experiments show that our video and audio feature enhancement and transformation approach significantly outperforms the baseline, improving the processing efficiency and performance of multimodal data. These improvements are crucial for multimodal intent detection tasks.