Scholarlink

<p>Emotions play a vital role in shaping human behavior, making accurate emotion recognition crucial in areas like social robotics, mental health diagnostics, and human–computer interaction. Existing multimodal systems often suffer from cross-modal interference, poor feature extraction, and weak integration across text, audio, and video data. This paper proposes a novel framework Random-Coupled Neural Network with Binary Light Spectrum Optimization (BLSO) for Context-Aware Multimodal Emotion Recognition that addresses these challenges through tailored preprocessing and advanced feature fusion. Text is processed with tokenization and lemmatization, audio with noise reduction and voice activity detection, and video with frame extraction and face normalization. Features are extracted using an Elastic Decision Transformer for text, Short-Time Fourier and Continuous Wavelet Transforms for audio, and Local Directional Structural Patterns (LDSP) for video. A Context-Aware Attentive Multilevel Feature Fusion (CAMFF) network integrates these features, while a Random-Coupled Feedback Attention Neural Network (RCFANN) optimized via BLSO performs classification. Experiments on the IEMOCAP dataset demonstrate the proposed system achieves over 99% accuracy, precision, recall, and F1-score, outperforming existing multimodal fusion methods by 4–6%.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Random-coupled neural network with binary light spectrum optimization based context-aware multimodal emotion recognition using audio, video, and text feature fusion approach

  • 期刊论文

  • 发表日期：

    2025年9月5日

查看全文
  • T. Indra Deepika,
  • A. N. Sigappi,
  • D. N. V. S. L. S. Indira

摘要

Emotions play a vital role in shaping human behavior, making accurate emotion recognition crucial in areas like social robotics, mental health diagnostics, and human–computer interaction. Existing multimodal systems often suffer from cross-modal interference, poor feature extraction, and weak integration across text, audio, and video data. This paper proposes a novel framework Random-Coupled Neural Network with Binary Light Spectrum Optimization (BLSO) for Context-Aware Multimodal Emotion Recognition that addresses these challenges through tailored preprocessing and advanced feature fusion. Text is processed with tokenization and lemmatization, audio with noise reduction and voice activity detection, and video with frame extraction and face normalization. Features are extracted using an Elastic Decision Transformer for text, Short-Time Fourier and Continuous Wavelet Transforms for audio, and Local Directional Structural Patterns (LDSP) for video. A Context-Aware Attentive Multilevel Feature Fusion (CAMFF) network integrates these features, while a Random-Coupled Feedback Attention Neural Network (RCFANN) optimized via BLSO performs classification. Experiments on the IEMOCAP dataset demonstrate the proposed system achieves over 99% accuracy, precision, recall, and F1-score, outperforming existing multimodal fusion methods by 4–6%.