<p>Social media have become dynamic platforms where people convey their emotions and sentiments through varied multimedia modes. This work suggests a novel hybrid method for multimedia sentiment analysis by integrating both the visual and textual data for more accurate sentiment classification. With increasing popularity of multimedia data covering images, videos, and text there is an urgent need for strong analytical techniques that can analyze sentiments from a variety of modalities. The suggested strategy presents a Hybrid Multimodal Sentiment Framework (HMSF) that relies on Convolutional Neural Networks (CNN) for the extraction of major visual features from videos and pictures, and on Transformer-based models, especially BERT (Bidirectional Encoder Representations from Transformers), for a deep context comprehension of text data. CNNs are good at extracting spatial and temporal features from visual inputs, whereas BERT’s self-attention mechanisms are best at comprehending intricate linguistic patterns and context in textual data. In the fusion phase, top-level features of CNN and BERT are fused to create a single, rich representation of the multimedia input, which is then passed through a classification layer for end sentiment prediction. The main contribution of this work is the fusion strategy, where visual and textual signals are synergistically combined to represent the subtle nature of sentiment expression. Experimental tests on typical multimedia sentiment databases prove that the introduced HMSF model outperforms current state-of-the-art in accuracy, precision, recall, and F1-score.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hybrid approach for sentiment analysis of multimedia contents: integrating visual and textual analysis

  • Prabakaran Thangavel,
  • Ravi Lourdusamy

摘要

Social media have become dynamic platforms where people convey their emotions and sentiments through varied multimedia modes. This work suggests a novel hybrid method for multimedia sentiment analysis by integrating both the visual and textual data for more accurate sentiment classification. With increasing popularity of multimedia data covering images, videos, and text there is an urgent need for strong analytical techniques that can analyze sentiments from a variety of modalities. The suggested strategy presents a Hybrid Multimodal Sentiment Framework (HMSF) that relies on Convolutional Neural Networks (CNN) for the extraction of major visual features from videos and pictures, and on Transformer-based models, especially BERT (Bidirectional Encoder Representations from Transformers), for a deep context comprehension of text data. CNNs are good at extracting spatial and temporal features from visual inputs, whereas BERT’s self-attention mechanisms are best at comprehending intricate linguistic patterns and context in textual data. In the fusion phase, top-level features of CNN and BERT are fused to create a single, rich representation of the multimedia input, which is then passed through a classification layer for end sentiment prediction. The main contribution of this work is the fusion strategy, where visual and textual signals are synergistically combined to represent the subtle nature of sentiment expression. Experimental tests on typical multimedia sentiment databases prove that the introduced HMSF model outperforms current state-of-the-art in accuracy, precision, recall, and F1-score.