<p>Real-world image–text posts mix subtle visual signals with short, informal language, making multimodal sentiment analysis susceptible to label noise, miscalibration, and brittle decisions. We present SentiGuard+, a reliability-aware model that processes images with a Swin Transformer and text with RoBERTa-large, fusing them via a bi-directional cross-attention module (CrossWeave). For each input, the system produces three interpretable outputs: class probabilities, a calibrated reliability score, and an accept/abstain decision for risk-controlled deployment. The fused representation is equipped with an evidential Dirichlet head and a hierarchical noisy-label channel (HD-SIT); we then apply reliability-preserving evidence calibration and a simple two-feature gate (P-Gate) that bases acceptance on predicted entropy and calibrated variance. Across standard multimodal sentiment benchmarks, SentiGuard+ improves overall accuracy and F1 while reducing calibration error, and it delivers stronger accepted-set performance at matched coverage. We find evidence that a single calibrated-and-gated operating point transfers between two datasets with minimal drift. These results indicate that combining Swin Transformer and RoBERTa-large with evidential modeling and selective gating yields better-conditioned probabilities, higher accepted-set F1, and portable operating points suitable for real-world use.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing multimodal sentiment analysis reliability: SentiGuard+ with Dirichlet evidence and selective prediction

  • Komal Rani Narejo,
  • Hongying Zan,
  • Samih M. Mostafa,
  • Faten Khalid Karim,
  • Faisal Mehmood,
  • Ayesha Yaseen

摘要

Real-world image–text posts mix subtle visual signals with short, informal language, making multimodal sentiment analysis susceptible to label noise, miscalibration, and brittle decisions. We present SentiGuard+, a reliability-aware model that processes images with a Swin Transformer and text with RoBERTa-large, fusing them via a bi-directional cross-attention module (CrossWeave). For each input, the system produces three interpretable outputs: class probabilities, a calibrated reliability score, and an accept/abstain decision for risk-controlled deployment. The fused representation is equipped with an evidential Dirichlet head and a hierarchical noisy-label channel (HD-SIT); we then apply reliability-preserving evidence calibration and a simple two-feature gate (P-Gate) that bases acceptance on predicted entropy and calibrated variance. Across standard multimodal sentiment benchmarks, SentiGuard+ improves overall accuracy and F1 while reducing calibration error, and it delivers stronger accepted-set performance at matched coverage. We find evidence that a single calibrated-and-gated operating point transfers between two datasets with minimal drift. These results indicate that combining Swin Transformer and RoBERTa-large with evidential modeling and selective gating yields better-conditioned probabilities, higher accepted-set F1, and portable operating points suitable for real-world use.