<p>Sentiment analysis is a critical task in natural language processing (NLP), driven by the exponential growth of user-generated content across social media and review platforms. Transformer-based models such as BERT have significantly advanced the field by capturing contextual nuances, yet they often struggle with subtle sentiment, sarcasm, and long-range dependencies common in movie reviews. We propose Dense Residual Attention Network (DRANet), a model designed to enhance transformer-based architectures for binary sentiment classification on IMDb and MR datasets. DRANet integrates a pre-trained BERT encoder with a customized classification head that employs dense (fully connected) layers and residual connections to improve gradient flow and feature propagation, enabling the capture of intricate cross-layer patterns. In addition, we introduce an attention-averaging mechanism that consolidates attention across heads in the final transformer layer, applying the aggregated weights to hidden states to emphasize sentiment-bearing tokens while down-weighting irrelevant context. Given the computational intensity of large transformer encoders and multi-layer residual heads, training and deployment at scale naturally require high-performance computing (HPC) resources. To assess scalability, we benchmarked DRANet under single and dual GPU settings. On IMDb (long-form reviews, 294 tokens), per-epoch training time dropped from 12.09 to 6.18 min (1.95<InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="11227_2025_7971_Article_IEq1.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\times$</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> speedup), while on MR (shorter sentences, 20 tokens), time decreased from 0.50 to 0.27 min (1.85<InlineEquation ID="IEq2"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="11227_2025_7971_Article_IEq1.gif" Format="GIF" Height="13" Rendition="HTML" Resolution="72" Type="Linedraw" Width="19" /> </InlineMediaObject> <EquationSource Format="TEX">$\times$</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> speedup), both with nearly identical validation accuracy. These benchmarks, though limited to 3 epochs for tractability, confirm that DRANet scales efficiently under parallel training, an essential property for real-world deployments such as high-throughput sentiment monitoring in e-commerce or social media streams. Empirical results further show that DRANet outperforms baseline transformer classifiers on IMDb and MR in both accuracy and F1 score, effectively handling nuanced and context-dependent sentiments. Together, these findings highlight DRANet’s dual contribution, advancing sentiment analysis performance and demonstrating practical scalability in HPC-class environments, making it a strong candidate for large-scale, compute-intensive NLP applications.</p>

Integrating dense layers with residual connections into transformers for enhanced sentiment classification

期刊论文
发表日期：

2025年11月8日

查看全文

Khaled Alahmadi,
Sultan Alharbi,
Xianzhi Wang

摘要

Sentiment analysis is a critical task in natural language processing (NLP), driven by the exponential growth of user-generated content across social media and review platforms. Transformer-based models such as BERT have significantly advanced the field by capturing contextual nuances, yet they often struggle with subtle sentiment, sarcasm, and long-range dependencies common in movie reviews. We propose Dense Residual Attention Network (DRANet), a model designed to enhance transformer-based architectures for binary sentiment classification on IMDb and MR datasets. DRANet integrates a pre-trained BERT encoder with a customized classification head that employs dense (fully connected) layers and residual connections to improve gradient flow and feature propagation, enabling the capture of intricate cross-layer patterns. In addition, we introduce an attention-averaging mechanism that consolidates attention across heads in the final transformer layer, applying the aggregated weights to hidden states to emphasize sentiment-bearing tokens while down-weighting irrelevant context. Given the computational intensity of large transformer encoders and multi-layer residual heads, training and deployment at scale naturally require high-performance computing (HPC) resources. To assess scalability, we benchmarked DRANet under single and dual GPU settings. On IMDb (long-form reviews, 294 tokens), per-epoch training time dropped from 12.09 to 6.18 min (1.95 $\times$ $\times$ speedup), while on MR (shorter sentences, 20 tokens), time decreased from 0.50 to 0.27 min (1.85 $\times$ $\times$ speedup), both with nearly identical validation accuracy. These benchmarks, though limited to 3 epochs for tractability, confirm that DRANet scales efficiently under parallel training, an essential property for real-world deployments such as high-throughput sentiment monitoring in e-commerce or social media streams. Empirical results further show that DRANet outperforms baseline transformer classifiers on IMDb and MR in both accuracy and F1 score, effectively handling nuanced and context-dependent sentiments. Together, these findings highlight DRANet’s dual contribution, advancing sentiment analysis performance and demonstrating practical scalability in HPC-class environments, making it a strong candidate for large-scale, compute-intensive NLP applications.