Scholarlink

<p>Recent advancements in image captioning have leveraged sophisticated deep learning and transformer architectures to enhance visual understanding. However, the increasing complexity of these models often yields diminishing returns in quality, while significantly increasing training time, slowing inference speed, and consuming more computational resources and energy. This raises sustainability concerns, particularly regarding the environmental impact of training large models. To address these challenges, we propose a novel image captioning approach based on a transformer encoder-decoder architecture with an adapted multi-head attention mechanism. The model utilizes EfficientNetV2B2 for feature extraction, Accelerated Linear Algebra (XLA) for computational acceleration, and the Gaussian error linear units (GELU) activation function to strike a balance between performance and efficiency. We systematically explore the impact of varying the number of attention heads, selecting the optimal configuration for a trade-off between caption quality and training efficiency towards a frugal model. Our results on the Flickr8k dataset demonstrate substantial improvements, including a 207% relative improvement in Bilingual Evaluation Understudy (BLEU)-4, a 133.93% improvement in BLEU-3, a 109.24% gain in Consensus-based Image Description Evaluation (CIDEr), along with a 10.75% reduction in training time and a 28.06% increase in inference speed. We subsequently conducted an ablation study to assess the contribution of each component within our model, with a particular focus on evaluating its frugality through estimates of energy consumption and greenhouse gas emissions (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(CO_2\)</EquationSource> </InlineEquation>). The gains reach 16.13% reduction in energy consumption and a 66.67% decrease in <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(CO_2\)</EquationSource> </InlineEquation> emissions. Additionally, to further examine scalability and generalization of our model, we conducted experiments on the larger and more diverse Flickr30k dataset, where it maintained strong performance, confirming its robustness across varying data conditions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Efficient Multimodal Generative AI Model Towards Frugal Image Captioning Using Deep Learning and Attention Mechanism

  • 期刊论文

  • 发表日期：

    2026年1月9日

查看全文
  • Leila Helali

摘要

Recent advancements in image captioning have leveraged sophisticated deep learning and transformer architectures to enhance visual understanding. However, the increasing complexity of these models often yields diminishing returns in quality, while significantly increasing training time, slowing inference speed, and consuming more computational resources and energy. This raises sustainability concerns, particularly regarding the environmental impact of training large models. To address these challenges, we propose a novel image captioning approach based on a transformer encoder-decoder architecture with an adapted multi-head attention mechanism. The model utilizes EfficientNetV2B2 for feature extraction, Accelerated Linear Algebra (XLA) for computational acceleration, and the Gaussian error linear units (GELU) activation function to strike a balance between performance and efficiency. We systematically explore the impact of varying the number of attention heads, selecting the optimal configuration for a trade-off between caption quality and training efficiency towards a frugal model. Our results on the Flickr8k dataset demonstrate substantial improvements, including a 207% relative improvement in Bilingual Evaluation Understudy (BLEU)-4, a 133.93% improvement in BLEU-3, a 109.24% gain in Consensus-based Image Description Evaluation (CIDEr), along with a 10.75% reduction in training time and a 28.06% increase in inference speed. We subsequently conducted an ablation study to assess the contribution of each component within our model, with a particular focus on evaluating its frugality through estimates of energy consumption and greenhouse gas emissions ( \(CO_2\) ). The gains reach 16.13% reduction in energy consumption and a 66.67% decrease in \(CO_2\) emissions. Additionally, to further examine scalability and generalization of our model, we conducted experiments on the larger and more diverse Flickr30k dataset, where it maintained strong performance, confirming its robustness across varying data conditions.