Scholarlink

<p>To address ineffective feature fusion and feature loss in gaze estimation under unconstrained environments, this study proposes a multi-scale feature fusion model, CAF-ViT (Cross-Attention Fusion Vision Transformer). The model takes multi-scale face images as input, uses ResNet-18 to extract feature maps at different granularities, and introduces learnable Class Tokens per scale. In the fusion stage, Class Tokens of different scales first perform self-attention in their respective Transformer Encoders to aggregate local details and global semantics. Then, by swapping the token sequences and computing cross-attention, the model achieves bidirectional interaction and deep fusion of coarse- and fine-grained features. To further refine feature representation, an additional attention layer is added after cross-attention. It linearly transforms the original query vector with Sigmoid activation to generate new query weights, and linearly maps the attention output to new value vectors, improving the representation of task-relevant features. The fused Class Token is finally regressed to gaze direction via a multilayer perceptron. Experiments show estimation errors of <InlineEquation ID="IEq1"><EquationSource Format="TEX">\(3.73^\circ\)</EquationSource></InlineEquation> on MPIIFaceGaze, representing an 8.6% improvement over the baseline hybrid CNN-Transformer, and errors of <InlineEquation ID="IEq2"><EquationSource Format="TEX">\(5.27^\circ\)</EquationSource></InlineEquation> on EyeDiap and <InlineEquation ID="IEq3"><EquationSource Format="TEX">\(10.43^\circ\)</EquationSource></InlineEquation> on Gaze360. Ablation studies validate the multi-scale fusion strategy and improved attention mechanism.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multi-scale feature fusion gaze estimation model based on convolutional neural network and vision transformer

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月1日

查看全文
  • Peng Wang,
  • Xuena Wang,
  • Shuo Yuan,
  • Shan Cong

摘要

To address ineffective feature fusion and feature loss in gaze estimation under unconstrained environments, this study proposes a multi-scale feature fusion model, CAF-ViT (Cross-Attention Fusion Vision Transformer). The model takes multi-scale face images as input, uses ResNet-18 to extract feature maps at different granularities, and introduces learnable Class Tokens per scale. In the fusion stage, Class Tokens of different scales first perform self-attention in their respective Transformer Encoders to aggregate local details and global semantics. Then, by swapping the token sequences and computing cross-attention, the model achieves bidirectional interaction and deep fusion of coarse- and fine-grained features. To further refine feature representation, an additional attention layer is added after cross-attention. It linearly transforms the original query vector with Sigmoid activation to generate new query weights, and linearly maps the attention output to new value vectors, improving the representation of task-relevant features. The fused Class Token is finally regressed to gaze direction via a multilayer perceptron. Experiments show estimation errors of \(3.73^\circ\) on MPIIFaceGaze, representing an 8.6% improvement over the baseline hybrid CNN-Transformer, and errors of \(5.27^\circ\) on EyeDiap and \(10.43^\circ\) on Gaze360. Ablation studies validate the multi-scale fusion strategy and improved attention mechanism.