Gaze Target Detection with Visual Prompt Tuning Based on Attention

Existing works focus on using convolutional networks as the feature map extractors for both scene and head in gaze target detection tasks. Cordonnier [1] demonstrated that attention mechanisms can entirely substitute convolutional layers and attain top-notch results in vision-related tasks. In this work, we utilize the Visual Prompt Tuning (VPT), an alternative to full fine-tuning for large-scale transformer models in vision, as feature map extractor. Besides, A binary image of the eye position, with black pixels designating the eye bounding box and white pixels on the rest of the head image, is introduced because the eye is particularly important for the gaze detection task, Specifically, our model introduces parameters equivalent to half of ResNet50’s and experiments demonstrate that our model performs favorably against VideoAtt on GazeFollow datasets and VideoAttentionTarget datasets. This implies that in the gaze target detection task, using VPT as the image feature extractor performs better than convolutional networks.