Unleash the Capabilities of the Vision-Language Pre-training Model in Gaze Object Prediction

In a retail environment, it is valuable to evaluate the products of interest to perform accurate recommendations. However, the existing method (gaze following) only predicts the gaze area, and the prediction problem of gaze objects has not been fully explored. To this end, this paper proposes a new visual language model based on pre-trained large language models for the gaze object prediction framework, named EdgeCLIP. Primarily, we employ a set of adaptable and instructive cues to judiciously infuse instructional cues into the extensive language model, while proficiently retaining its pre-training knowledge. Secondly, we introduce a multi-head pooled attention block, MPATB, to achieve semantic enhancement and extract the joint representation of multimodal components, thereby mitigating the discrepancy in fixation points and subsequently reducing inaccurate predictions of gaze objects. Furthermore, we introduce a regulatory loss function that effectively governs the gaze heatmap within the stared box. A large number of experiments have proved that our model outperforms previous models. The code will be available in: https://github.com/fadaishaitaiyang/EdgeCLIP.