Gaze serves as a pivotal element in non-verbal communication, effectively reflecting human visual attention. In this paper, we introduce object saliency as a critical prior for predicting gaze targets. Most existing methods predominantly rely on extracting global scene features, thereby neglecting the semantic distinction between background and foreground objects. However, compared to the unstructured background, human gaze is inherently biased towards specific, salient objects. To address this, we propose a novel Gaze Target Detection with Object Saliency and Vector Modeling framework (Gaze-OSV), which incorporates a human-object context module. Unlike prior works, Gaze-OSV features a Residual Object Saliency Injection mechanism that utilizes a trainable object detector to explicitly highlight potential targets. Furthermore, it implicitly models the human-object visual interaction by combining these enhanced features with a learnable Gaze Vector Token. Extensive experiments on two standard benchmarks, GazeFollow and VideoAttentionTarget, demonstrate the effectiveness and superior performance of Gaze-OSV.