Gaze target detection, which involves perceiving human gaze target position, provides crucial insights for understanding human attention and intention. However, most existing methods struggle to extract effective features enriched with prior knowledge from the original dataset, overlooking information loss during decoding and the perceptual discrepancies in/out predictions. In this work, we propose a Filtered human-environment Gaze Interaction (FGI-Gaze) method for gaze target detection in daily scenarios. Our method constructs gaze interactions between humans and the environment from a single image modality, and then applies filtering to preserve the fine details in feature maps, enabling more accurate detection. Additionally, we propose to use distinct global semantic features to determine whether the gaze target is out of frame. Experiment results demonstrate that our FGI-Gaze method has achieved state-of-the-art (SOTA) performance on the GazeFollow, VideoAttentionTarget, and GFIE datasets, reducing the average distance error by 6%, 11% and 19%, compared to previous SOTA methods.