The goal of the gaze target prediction is to determine the location where the person is focusing and the probability of the gaze falls outside the image. Although prior works have addressed this task by regressing heatmaps centered on the gaze location, they typically fail to incorporate the scene's semantic information. In this work, we first generate 3D point cloud of the given image based on depth estimation and camera intrinsics. Then we combine the point cloud and estimated 3D gaze vector to generate the 3D field of view (FoV) heatmap. Scene contextual cues are finally merged to get the output heatmap. Our method achieves competitive results on the ChildPlay, GazeFollow, and VideoAttentionTarget datasets.