Infrared detectors play an essential role in real-world applications due to their robustness under adverse illumination and lighting conditions. Producing high-resolution infrared cameras is significantly more expensive than visible cameras, leading to a lower volume of labeled infrared data than visible ones. In this paper, we consider a realistic yet challenging application for infrared domain object detection, where the training process relies only on visible data. For cross-domain object detection, the generalization ability is often reduced by domain shifting. We propose a Gaussian Fourier Convolution (GFC) to learn more common representations among infrared and visible images. GFC integrates the global properties of Fast Fourier Convolution into a Gaussian edge detector, enhancing the capacity to capture shape edge information while minimizing the influence of local edges. Experimental results on the FLIR and KAIST datasets demonstrate that the proposed method significantly improves compared to the baseline across various detectors.