Image-goal navigation is an important and difficult problem that requires the robot to reason about the target location and reach the goal based only on visual input received in real time. It gets even trickier when considering the problem of domain transfer to simulators with higher fidelity or real-world scenarios. Existing work has demonstrated promising results using reinforcement learning (RL) with extensive domain randomization, but typical end-to-end approaches are limited by poor extrapolation ability to new scenarios. In this paper, we propose a learning framework that enables robots to navigate under a unified style, to learn a general strategy in environments with different fidelity and styles. The experiments demonstrate that compared with the existing baseline, the proposed method can better generalize to new scenarios and enhance the performance of image-goal tasks.