Prediction of driver fixation can improve driving safety by simulating how experienced drivers allocate their attention. However, most of the research on the traditional driver’s fixation prediction model focus on the scene in clear days, the research on adverse weather(such as rainy and night) are relatively few, while adverse weather are more susceptible to traffic accidents. Training a model for each specific scenario is unrealistic, as collecting and annotating a large amount of data for every weather condition is too expensive. Data often dictates the effectiveness of model training, and insufficient data can hinder the model’s ability to adapt effectively to the task. Unsupervised domain adaptation methods can learn generalized features of different domains and improve the cross-scene ability of the model, so that it can also show good performance in new domains. This paper proposes a novel dual-branch network designed to leverage high-level semantic information extracted by CLIP, thereby enhancing the feature representation capability of the model and improving its understanding and adaptability across diverse scenarios. Additionally, an adversarial learning module is introduced to guide the model in learning domain-invariant features during training, thereby facilitating its adaptation to the cross-weather prediction task. We conducted experiments from TrafficGaze to DrFixD(rainy) and DrFixD(night). The results demonstrate that our model outperformed existing methods across five evaluation metrics and achieved the overall best performance.