Offline reinforcement learning (RL) aims to derive effective policies from pre-recorded and fixed dataset without additional interactions with the environment, which is conducive to decrease the costs and risks when training in the real world. However, offline RL methods usually suffer the extrapolation error caused by out-of-distribution (OOD) data. Reducing the uncertainty caused by the agent’s exploration can contribute to avoiding OOD actions and facilitate learning a robust policy, but it depends on an accurate representation of the agent’s exploration capability. Besides, it is difficult to achieve a balance between exploration and conservation. In this paper, we propose a new offline RL method named contrastive novelty (CONO) for anti-exploration. CONO makes a precise evaluation on the exploration capability of the agent by contrasting the novelty between adjacent state-action pairs, where the novelty indicates the agent’s familiarity with a given state-action pair. In addition, combining with anti-exploration, CONO can prevent the agent from entering excessively novel state-action space to achieve a balance between exploration and conservation. Comprehensive experiments are conducted on the D4RL benchmark that is usually used to simulate the environment of robot control tasks. Experimental results show the proposed CONO outperforms state-of-the-art (SOTA) methods on the standard benchmark for offline RL.