Self-supervised pre-training is effective in learning discriminative features from unlabeled medical images. However, typical self-supervised models lead to sub-optimal representations due to negligence of high anatomical similarity present in the medical images. This affects the negative and positive pairs in discriminative self-supervised models to learn view-invariant representations. Various methods are proposed to address this issue. However, many of them either concentrate on preserving pixel-level details or offer solutions for specific modalities. In this context, we propose a generalized solution to leverage the anatomical similarities while relaxing the requirements of complex pixel-preservation learning. Specifically, we introduce IDQCE: Instance Discrimination Learning through Quantized Contextual Embeddings. The proposed approach leverages the sparse discrete contextual information to guide the self-supervised framework to learn more informative representations for medical images. We evaluate the representations learned by IDQCE through comprehensive experiments and observe more than 3% performance gain under linear evaluation protocol over other SOTA approaches in multiple downstream tasks.