<p>Contrastive learning is a deep learning paradigm that enables training models without supervision. These methods learn by maximizing similarity between augmented variants of an instance (positive pairs) while minimizing their similarity with the others (negative pairs). In several fields, including image-based sign language recognition (SLR), they have achieved considerable success. Image-based SLR focuses on identifying signs from images. In this context, signs often differ by subtle variations in orientation (e.g., flips, rotations). Hence, when generating positive pairs by applying augmentation methods, it may happen that at least one of the augmented instances no longer belongs to the same class as the original. This results in false positive pairs, leading to a similarity maximization between instances of different classes. This issue, referred to as <i>semantic drift</i>, leads to a poorly discriminative embedding space and limited performance on downstream tasks. To address this limitation, this paper introduces a general framework in which any contrastive method can be applied while remaining robust to semantic drift. Experimental results show improved robustness in scenarios both with and without semantic drift.</p>