Deepfakes, synthetic media manipulated using AI, pose significant challenges to credibility and security. With the increasing sophistication of deepfake generation, robust detection methods are crucial. In this paper, CoDeiT (Contrastive Data-efficient Transformers) is introduced, a framework for deepfake detection integrating a hierarchical attention mechanism in HiLo Transformer architecture with contrastive learning. It uses HiLo Attention to separate high-frequency (Hi-Fi) and low-frequency (Lo-Fi) information, enhancing computational efficiency and detection accuracy. The contrastive learning framework further increases discriminative power by maximizing the similarity between genuine instances and minimizing it between genuine and fake ones. Extensive data augmentation improves robustness across diverse datasets. Comprehensive experiments on benchmark datasets validate CoDeiT’s effectiveness. Three variations of the architecture have been proposed: CoDeiT-S, CoDeiT-L, and CoDeiT-XL, each differing in the number of parameters and attention heads. CoDeiT-XL has achieved 86.9% accuracy and 0.95 AUC on DFDC, and 78.5% accuracy and 0.89 AUC on the challenging CelebDF dataset when trained on the FaceForensic++ dataset. It outperformed all state-of-the-art deepfake detection methods. CoDeiT is effective for deepfake detection due to its unique architecture and ability to capture both high-frequency and low-frequency information efficiently. The combination of high and low-frequency information allows the CoDeiT to extract rich and detailed features from the data. This dual focus is particularly effective in detecting subtle inconsistencies and manipulations present in deepfakes.