Deepfakes have become commonplace due to the rise of GANs and LLM based generative models. With this, there has also been a rise in research towards effective deepfake-detection. In this work, we propose a transformer based model for deepfake-detection, operating on frequency-domain image features. Alongside, we also test the effectiveness of our solution with a smaller CNN-based model, for the sake of comparing model complexity vis-a-vis feature set effectiveness. In this paper, we focus on detection of Diffusion Model (DM) generated deepfake images. We train our models on an extensive dataset of diffusion model–generated images. We also demonstrate their generalization capabilities and report that while smaller models work well for in-domain testing, we do need larger, deeper neural networks for improved generalization. Our experiments on the recently proposed DeepFakeEval dataset corroborate our findings.