Hybrid Vision Transformer and CNN for Deepfake Image Detection
摘要
The rapid advancement of deep learning models that can synthesize and generate hyper-realistic videos, known as Deepfakes, and their public accessibility have raised concerns about potential malicious uses. Deep learning techniques can now generate new faces, swap faces between two people in a photo or a video, alter facial expressions and facial features. These powerful video and photo manipulation methods can find potential use in many fields. However, they also pose a growing threat to everyone if individuals use them for harmful purposes such as identity theft, phishing, and scams. In this work, we propose a Vision Transformer and CNN hybrid for detecting fake faces. The hybrid model consists of two components: a Convolutional Neural Network (CNN) and a Vision Transformer (ViT). The CNN extracts local features, while the ViT captures and analyzes the global features that CNN might have missed. We trained our model on a combination of 140k Real and Fake Faces and Real and Fake Face Detection datasets and achieved 92.33% accuracy. Our contribution is that we added CNN for local feature capturing to the ViT architecture and got a competitive result on a hybrid dataset.