MIM-ViT: Deepfake Detection Using Masked Image Modelling and Vision Transformer

Over the last decade, deep learning has become one of the fastest-growing fields in computer science. It finds applications in several sectors, such as Healthcare, Agriculture, Financial Services, and Crime Investigation. However, with the rapid development of Generative Adversarial Networks in 2017, the concept of deepfakes emerged. The term deepfake refers to an artificially synthesized image or video generated using techniques like Face Swapping and Face Expression Reenactment. Such face manipulation techniques have become highly refined, making deepfakes impossible to detect by the naked eye. Deepfake videos can potentially be used for various malicious purposes. Spreading misleading news via politicians or celebrities is one such example. Such ill-intended videos can exacerbate the increasingly prevalent problem of false information online. Therefore, being able to differentiate between real and fake videos is crucial. This work proposes a solution based on Masked Image Modelling using Auto Encoders and Vision Transformers to tackle the problem of Deepfake Detection. The solution consists of two sub-models working in parallel, namely the Multiscale Vision Transformer and the Masked Autoencoder, ConvNeXt. A novel facial quality detection algorithm is developed, which helps improve the data quality by overcoming the challenge of misrepresented facial data. The proposed model, MIM-ViT, achieves competitive results on popular datasets like the Deepfake Detection Challenge Preview and Face Forensics++ with a test accuracy of 80.22 and 98.1% along with an Area Under the Receiver Operating Characteristics score of 84.48 and 99.8% respectively. The proposed model generalizes well and performs competitively on unseen data, achieving an AUCROC score of 68.21%.