Detecting Deepfakes Across Modalities Using Image and Audio Cues
摘要
As a result of increased cases of manipulated digital content, the capacity to identify fake media has emerged as a problematic research issue. This paper presents a multimodal approach to detect fake content through a combination of image and audio features. The presented system is developed on the basis of two reliable benchmark datasets: CASIA v2 to detect tampered images and ASVspoof 21 to distinguish between spoofed audio. The modality streams are processed by separate streams of ResNet-50 and the high-level feature representations are extracted. Such characteristics are then combined into one embedding and subjected to a collective classification model. Even though trained on unimodal samples, the framework is tested on generated video samples by extracting a representative frame and audio that shows the ability of the model to generalize outside of its training regime. The results of the comparative procedure show that the hybrid method provides a significant increase in comparison with individual modalities, and the accuracy of the hybrid method is 95.4 per cent and AUC is 0.96. These results highlight the potential of the multimodal learning approach to the increasing risks of fake media.