Constructing Audio-Visual Heterogeneous Graphs for Deepfake Detection
摘要
The rapid advancement of deepfake technologies poses significant threats to information credibility, necessitating the urgent development of generalizable detection methods. Most existing methods either rely on unimodal cues or utilize multimodal fusion strategy to capture audio-visual dissonances. However, unimodal methods are less adaptable for cases involving both audio and visual forgeries. Multimodal fusion approaches, which predominantly focus on cross-modal cues, may hinder the mining of useful intramodal information. In this chapter, we propose to represent audio-visual data as heterogeneous graphs and design a novel heterogeneous graph modeling module for deepfake detection. We sample audio and visual segments from input data and encode them using separate transformer encoders. These segments are treated as graph vertices, and different types of edges are assigned to model both intra- and intermodal relationships explicitly. During the graph modeling, we update the audio and video subgraphs independently to exploit inconsistencies within single modalities and integrate the aligned multimodal features through graph attention mechanism to capture audio-visual mismatches. Finally, we combine both intra- and intermodal information to produce a graph-level representation for discrimination. Experiments conducted on DFDC and FakeAVCeleb datasets demonstrate the superiority of our method, which exhibits generalizable and robust performance in both single-dataset and cross-dataset evaluations.